From Idea to Production in Eight Hours: How we Work at Clipboard Health

It’s also relevant to the beginning of this story to note this happened at 9:15 AM PST.

Member of Product Team: We’re running into some issues where facilities aren’t getting any HCPs (healthcare professionals) or are getting 1 fill and that HCP NCNS (healthcare professional no call no shows, meaning they booked a shift and did not show up to work). We’ve previously discussed delivering a “minimum quality level” across the marketplace… Even if we’re losing money (negative unit economics)... we need to prove some value to them regardless…

In the discussion that followed, a couple of important responses came from Bo and George (our President/COO and Head of Product, respectively):

Bo: (Note: Bo provided some context around previous spend and a Holiday pricing promotion, which reinforced the price elasticity of our HCPs, or our supply side) …so I’d guess we’re very much in the steep part of the return curve for spending more subsidies to ensure a minimum HCF experience (which I suspect would lift HCF LTV by far more than the subsidy). I think we should prioritize this, but @George Markoulakis want to get your thoughts.

George: Agree, though would prefer the scrappier/faster route if possible (even if we end up overpricing)...

For the next few hours, a few members of our product team stayed active on the Slack thread and talked through potential solutions: how could we execute something as soon as possible to solve the immediate customer problem? It was clear we wanted to take action, but we had to do it while either solving or at a minimum acknowledging the following potential outcomes:

This culminated in a call around 5 pm PST, which was recorded and added to the Slack thread. In that ~15-minute call, we finalized a plan to run a pricing script that would automatically increase the pay rates for facilities that had low fill rates previously, helping to make hard-to-fill shifts more appealing to HCPs. We split the “poor experience” facilities into a treatment and control group to make sure we would be able to see if our pricing script was effective after the experiment period. This would ultimately help us determine whether we’d want to continue subsidizing these facilities as well.

While getting the experiment out there (and seeing the results) was exciting, it was much more exciting to see just how fast something could happen in a fully remote work environment. Specifically, this experiment shed light on the way we work:

In the span of ~8 hours, we went from idea to execution on a large-scale pricing experiment impacting ~250+ healthcare facilities and thousands of healthcare professionals who would have the opportunity to earn more than they typically would on shifts at these facilities.

In most marketplaces, the process of releasing an experiment like this may have required multiple levels of approval or many hoops to jump through before launching. At Clipboard, we believe that speed is our number one differentiator; there might be a handful of companies that can do the things we do, but our advantage lies in doing those things faster, with no loss in quality and while learning as much or more.

As cool as the experiment was, it wouldn't have existed if we didn't put a constant high emphasis on the importance of customer conversations. We knew how important this issue was to customers because our operations and product teams were willing to reach out and ask. Without those conversations, we likely wouldn't have had the correct feelings of immediacy.

Because we do have those conversations, we knew this was a problem that made some of our HCFs unhappy, caused churn, and demanded immediate action. If we didn't solve the problem today, the customer was unlikely to still be a customer tomorrow.

An inability to provide value to both sides of the market is largely a marketplace failure, and as managers of the marketplace it was our responsibility to make sure we provided a positive experience to our facility customers. In this case, our speed was the difference between meeting those responsibilities or falling short in the eyes of our customer.

If the original Slack message was sent in a DM instead of to a public channel, there’s a chance that the experiment never would have happened. Since the message was in a public Slack channel, it ended up getting wide engagement (including from leadership) which helped maintain forward momentum.

It was also important that the responses to the problem happened in a public channel. Without the direct request that we do something scrappy to quickly to solve the problem, there’s a high probability that the implementation could have taken days or weeks before we got something out to the customer. At Clipboard, we focus on public writing, forcing ideas to go through scrutiny (think open source, but with ideas not just code) while empowering individuals to take action.

In this case, our experience told us that the experiment we ran was unlikely to be the long-term solution. With that said, we also knew that in a marketplace, if you can’t provide value to customers, they’re not going to use you in the future. Imagine requesting 10 Lyft rides and never getting a driver – would you use the app again?

Even though we knew that this was unlikely to be a permanent solution, the alternatives were worse. It’s easy to let the perfect be the enemy of the good. Teams that over-optimize instead of acting run the risk of moving too slow or doing nothing. Zero customers are helped by the actions you don’t take. We believe customers deserve to see an almost instant response to the concerns they raise.

The pricing experiment was a great example of how we work at Clipboard Health: we solve customer problems as fast as possible. We hear about issues, tackle them immediately, and use the lessons from our experiments to improve.

The results of that pricing experiment were mixed. It was successful in increasing the fill rate in some markets, and unsuccessful in others. The business logic in our initial pricing script, intended to avoid negative unit economics for our treatment group, also prevented us from increasing pay rates enough in some markets.

Using the lessons of that experiment as a springboard, we are now conducting several other, more aggressive, trials that aim to raise healthcare facilities’ experience of our product above the minimum quality bar we seek to deliver.

The first experiment influenced our decision to relax some business logic on our system-wide pricing model and gave us data we needed to make our subsequent experiments better so that we’ll arrive at better solutions sooner.