Working Backwards Document - How We Ship Features: OverFilling

We think hard about ways to improve our user’s experience with our app, and about more conventional business-oriented concerns like churn and lost opportunities.

This piece is an example of an initiative to improve our healthcare facilities’ experience by ensuring that shifts filled on our app were fulfilled. Since we achieved this using unconventional thinking and methods, the Working Backwards Document format was unusually important for communicating the concept and getting everyone on the same page.

“Sometimes shifts get canceled or unassigned, but Clipboard has a replacement right away. There’s no hassle and knowing that someone is going to show up makes me more confident posting shifts on Clipboard.”

“I’m not really sure what you guys did, but even when there are cancellations the shift gets refilled right away and I don’t have to worry about shuffling around my own staff.”

“For the last few months, it’s been really hard to find shifts on Clipboard. Recently I’ve noticed that there are more open shifts at the facilities I like to work at and it’s easier to put my schedule together.”

Having a shift filled by Clipboard Health is not a guarantee that a worker will show up; 17% of non-deleted shifts claimed on our platform end up going unworked. For facilities, this makes Clipboard an unreliable proposition; having a cancellation on a shift where a facility expects someone to show up is worse than having no shift filled at all.

At the same time, in our most demand-constrained MSAs nurses who could otherwise work are unable to book shifts because they’re already filled.

“We stopped working with Clipboard because your nurses would always cancel. There isn’t anything you could say right now that would make me come back. We’re working with a different agency now, and it has been much better.”

The reliability of our network is a consistent concern when speaking to HCFs. HCFs frequently state that they do not experience cancellation issues with other agencies. To limit HCF churn (an event to which our business is particularly sensitive) we have to do whatever we can to improve the reliability of our network and give HCFs the best possible experience in the process.

Another frequent HCF complaint is that when other agencies have issues, those agencies ensure someone shows up regardless. We can eliminate this complaint cheaply and scale-ably.

This chart shows that the share of HCFs experiencing a 60-100% fill rate has steadily increased and stabilized over the last five months. This indicates that many of our markets are mature enough to support overfilling. We currently have enough network density to fill all the shifts that HCFs post in about 40 mature markets.

A slot is a group of shifts at a facility with the same agent requirement, start time, and duration. It’s essentially a block of work at the facility for which as many workers are needed as there are shifts posted. The physical constraints of the shifts (time, duration, and place) are the same for each shift within a single slot.

When we see the facility fill rate in oversupplied target markets exceeds 95% (this metric currently sits at 70-80%), we will know that we’ve been successful.

When HCFs post on shifts CBH, they’ll be fairly certain that we can fill them and extremely certain that once the shift is filled someone will actually show up. This is the ideal impossible customer experience we’re chasing.

We can leverage our network to efficiently hedge against potential cancellations and make targeted bets that ensure we have low incidences of shifts being claimed, canceled then going unworked. The core idea is to overfill shifts, i.e. naively to book multiple HCPs into a single shift, effectively buffering ourselves against a cancellation. In practice, we can be much more intelligent (expanded upon below) in how we allocate workers.

What we propose is overfilling at the slot level, not at the shift level. Let’s take a simple example to explain how this will work and why we are going to opt to overfill at the slot level.

In the following example, we have 3 generic slots that are all completely filled. Slot 1 is 6 shifts deep, slot 2 is 1 shift deep, while slot 3 is 3 shifts deep. Each square below represents a nurse while the number represents the probability of the nurse working the shift once they’ve claimed it.

Looking at this picture, you would probably say we should overfill the shift being worked by the nurse in slot 2. They have a 70% chance of canceling the shift and are the least reliable nurse in the entire diagram. In reality, it’s more efficient to overfill slot 1. The chance of at least one cancellation is 83% in slot 1, while it’s 70% in slot 2.

If we determine that we’d like to overfill a certain slot, we will create an extra shift (I will refer to this as a synthetic shift going forward) with the same start, end, charge, qualification, and facility as the other shifts in the slot. This synthetic shift can be merged into the place of any existing shift that gets canceled.

We will FCF the synthetic shift. In all cases, this will be the last worker to book into the slot. We will not FCF the least reliable nurse in the slot to reduce legal liability: we are not making choices on behalf of the facility, we serve the shift to whoever claimed first.

Eventually, we might be able to FCF the shift with the least reliable/lowest quality worker, but right now we don’t have good signals for quality. In addition, adding additional logic to the FCF procedure for the lowest reliability worker will add complexity we would like to avoid. Additionally, there’s a legal risk we would need to evaluate in choosing which HCP shows up for the facility.

We will do so at 4 hour lead time. This will ensure HCPs don’t begin driving to facilities prior to FCF (one of the biggest complaints around FCFs today). Because we pay out HCPs for 2 hours of shift time for FCFs within 24 hours lead time, this will compensate HCPs for the sunk cost of keeping the slot booked for us. Finally, it will allow us enough time to refill an upper bound of roughly 40% of marketplace cancellations in target markets using my back-of-the-envelope math.

FCF-ing at 24 hours, for example, would only allow us to refill an upper bound of 25% of cancellations in target markets. FCF-ing at 0 hours would allow us to refill an upper bound of 58% of cancellations in target markets. This number is not 100% because not all cancellations occur in fully filled slots.

The true percentage will be lower because we will threshold to minimize FCFs.

CBH will pay the cost of the synthetic FCFS. For all facilities, any FCF that occurs within 24 hours lead time results in a backpay of 2 hours to the nurse. For synthetic shifts that are FCF’d, CBH will pay the nurse 2 hours and not charge facilities.

Assuming that we overfill every 100% filled slot in May’s 40 highest volume demand-constrained markets, the synthetic FCFs themselves will cost about $700K monthly. Using the same back-of-the-envelope math we would make an additional $1 million in revenue from the shifts successfully overfilled. In other words. we make about $1.4 for every $1 we spend.

This is not accounting for other benefits of increased retention, which are likely to be significant but are more challenging to model.

The TLDR is that this implementation should generate profit.

From a pure revenue/cost-driven framework there is a more optimal solution that would vary from slot to slot, depending on which HCPs are filled in the shifts. We would continuously recalculate the likelihood of a cancellation in the slot, and once it drops below a certain threshold we would FCF the synthetic shift until we hit the FCF payout period at which point we would revert to 4-hour FCF.

We won’t optimize FCF time because while this profit-centered framework is better for CBH in the short term, it is worse for both sides of our marketplace (and thus worse for us in the long term). If we continuously recalculate the likelihood of cancellation, and FCF when we fall below a certain threshold, we will reduce the number of FCFs with a payout. We will also overfill fewer slots, so we will catch fewer edge cases.

Further, this is a more complex framework architecturally. For a v1 in which we’re building a lot of new infrastructure, we want to minimize the bug surface area by keeping the cancellation probability constant after the initial calculation.

We definitely could. This route would come at a distinct cost to the user experience (because more HCPs would experience FCFs once they arrive at the facility and no one has NCNSd) and has a few dependencies and conflicts.

The only viable method I know of to do this is with radar, a geolocation tool for tracking HCPs en route to their work. We want to avoid two HCPs showing up for the same shift to remove the burden of having to deal with a nurse who shows up and is surplus to requirements and to avoid paying for a full extra shift.

In the case where all HCPs in normal shifts are moving towards the facility, we would FCF the synth shift immediately. In the case where one HCP is not moving towards the facility, we would record a NCNS and merge in our synthetic shift. Further, we would have to prevent our backend from creating an urgent shift at this point.

Because this type of integration has a dependency on radar and a conflict with urgent shifts we will opt to continue with the basic four-hour lead time path. This may be an integration that we incorporate in the future, but we will wait until urgent shifts is developed before layering logic on top of it.

We will create a synthetic shift when a slot is fully filled and if the probability of all shifts in the slot being worked falls below a market-dependent dynamic threshold.

We will only overfill in markets that are oversupplied and have deep supply pools. HCPs in oversupplied markets are often unable to find a shift when they shop, therefore, a loose definition of an oversupplied market is a market with high fill rates, where HCPs have a low chance of a shopping session converting into a booking session. It doesn’t make sense initially to attempt to overfill in markets where we have low fill rates and low supply liquidity. The scant amount of synthetic shifts generated will draw supply away from normal shifts, and worsen the facility fill experience in undersupplied markets.

We may eventually want to run overfilling in undersupplied markets to provide a better experience to facilities HCPs prefer (i.e. abnormal facilities with high fill rates).

For the MVP of overfilling, we will set the threshold to a 70% chance of cancellation in the slot and walk it up or down manually depending on our ability to fill synthetic shifts. We’ll work dynamic thresholding into the service once we have the core infrastructure for synthetic shifts built.

I picked a 70% threshold because that should catch about 20% of cancellations in target markets without flooding the market with shifts at once and creating too many incremental FCFs. The median worker has an attendance rate of 74% on claimed shifts and the 75th percentile of slots is 3 deep. That gives the 75th percentile slot a 62% chance of cancellation. This is more back-of-the-envelope math used to pick the threshold, but we’ll monitor closely and tune the threshold. We just need an approximate as a starting point and we’ll find the equilibrium via experimentation.

We will not use overfilling in any capacity for those shifts. Pricing is the correct tool to get a slot to 100% fill, overfilling will keep it there.  Synth shifts will reduce the probability of deeper shifts being filled. Functionally, all regular shifts in the slot work as a hedge against each other. We should only hedge once the facility’s needs have been fully met.

If we overfill a slot that is < 100% filled, we are drawing away supply from normal shifts in non-overfilled slots. Functionally, synthetic shifts are a hedge against regular shifts, but in unfilled slots, deeper shifts function as a hedge against shallower shifts.

If a slot is overfilled, then a cancellation occurs we will merge in the synthetic shift. If the probability of a cancellation hits the threshold we will overfill again. This could theoretically occur ad-infinitum until the shifts start/FCF window.

We’ll leave the synthetic shift open to be refilled. For HCPs, it will functionally look like a normal shift. We will delete the synthetic shift at four hours lead time if no one has filled into it. If the cancellation falls into our attendance policy, it will impact the HCP the same way a cancellation on a normal shift would.

Synthetic shifts will be available through the calendar and map view as regular shifts. The flow will be fundamentally the same for HCPs, i.e. no cognitive load added.

HCFs will not be able to see anything related to overfilling unless a synthetic shift is merged into a normal shift. If a synthetic shift is merged (when a worker in a normal shift cancels), the HCF will receive a notification saying that we’ve found a better worker for them in their shift on dd/mm/yy at hh:mm am/pm.

When there are synthetic shifts involved, we will be reworking the way we deliver notifications to HCFs to mitigate the “drama” and noise around each shift. Based on the work that our team members did around the notifications experience we will make changes to the way our email /SMS notifications are triggered.

When a worker cancels and a synthetic shift is merged in, we’ll a singular notification saying that we’ve found someone better. We will trigger no notifications for cancellations on synthetic shifts.

This is not an essential component of experience we’re prioritizing, but it will be something we fix eventually.

We want to reduce the # of shifts booked and not worked in our test markets by 30%. This would mean that we’re tangibly improving the experience of HCFs posting on our platform while optimizing our market to allow more workers to work. The number of booked and not worked from about 40K monthly to 28K monthly.

We can, but we will not pursue a manual test. The feature as spec’ed does not really add anything to the surface area of our app experience. As far as I can project, the only thing we would really be testing is if we can get marginally more shifts filled in oversupplied markets. Given the risk of a manual test (staffing coordinators getting confused, multiple HCPs showing up, human errors in operating overfilling), and the cost of running it (man hours + delaying development) I don’t think it’s worth it.

There are 2 further risks to holding up product development in favor of a manual test:

We will proceed without manual experimentation, sprint to an engineered MVP, and test in a high-volume oversupplied market. This will allow us to fine-tune the feature before launching market-wide while also providing a high-fidelity pressure test of our assumptions

We would sprint to a minimum version that can create synthetic shifts based on a static threshold in specified markets. There are many dependencies on shifts and so creating new shift objects in the product could pose serious downstream or upstream issues. That said, recreating the shift’s infrastructure in a separate collection could take months due to the amount of business logic buried into each shift.

After coordinating with devs, we determined the following would make sense.

The likeliest cause of failure is if HCFs receive workers they did not expect/want through synthetic shifts being merged and requesting us to turn off the feature for them. Making sure HCPs receive and acknowledge FCF notifications is the best way to avoid this.

The next most likely cause of failure is that this spikes up FCFs, we set our threshold wrong, and spend a lot of money paying out HCPs for synthetic FCFS while making the experience worse for them and churning a lot of supply from these markets. This is especially bad if HCPs state they do not want to return to certain facilities because of the increase in FCFs, and then facilities ask us to turn off the feature for them. Setting thresholding correctly is the key to avoiding this outcome.

This is not the best customer experience I can think of.

On the HCF side, the ideal scenario is that we fill 100% of shifts and keep them 100% full, with the best quality nurses regardless of the market the HCF participates in. They have no worries about NCNS through our platform, and they can use our platform in case another agency or FTE turns out to be unreliable.

On the HCP side, the ideal scenario is that when they sign up for a shift they are sure that they’re not going to have it canceled on them, and if it is canceled we find them a shift within the same distance with equal to or greater pay.

Supply liquidity is an obstacle to achieving 100% fill in many markets. Maintaining supply liquidity while also enforcing high-quality standards for our workers is, at the moment, untenable. We’ll be able to start gaining worker quality signals once reviews are more widely used, at which point we’ll be able to evaluate the question of if we have enough high-quality supply to sustain a feature like this (at least in our oversupplied markets).

On the other side, we simply don’t have the shift liquidity in oversupplied MSAs to find an equal replacement to the synthetic shift that is FCF’d.

We will not. While this is a reasonable idea in principle, I would argue that overfilling is a feature intended to leverage one of our superpowers: filling shifts quickly. Anything that limits our ability to do so jeopardizes overfilling's ability to hedge cancellations. We want to fully leverage our network, and let it do the work rather than imposing artificial constraints. Synthetic shifts are invisible to HCFs, so we actually don’t really care if synthetic shifts get canceled and refilled a bunch of times. They cause no flak for HCFs, as per the “We just need someone to show up.” from Jane Doe.

No.

We will release documentation about “backup shifts” to our support personnel in the case that a healthcare professional shows up for a synthetic shift that was never merged. It will say that we create backup shifts on behalf of the HCF. They can send the worker home. The worker will be paid out of CBH’s pocket.

This is the SMS/Email copy for when a synthetic shift is merged:

“We found someone else for your shift on mm:dd hh:mm am/pm. {{name}} will be working that shift, and you can see their documents in the web app.”

Other than this, we’ll be able to keep most of overfilling on the backend and avoid having to explain to customers what we’re doing.

We understand that every cancellation that occurs on our marketplace causes stress, creates confusion, and ultimately impacts patients at your healthcare facility. We use marketplace policy to ensure that our workers are held to a high standard, and prevent bad actors from remaining on the platform. We also create backup shifts to make sure that if a worker does end up canceling a shift, we have a replacement for you immediately.

Clipboard health creates backup shifts to ensure that shifts you post on our platform remain filled, even if a worker cancels one of them. Sometimes we experience issues where a backup shift was filled, but no worker ever canceled. In this case, the worker assigned to the backup shift should have received a notification telling them their shift was canceled. In this case, there was likely a system malfunction.

You can send the worker home. They will be compensated for their time, and Clipboard will pay them out of our pocket so there’s no loss to you.

We handle backup shifts on our end so that there’s no hassle for you. Our algorithm is built to ensure that workers will show up and all of your shifts will get filled. We don’t share information about backup shifts to avoid potential confusion as they’re created or deleted by us.

You cannot; if you don’t want specific workers showing up to your facilities and are worried about them showing up through backup shifts then you can DNR them through the facility app or our support line. Backup shifts help ensure that when your shifts get filled, they stay filled. We don’t charge anything for creating backup shifts.

We don’t charge anything specific for backup shifts. They’re one of the advantages of staffing through our platform. The only charge you will be billed for is from shifts that are posted and filled.

We have reworked the way we deliver notifications to healthcare facilities. We received consistent feedback that there are a lot of unnecessary emails that cause confusion and clutter in inboxes. We’ve changed the notification experience to remove about 20% of the emails/SMS that we send to healthcare facilities which proved irrelevant.

Unfortunately, there isn’t much that we can do to control facility needs. In most cases, you are compensated 2 hours of pay for cancellations that occur within 24 hours of the shift’s start. We apologize for any inconvenience caused by these cancellations.

We are starting to introduce backup shifts into our marketplace. When workers cancel in shifts that were originally posted by healthcare facilities, we move nurses in backup shifts into the shifts that were canceled. This gives more work for nurses in places where it’s hard to find shifts. It also helps healthcare facilities know for sure that someone from clipboard will show up when they post shifts.

In cases like the one that you mentioned, we will pay you for 2 hours of your shift pay. We apologize for any inconvenience and we recommend that you turn on text notifications for cancellations.

To get an approximate view of the potential marketplace impact of overfilling on the slot level, I looked into how many slots are 100% filled at 4 days lead time, and how many slots are 100% filled at the time of the shift start (in our 40 highest volume markets with fill rates over 70%). I used the month of May 2022 as a proxy.

About 70% of all slots are fully filled at 4 days lead time, 32K slots. About 45% of all slots are fully filled at shift start, 21K slots. There’s a gap of about 11K slots, or 17K shifts (some slots experience multiple cancellations so there’s more than 1 shift that goes unfilled).

To do the back-of-the-envelope calculations, I assumed an upper bound where we overfill all of these slots, regardless of thresholding, and that all synthetic shifts get filled. I also used the mean pay and net revenue on shifts in those markets to do revenue calculations. We would make approximately an additional $1.4 million in net revenue for may (about 20% of net rev), and we would spend about $630K on synthetic FCFs. In this scenario, we would add about 21K FCFs. In reality, the number will be significantly lower because we will only overfill slots with low probabilities of being worked.

We will only run overfilling in oversupplied markets. We will use markets with a weekly shopping success average below 20% (i.e. 20% of HCPs who shop end up booking in a week), at least 250 shopping HCPs, and a fill rate above 70%. We can slowly relax fill rate and shopping success conditions to integrate more markets over time.

You can view the table here. In our rollout, we’ll start with Dallas CNAs to prove out the feature. Success will look like a 100% verification rate on available shifts (posted-FCFd). We will measure the increase in FCFs in the market, and talk to HCPs to see if they have registered a change in the marketplace (more shifts showing up but also more FCFs).

Take each worker’s historical work rate: ri = verified/(claimed - FCF’d). If they have fewer than 10 claims, substitute the market average.

For every shift in the slot multiply the work rate, W =  ∏ri

Then take probability of cancellation, C = 1-W

I used March to flag hoarders marketwide. Every time a hoarder claimed in April, I used the facilities fill rate history to compute the chance of a second claim. Obviously, there are more confounding factors (price, open shopping time, etc.) but this was the fastest path to gaining value from my time.

If no one additional picked up the shift, then the simulation’s outcome defaulted to the actual outcome of the shift. If someone did pick up the shift, then I picked a random non-hoarding worker using the likelihood of the user picking up a shift (based on their share of the total claims in April).

Once this user was selected I simulated the probability of either user in the shift canceling ahead of the 48-hour mark. If both were canceled I said the shift went unworked. If one canceled and one did not, I computed the probability of the remaining worker canceling under the 48-hour mark. If the remaining worker canceled the shift was simulated to be unworked. If it was not canceled, then I said that it was worked. If neither user was simulated to cancel above the 48-hour mark, I flipped a weighted coin based on their attendance record (verified / (claimed - FCF)). I FCF’d the user the coin did not pull for, then computed the probability of sub-48-hour cancellation for the other user to determine if the shift ended up being worked or not.

I ran the simulation 500 times, sampling 100 shifts in April each time. While there are definitely simplifications made by my simulation, it looks like we could bring the mean percentage of shifts worked for hoarder-claimed shifts up from about 60% to about 80%. We would also increase the number of shifts experiencing FCFs in this group, but we’d also be roughly doubling the number of “shifts” by creating synthetic shifts.

From first principles, overfilling on the slot level will be far more effective. We’ll be able to create synthetic shifts in slots with higher probability of cancellations than a hoarder may have. This will lead to fewer FCFs and more worked shifts.

If a >=2 depth slot is fully filled, and the HCF cancels a shift, we will not convert that shift into a synthetic shift. In some cases, HCFs cancel shifts as a DNR workaround. We don’t want to end up refilling the shift with a worker that HCFs don’t desire.

Once the ability to DNR is built into the web app we can convert a canceled shift into a synthetic shift if the slot meets the criteria for overfilling [p(cancel) > threshold].

For every market, we have a set of slots each with a probability of single cancellation {P0 , … , PN}. We want to set our threshold to create adequate extra demand via synthetic shifts while limiting the frequency of synthetic FCFs.

Let’s say that the set of P is ordered from highest to lowest probability of cancellation, so that P0 > P1 > … > PN . We want to pick Pi such that the i incremental shifts it adds does not create too many extra shifts for shopping supply.  Let’s assume an ideal ratio (k) of available shifts (a) to shopping nurses (S), k = S/a. Finally, let’s say that number of normal shifts available is n. We then have i = S/k - n . We can use that to pick Pi. There will be cases where i > N (number of filled slots). We will set Pmin in that case to be 50% so that there’s an even probability of a synthetic merge or a synthetic FCF.

Not only will this allow us to deliver overfilling information in a way that does not add to the clutter, but it will also drastically improve the HCF notification experience and reduce the “drama” around each shift. We will transfer this logic into our batched notifications as well i.e. the notifications that are batched follow the above logic.