Who Has Time to Review?

Human review can become the bottleneck in a product that promises to save time.

By Ismail Jai Hokimi · September 23, 2026

An AI product that handles 90% of a job could be well worth buying. I don’t want to lose sight of that when discussing everything that might go wrong. My concern is with the other 10%, especially when the proposal assumes that someone at the customer will take care of it alongside their existing work.

Here is a hypothetical example. A team receives 100 cases an hour and the software refers 10% for review. Each review takes eight minutes. That leaves 80 minutes of work for a person to do every hour, before they have answered an email or dealt with anything else. There is no way to make that arrangement work for one person at the proposed volume.

Hiring another person might still make this a good purchase. It would change the price, though, and I would want that in the proposal. I have a harder time with an answer along the lines of “the team will adjust.” What exactly are they supposed to adjust? If the work has to be checked, doing it more quickly is only useful while the check remains good enough.

I’d cap this launch at 60 incoming cases an hour. That is what one person can support with 48 minutes available for review and eight minutes per case. The customer wants to send 100, so there is still a staffing decision to make. A second qualified reviewer with the same allocation brings capacity to 120. Agree who covers absences and excess work, then try the arrangement at the expected volume before making the old process unavailable.

There is something useful for the founder to investigate here. Suppose two of the eight minutes are spent hunting for the supporting record. Putting it beside the answer might bring review time down to six minutes, if a trial bears that out. One reviewer could then support 80 incoming cases an hour. That still falls short of 100, but it is a credible improvement to discuss with a buyer. I’d be more interested in seeing that demonstrated than in shaving another second off the initial answer.

The appropriate check also depends on what happens next. For an internal draft based on public information, later sampling may be enough. For a payment requiring approval, the payment waits and the reviewer must be able to stop it. I wouldn’t infer much from the approval rate alone: approving nearly everything might mean the outputs are good. A controlled test with known errors would tell us more about whether the person is actually catching problems.

This is part of how I’d assess the founder, too. Can they explain what the customer needs to change, including a change that makes the sale harder? I’d rather hear about the extra hire while there is still time to decide whether the product is worth it.

Further reading: NIST’s Generative AI Profile, MAP 3.4, on operator proficiency and human review. The staffing assumptions above are hypothetical.