A perfect AI demo can still be unsafe to ship when nobody owns the action that follows its recommendation. Until one person has clear authority to accept, reject, or investigate the output, delaying the release protects the customer and the product.
The model had passed every test by Friday afternoon. At 4:38 p.m. in Accra, Kwame, a composite founder, stood beside a meeting-room screen with his laptop charger looped around one wrist. The demo read a supplier request, compared the available options, and recommended which order to place. Every test case returned the expected answer.
Then I asked, “Who places the order?”
The question the tests had missed
Kwame said the purchasing officer would handle it. The product lead thought the finance manager would approve it first. The engineer assumed the system would create the order automatically once the confidence score cleared the agreed threshold.
Three answers. One next step.
The difference mattered because Monday’s demo was meant to run against a live workflow. If the AI recommended the wrong supplier, who would catch it? If the recommendation was correct but nobody acted, who would notice the delay? If two people believed the other had approved the purchase, the company could place the same order twice.
The model tests could not answer any of those questions. They measured extraction, ranking, and response quality. They did not measure ownership.
Kwame had already told the team the demo was ready. A delay would mean another uncomfortable call and another week without the proof he hoped would unlock a commercial conversation. Shipping carried a different risk: one ambiguous recommendation could become a real purchasing decision with no accountable person behind it.
For a few minutes, either choice looked expensive.
A correct answer can still create a bad outcome
AI product teams often focus on whether the model produces the right output. That is necessary, but the customer experiences the workflow around the output.
A purchasing recommendation has no value while it sits in a dashboard. Someone must review it, decide what to do, and remain responsible when the available evidence is incomplete. The moment an AI system enters an operational process, its output becomes part of a chain of human decisions.
That chain needs a named owner.
This is where a polished demo can hide danger. The generated answer looks complete, so everyone assumes the work is complete. Yet the real work may begin after the answer appears: checking an unusual price, confirming stock, asking why a familiar supplier was excluded, or deciding whether the purchase can wait.
I had seen the same boundary in a related AI purchasing assistant demo. Human approval mattered because the system could help someone decide without quietly inheriting authority nobody had agreed to give it.
On that Friday, the unresolved issue was even more basic. We had built the recommendation before assigning the decision.
We changed the workflow before touching the model
I stopped the release.
We did not spend the next week tuning prompts or adding more test cases. First, we drew the next step on the whiteboard.
The AI would prepare a recommendation and show the evidence behind it. One named purchasing owner would review the recommendation. That person could approve it, reject it, or send it back with a reason. Nothing would move forward automatically. If no action happened, the recommendation would remain visibly pending rather than disappearing into a list of completed model runs.
That decision narrowed the product. It also made the product safer to evaluate.
Kwame’s concern was that a human approval step would make the demo feel less impressive. The opposite happened. The revised flow made the product easier to explain because every person in the room could see where the AI’s job ended and human responsibility began.
This is close to another product problem I have written about: pausing the roadmap when nobody can explain what the company does. In both cases, more building would have deepened the confusion. The useful move was to stop and make the responsibility legible.
Put a name beside every consequential step
Before shipping an AI feature, write down what happens immediately after the output appears. Avoid role-shaped answers such as “operations handles it” or “finance checks it.” Name the person who owns the next action in the first deployment.
Then ask what happens if that person does nothing.
Does the request expire? Does somebody receive an alert? Can another person take over? Does the system act without approval? A workflow has not been designed until silence has a defined consequence.
Finally, test disagreement rather than accuracy alone. Give the reviewer a recommendation they believe is wrong. Watch whether they can inspect the evidence, reject the answer, and leave a reason that the next person can understand. A green model score does not prove that the surrounding decision can survive conflict.
By the following session, Kwame was no longer presenting a machine that appeared to make purchases. He was showing a purchasing owner a recommendation, the evidence behind it, and a clear decision waiting for their name.
The demo looked less magical.
It was finally ready to become real.
Comments
No comments yet.