A convincing AI demo proves that one prepared path works. A dependable AI product handles unfamiliar inputs, missing context and awkward customer behaviour without quietly producing the wrong result.
On 27 January 1986, engineer Roger Boisjoly joined a conference call that could still have changed what happened next. The Space Shuttle Challenger was due to launch from Kennedy Space Center the following morning, and unusually cold conditions raised concerns about the solid rocket booster O-rings.
Boisjoly and other Morton Thiokol engineers had evidence that the seals performed poorly in cold weather. NASA managers asked for a recommendation. Morton Thiokol initially advised against launching below the temperature supported by its flight data, then reversed that position after an internal management discussion.
Challenger launched on 28 January. Seventy-three seconds later, it broke apart, killing all seven crew members.
The Rogers Commission documented the decision, the disputed evidence and the failure. Its report remains available through NASA’s history archive.
The prepared path creates false confidence
An AI feature can pass every rehearsal and still fail its first serious customer test.
The demo prompts are usually written by people who understand the system. They use the expected vocabulary, provide enough context and avoid combinations already known to cause trouble. Even the difficult examples have often been seen before. The model is walking through a room whose furniture the team arranged.
Then the first customer arrives.
They paste a partial email thread instead of completing the form. They refer to a company by an old trading name. They ask one question that contains three separate decisions. They assume the product remembers something entered elsewhere. Their spelling varies. Their account data contains a duplicate record created months ago.
The feature returns a polished answer. That polish makes the failure harder to detect.
A broken button announces itself. An AI response can be grammatically clean, structurally convincing and operationally wrong. The customer may act on it before anyone knows the product crossed from uncertainty into error.
This is the same boundary the Challenger decision exposed at far higher stakes: previous operation inside a known range did not establish reliability outside that range. Evidence from rehearsed conditions cannot answer a question about conditions the rehearsal excluded.
Test the boundary, not the script
I would ask one question before approving an AI release: what assumption makes the demonstration look better than ordinary use will?
Perhaps every test includes complete source material. Remove a page.
Perhaps customer names always match the database. Introduce a trading name, a spelling variation and two customers with similar names.
Perhaps the model always receives one instruction. Give it a request with conflicting goals.
Perhaps the expected answer exists in the documents. Ask a question the evidence cannot resolve, then check whether the system admits that or fills the gap.
The point is to find the edge where behaviour changes. A useful test suite should contain incomplete inputs, contradictory records, duplicates, ambiguous instructions and requests that require refusal or escalation. It should also check the action surrounding the model. A correct response attached to the wrong customer remains a product failure.
This is why Ama’s identity question matters. Identity, record matching and context boundaries often look secondary during a demonstration. In production, they determine whether a plausible answer belongs to the right person.
Decide what failure is allowed to do
Small teams rarely have enough runway to eliminate every model error. They can still decide which errors may reach a customer and what happens when confidence falls.
A writing assistant can offer a weak paragraph and let the user revise it. A system that changes a payment status, sends a customer message or recommends an operational decision needs a narrower permission boundary.
That boundary can be simple. Require confirmation before an external action. Show the source used for a consequential answer. Route unresolved identity matches to a person. Record the prompt, retrieved context, output and final action so the team can reconstruct what happened.
The important distinction is consequence. Teams often group failures by technical cause: retrieval, classification, extraction or generation. Customers experience them by outcome: the wrong invoice was approved, the wrong person received the message, or a deadline was missed.
Two settlement totals that do not match should stop a workflow before the system chooses whichever figure looks more likely. Uncertainty must change product behaviour.
Make Friday’s customer boring
The first customer session should reveal small usability problems, not a category of failure the demo concealed.
Before Friday, give the feature to someone who did not help build it. Do not explain the preferred phrasing. Give them a real task, then watch where their assumptions differ from the system’s assumptions. Keep every failed attempt, especially the ones that produce confident answers.
For each failure, choose one response: constrain the input, improve the underlying data, add a deterministic check, request confirmation, escalate to a person or remove the capability from the release.
The Rogers Commission did not describe an absence of data. It described evidence that failed to control the decision. AI teams face a smaller but recognisable version of that problem whenever a smooth demonstration outweighs an unresolved production risk.
Before the customer arrives, run the test your demo has been quietly avoiding.
Comments
No comments yet.