A working AI demo proves that one model produced a useful answer under one set of conditions. Before treating it as product evidence, test whether the answer survives a model change, repeated runs, and the edge cases your customer cares about.
On 1 August 2012, Knight Capital deployed new trading software across its servers. Seven received the new code. One retained older code, and when the system went live, that difference triggered unintended orders. The company lost about $460 million in roughly 45 minutes.
The hidden variable inside a successful run
Knight’s software had passed through a deployment process that appeared complete. The eighth server made that appearance misleading.
The US Securities and Exchange Commission documented the incident in its 2013 order against Knight Capital Americas. Thomas Joyce, Knight Capital Group’s chairman and CEO at the time, later described the damage publicly. The company survived, but only after securing emergency financing that diluted existing shareholders and changed its future.
The useful analogy for an AI founder sits inside the mismatch. Seven servers behaved one way. The eighth behaved another. A system that looked consistent contained an operational difference large enough to determine the outcome.
Changing an AI model can expose the same kind of hidden dependency.
Imagine the demo is ready on Friday afternoon, a few hours before a customer call. The workflow takes an uploaded purchasing request, extracts the relevant constraints, compares suppliers, and recommends an option. The result finally reads like something a buyer could use.
Then the founder changes the underlying model.
The recommendation changes.
Perhaps the first model prioritised delivery risk while the second gave more weight to price. Perhaps one treated an ambiguous sentence as a firm constraint and the other treated it as background. Both outputs sound plausible. Only one matches the customer’s actual decision.
The question has changed. It is no longer, “Does the demo work?” It becomes, “What exactly did this demo prove?”
Separate the product decision from the model’s wording
A convincing answer can hide a weak product contract.
If the product promises to summarise a document, variation may be acceptable. If it promises to approve a supplier, calculate a closing balance, or identify a compliance problem, unexplained variation carries a different cost. The founder needs to define which parts may vary and which must remain stable.
I would freeze the Friday build before touching the prompt again. Save the input, output, model version, system instructions, retrieval results, and any tool responses. That gives you a reproducible case instead of a memory that “it worked earlier.”
Next, run the same case several times on both models. Do not judge the outputs by polish. Compare the decision-bearing facts:
- Did each model identify the same constraints?
- Did it use the same evidence?
- Did it reach the same recommendation?
- Could a customer see why the recommendation changed?
- Would a wrong answer create inconvenience, financial loss, or an irreversible action?
This is where human approval becomes a product boundary rather than a temporary patch. If the system cannot yet explain a consequential difference, it should prepare the decision for a person instead of making the decision itself. That is the same reason Kofi added human approval to an AI purchasing assistant demo.
Decide what the customer should see
A few hours before the call, there are three honest choices.
You can demonstrate the narrow path that works and name its limits. You can show the variation and use it to ask how the customer resolves that decision today. Or you can postpone the recommendation feature and demonstrate the stable parts, such as extraction, evidence gathering, or comparison.
The wrong move is to keep switching models until one restores the answer you wanted. That turns model selection into answer shopping.
For a customer evaluating an early product, visible uncertainty can be more credible than a smooth result. Show the source material beside the recommendation. Mark the fields that came from deterministic rules. Put approval before any action that changes money, inventory, access, or customer records.
This does not weaken the demo. It tells the customer where the software ends and judgment begins.
The current usage pattern points in the same direction. In the supplied survey context, 81% of 1,053 respondents use multiple models, 75% adjust AI use based on cost, and human approvals are the most common control around agents. Model choice already moves in real products. The control layer has to assume it will.
Turn Friday’s surprise into a test
Before the customer call, write down one sentence:
“This demo succeeds if the system consistently identifies X, shows evidence Y, and asks for approval before Z.”
Then test that sentence against both models.
If both pass, the demo has shown something useful about the product. If only one passes, you have found a model dependency that needs to be disclosed or contained. If neither passes, the polished answer was luck.
Knight Capital’s eighth server mattered because the company’s deployment process did not establish that every machine held the intended state. Your AI demo needs the equivalent check. Establish which model ran, which evidence it used, which answer may vary, and which action remains blocked until a person agrees.
Save Friday’s good output. Then try to break it before the customer does.
Comments
No comments yet.