A demo can return the expected answer for the wrong reason. Before you approve the model, remove every input it will lose in production and test whether the answer still holds.
At NASA’s Jet Propulsion Laboratory in Pasadena on September 23, 1999, engineers waited for the Mars Climate Orbiter to reappear from behind Mars. The spacecraft had completed its journey from Earth. The team expected to regain its signal.
They never did.
The result looked right until the system changed
NASA’s investigation traced the loss to a mismatch between two systems. Software supplied by Lockheed Martin produced thruster data in pound-seconds. Navigation software at JPL expected newton-seconds.
The numbers moved through the process. The spacecraft followed commands. Nothing about receiving a number guarantees that the number means what the next system thinks it means.
The Mars Climate Orbiter Mission Failure Investigation Board documented the mismatch in its 1999 Phase I report. Arthur Stephenson chaired the board. Its finding became a durable engineering warning because the failure sat at the boundary between components, hidden behind outputs that looked usable until the spacecraft reached Mars.
The scale is different, but the mechanism is familiar to anyone building an AI product.
It is Friday at 4:47 PM. The customer asks the question they used in Monday’s discovery call. The model returns the expected answer. The customer recognises it immediately. The founder sees the green light: the workflow works.
Then one person asks which fields the model used.
The strongest signal came from a stray column in the demo export. Perhaps it was an internal status label, a manually corrected category, or a note added by the customer’s operations team. The production integration will not include it.
The model passed because the test data quietly contained part of the answer.
A correct answer can still fail validation
A demo usually tests whether the product can produce a convincing result. Production asks a harsher question: can it produce that result using only the information that will actually exist at decision time?
Those are separate tests.
Early-stage teams miss the distinction because the expected answer feels like proof. It brings relief after days of cleaning data, adjusting prompts and wiring the interface. When runway is limited and a pilot depends on Friday’s call, there is a strong incentive to treat recognition from the customer as validation.
I would pause there.
First, record the exact fields available when the live system must make the decision. Remove everything created after that decision, everything maintained only for the demo, and everything supplied through a manual process nobody has agreed to continue.
Then run the same examples again.
If performance collapses, the experiment has still produced something useful. You have identified the information the model needs, and you now know production does not provide it. That is a product decision, not a prompt adjustment.
You can change the workflow to capture the missing field earlier. You can find a weaker but available signal. You can narrow the claim the product makes. You can also stop the pilot before committing engineering time to an integration built around unavailable data.
That may feel slower on Friday. It is cheaper than discovering the dependency after the customer has started relying on the output.
Test the information path, not the prepared dataset
The fastest check is an input audit conducted before celebrating the answer.
For each field, ask when it is created, who creates it, and whether the production system can access it at the moment of prediction. A customer may own the data without exposing it through the integration. A field may exist in historical exports because someone corrected it later. A clean label may represent weeks of manual work that will not happen for new records.
Next, run an ablation test. Remove one field or group of fields at a time and compare the model’s output. You do not need a complex evaluation platform to catch the obvious dependency. A small set of representative cases can show whether the model understands the underlying pattern or follows one convenient shortcut.
Finally, include examples where the shortcut points in the wrong direction. If a status field usually correlates with the answer, create or find a legitimate case where it does not. The goal is to learn what the system relies on before a customer does.
This is closely related to the source-of-truth problem in what the third broken build taught Kwame about product design handoff. A model can only be as dependable as the information contract around it.
The next decision belongs in the test plan
The Mars Climate Orbiter was lost because two teams attached different meanings to data crossing a system boundary. Your demo faces the same class of risk whenever a model receives information from an export, spreadsheet, CRM or manually prepared file.
Before the next customer call, write down the production input contract. List only the fields the live system will receive, at the time it must answer, in the form the integration will supply them.
At 4:47 PM, the customer’s approval can wait ten minutes. Delete the suspicious field. Run the case again. The second result is the one that decides what you have built.
Comments
No comments yet.