An AI agent is ready to deploy when the team has defined which actions it may take alone, which require approval, and who owns that approval. Passing technical tests is insufficient when one disputed action can spend money, contact a customer, alter data, or create a commitment nobody intended.
On 26 September 1983, Lieutenant Colonel Stanislav Petrov was on duty at Serpukhov-15, a Soviet early-warning facility near Moscow, when the Oko satellite system reported a missile launch from the United States. It then indicated additional launches.
The system had done what it was built to do: detect a possible attack and produce an alert. The unresolved question sat one level higher. Should the machine’s output trigger a human report that could move the Soviet command system closer to retaliation?
Petrov judged the warning to be false. A real first strike involving so few missiles did not make sense to him, and ground radar had not confirmed the satellite warning. He reported a system malfunction. The alert was later traced to sunlight reflecting from high-altitude clouds.
David E. Hoffman documents the incident in The Dead Hand. Petrov’s decision is often remembered as an individual act of restraint, but it also exposed a system-design problem: the technology could detect a signal, while a person still had to decide what that signal meant and what should happen next.
Technical readiness leaves a policy gap
I have seen deployment reviews spend most of their time on the agent’s accuracy, latency, tool access, and recovery behaviour. Those checks matter. They can still leave the most dangerous question unanswered.
Suppose an agent reviews supplier requests. Everyone agrees it can extract line items, compare quotes, and draft a recommendation. Then someone asks whether it can send the selected supplier a purchase confirmation.
The engineer says the agent should proceed when confidence exceeds the agreed threshold. Operations says every purchase needs approval. The founder says small purchases should go through automatically, but cannot define “small” without considering the supplier, currency, delivery date, and customer commitment.
The agent has passed its tests. The company has failed to state its decision policy.
This is where teams confuse model confidence with permission. Confidence describes how strongly the system supports an interpretation. Permission describes whether the company accepts the consequences of acting on it. A 98 percent confidence score cannot authorize a bank transfer, cancel a customer account, or accept a contractual term.
Map actions by consequence, not by feature
The useful unit in a deployment review is the action.
“Email access” is too broad. Drafting an email, sending an internal update, replying to a known customer, and committing to a delivery date carry different consequences. Giving all four actions one approval rule hides the actual risk.
I would put each proposed action into one of three groups:
- The agent may perform reversible, low-consequence actions and record what it did.
- The agent must request approval before actions that move money, change permissions, contact an external party, delete records, or create a commitment.
- The agent may never perform actions whose legal, safety, or commercial consequences the team cannot reliably evaluate.
The boundaries will differ between a two-person SaaS company in Accra, an automation engagement in Berlin, and a startup selling into the United States. The method stays useful because it forces the team to discuss consequences in its own operating context.
The approval step also needs a named owner. “A human reviews it” sounds safe until a request waits in a shared inbox or two people assume the other person checked it. Name the role, define the evidence shown to that reviewer, and state what happens when nobody responds.
That was the practical strength of the decision described in AI Agent Launch Decisions: Why Kwame Shipped Berlin With Human Approval. Human approval became part of the operating design, with a specific reason for being there.
Test disagreement before deployment
A good review should include cases where reasonable people disagree.
Ask the agent to handle a duplicate payment attempt, an urgent customer request with missing context, or a supplier change that improves price but delays delivery. Do not stop after checking whether it selects the expected answer. Ask who may authorize the resulting action and what information that person needs.
The disagreement is useful. It reveals where the company has been relying on unwritten judgment.
Write the decision down in operational language. For example: the agent may prepare a refund, but a person must approve it before funds move. It may draft a reply, but it cannot promise a delivery date absent confirmed capacity. It may flag an account for review, but it cannot suspend access.
Then test failure paths. What happens if the approval service is unavailable? Does the agent pause, retry, choose a safer action, or continue? A fallback that quietly expands the agent’s authority defeats the approval boundary.
The unexplained payment retry shows why this detail matters. One technically plausible retry can become a duplicate charge, and the customer experiences the consequence rather than the confidence score behind it.
Make the boundary part of the product
Petrov stood between an alert and an escalation because the system still depended on human interpretation at the consequential moment. An AI agent needs the same boundary made explicit before launch, without relying on one person to improvise it under pressure.
End the deployment review with an action register. For every tool the agent can call, record the permitted action, prohibited action, approval owner, evidence required, timeout behaviour, and audit record. Any row the team cannot agree on stays disabled.
That unresolved row is not evidence that the agent needs another model evaluation. It is evidence that the humans still have a decision to make.
Comments
No comments yet.