A signed agreement does not make an AI agent ready for a customer’s workflow. Before the agent takes its first irreversible action, test that exact action with the customer’s permissions, exceptions, approval path and recovery process.
On August 1, 2012, Knight Capital Group released software that began sending unintended orders into US equity markets. The company could not immediately stop the activity. By the time it did, roughly 45 minutes later, Knight had lost about $440 million.
The US Securities and Exchange Commission’s 2013 order documents the failure in unusually useful detail. Knight had deployed new code across its servers, but one server did not receive the update. A reused software flag activated old code on that machine. The system then entered the market with real authority and real money behind its actions.
Knight’s chairman and CEO at the time, Thomas Joyce, had spent years building the firm into a major trading operation. That history offered no protection once a production system began acting outside the workflow its operators believed they had deployed.
The contract confirms intent, not operating safety
For an early-stage founder, the 4:47 signature can feel like the hard part ending.
The customer agreed. Procurement cleared the document. Someone on the team may already be preparing the invoice. If runway is short in Accra, Lagos, Berlin or New York, the pressure to activate the system before close of business is understandable.
Then one question changes the mood: what is the first thing the agent can do that cannot be cleanly reversed?
Perhaps it sends a customer email, changes a record used for billing, approves a refund, updates inventory, books an appointment or triggers a payment. The demo may have completed that action many times. The internal test account may look fine. Neither proves that the agent can operate inside this customer’s actual sequence of handoffs.
A contract establishes what both parties intend to buy and deliver. It does not reveal that the finance manager sometimes holds an invoice while checking a WhatsApp message, or that two people share responsibility for approving a refund, or that a “closed” support case may be reopened when a customer replies by email.
Those details are the workflow.
Test the authority boundary first
Teams often test whether an agent can produce the right answer. The more important launch test is whether it has the right to act when the answer is uncertain.
I would define the first irreversible action in one sentence, then run it through the customer’s real process using controlled data. The customer should be able to see the proposed action, the information the agent used, who approved it and what happens when a required field is missing.
The test should include at least one ordinary case and one exception the customer sees in practice. If the agent creates invoices, test the account with incomplete billing details. If it sends messages, test a contact who has opted out. If it changes an order, test the point where warehouse staff have already started work.
This is where deployment confidence becomes specific. “The agent works” is too broad to verify. “The agent cannot issue a refund above the customer’s chosen threshold without approval” can be tested.
The same principle appeared in a different form when Kweku encountered a changed liability clause. Capability matters, but authority determines the size of the mistake a system can make.
Build the stop before allowing the start
Knight Capital’s problem continued for minutes because stopping the activity was harder than starting it. That is the part AI product teams should carry into customer deployment.
Before activation, identify one person who can pause the agent. Give that person a visible control. Decide what gets preserved when the pause happens: pending actions, logs, source data and approval history. Then rehearse the stop with the same seriousness as the successful path.
A rollback may not undo an email already sent or a payment already released. For those actions, the safer design is staged authority. The agent prepares the action first. A human approves it. Only after the system behaves correctly in the customer’s workflow should the team consider removing that approval.
This can feel slower when a founder needs the contract to become revenue. It is usually faster than investigating a production action across the customer’s CRM, inbox, accounting records and internal conversations.
The most valuable launch evidence is rarely another clean demo. It is watching the system meet a messy exception, refuse to guess and leave the customer in control.
What I would do at 4:48
I would acknowledge the signed agreement and resist activating the final permission.
Then I would book a short workflow rehearsal with the person closest to the action, not only the executive who signed. We would identify the first irreversible step, run one real-world exception with controlled data and confirm that the pause control works.
If any of those checks fail, the agent stays in draft or approval mode. The customer can still begin using it, the team can still learn and the contract can still move forward. What changes is the amount of authority placed behind an untested assumption.
Knight Capital had software capable of acting at market speed. One deployment inconsistency turned that capability into exposure. At 4:47, the signature is good news. At 4:48, test the boundary before the agent crosses it.
Comments
No comments yet.