An agent should never write to customer production data without a narrow permission boundary, a reversible path, and a human decision for consequential changes. When it does, the first job is to stop further writes, establish exactly what changed, and speak to the customer before they discover it themselves.
The moment the change appeared
At 5:43 p.m. on a Friday, Imani was at a small table in a Berlin coworking space, packing her laptop before a weekend release. Her agent had been built to read customer support notes, suggest account updates, and prepare drafts for an operations team. A notification caught her eye: an account record had changed status.
Then another one appeared.
The agent had followed a tool path meant for an internal test environment and written to a live customer record. It had changed an active account’s routing field, which meant the customer’s team could open their dashboard on Monday and find work assigned to the wrong queue.
Imani had a choice that felt smaller than it was. She could patch the affected record, switch off the agent, and hope no one noticed. Or she could pause the release, trace every action, and call the customer before the weekend began.
The first option protected the release schedule. The second protected the relationship.
For ten minutes, she did neither. She read the logs while the office emptied around her. The problem was worse than one field. The agent had been given permission to act after interpreting a support message, and its confidence had been treated as sufficient evidence. There was no approval step between “I think this should change” and “this customer’s live data is now different.”
The Monday release was at risk. So was the customer’s trust.
Production access is a product decision
Teams often treat production access as an engineering detail that can be solved with a cleaner API key or a better prompt. It is a product decision because the customer carries the consequence.
An agent that drafts an email and gets the tone wrong creates an awkward review. An agent that changes a billing status, routes a support case, deletes a record, or updates a customer profile can create a problem that the customer has to untangle under pressure.
The useful question is not whether an agent can take an action. It is whether the action can be safely wrong.
Imani’s team had focused on a familiar demo path: support note arrives, agent interprets it, account is updated. It looked convincing. It also skipped the difficult part, deciding which updates deserved automation and which ones required a person who understood the account history.
That distinction matters most when runway is tight. A founder can feel pressure to show the full loop working because the partial loop looks unfinished. But a production agent with broad access is not a stronger demo. It is an unpriced risk.
This is close to the accountability problem buyers raise before they agree to an AI demo. What Happens When Buyers Need Accountability Before an AI Demo? explores the commercial version of that question. Friday afternoon reveals the operational version.
The release paused for a reason
Imani paused the release and disabled the write action. Then the team did three less glamorous things.
They identified every record the agent had touched, compared each change with the source message, and restored the records that did not have clear human approval behind them. They separated read access from write access. They also changed the workflow so the agent could prepare a proposed update with its reasoning and source, while a person approved any customer-facing change.
That last part felt like a retreat at first. The agent was supposed to reduce operations work. Now someone still had to review a queue.
But the review queue made the product more useful. It exposed where the agent was reliable, where customer language was ambiguous, and where the team’s own account rules were too loosely defined. “Move this to the priority queue” meant one thing for a new trial account and another for a customer with an unresolved issue.
By Monday morning, Imani had a short, factual explanation ready for the affected customer: what changed, what had been restored, what the team had changed to prevent a repeat, and who they could contact. There was no attempt to make the incident sound clever or minor.
The customer relationship had not been lost. The release had changed shape.
Give the agent a smaller first job
The safer route is to earn production autonomy in stages. Start with an agent that observes, summarizes, classifies, or drafts. Make its evidence visible. Put it beside a human workflow long enough to learn where it fails.
Then define narrow write permissions around actions that are easy to reverse and easy to audit. A customer note can become a suggested tag. A suggested tag can become an approved update. The agent should not gain permission to alter account state simply because it performed well on a set of examples.
Imani returned to the same coworking table the following Friday with a different release candidate. The agent could still read the support note and propose the account change. Beside each proposal sat the source text, the action it wanted to take, and an approval button.
It was less dramatic than an agent writing directly to production.
It was ready to be used.
Comments
No comments yet.