I would hold the approval until the team defines the decisions Ama’s avatar can make, the decisions it must escalate, and the actions it can never take. A convincing likeness increases the cost of ambiguity because customers may treat every answer as Ama’s own judgment.
On 26 September 1983, Stanislav Petrov was the duty officer at Serpukhov-15, a Soviet early-warning facility near Moscow, when the system reported an incoming American missile. It then reported four more.
The alert carried the authority of machinery built for one of the highest-stakes decisions imaginable. Petrov still had to decide whether the warning described reality.
The system produced an answer, not a decision
Petrov did not pass the alert up as a confirmed attack. He judged it a false alarm, partly because a real first strike involving only a handful of missiles did not make sense to him. Later investigation found that sunlight reflected from high-altitude clouds had confused the satellite system.
The BBC’s account of the incident describes the uncomfortable part clearly: Petrov had minutes to judge whether the system was wrong, while knowing what escalation could follow if he was wrong.
The parallel with an AI avatar has smaller stakes, but the mechanism is the same. A system can produce a confident signal without carrying responsibility for what happens next. Someone still needs to define where machine output ends and accountable judgment begins.
Ama’s team had started with resemblance. Did the face look right? Did the pauses sound natural? Would someone who knew her recognize the voice?
Those questions matter for presentation. They say little about authority.
If the avatar tells a prospect that a feature will ship next month, has Ama made that commitment? If it recommends a price concession, can the sales team honour it? If a customer describes a security incident, does the avatar keep talking, collect details, or move the conversation to a named person?
Until those boundaries exist, approving the likeness approves the most persuasive part of the system before defining the safest part.
Write the refusal list before polishing the face
I would ask the team to stop the visual review and write a refusal list.
The first version can fit on one page. It should name the decisions the avatar cannot make under any wording or customer pressure. Those might include changing prices, promising delivery dates, accepting contract terms, discussing private employee matters, giving legal or financial advice, or confirming how the company will respond to a security incident.
The exact list depends on the business. The structure does not.
Each prohibited decision needs a next step. “I cannot answer that” leaves the customer stranded and invites them to rephrase the request. “I need Ama or the account lead to confirm that commitment” states the boundary and names the route forward.
This is where product work becomes more demanding than demo work. The team must translate Ama’s judgment into rules that can survive an impatient prospect, an unusual request, and a conversation nobody rehearsed.
The same issue appears in The Ten Costly Decisions a Useful AI Clone Needs Before Taking Every Call. Coverage feels valuable until the clone reaches a decision whose cost exceeds the value of answering quickly.
Test authority, not resemblance
A normal avatar review rewards familiarity. The smile looks like Ama’s. The voice cadence feels close. The demo lands.
A useful approval test should try to make the avatar overreach.
Give it a prospect who wants an exception before signing. Ask for a delivery promise the roadmap cannot support. Introduce a complaint that requires access to customer records. Repeat the request using softer language. Then try again with urgency: payroll is due, the board meeting is tomorrow, or the contract disappears today.
The test passes when the avatar refuses consistently, explains the limit plainly, and hands the decision to the right person with enough context to continue. A polished video response that crosses a commercial or safety boundary still fails.
I would also record every escalation during the test. Repeated escalations may reveal a boundary that is too restrictive, a missing internal policy, or a question the company has avoided deciding. That evidence is more useful than a room agreeing that the avatar feels realistic.
When a system’s reasoning remains unclear, restraint deserves the same attention as capability. I made a similar argument in Should We Merge an AI Security Fix We Cannot Yet Explain?: a technically successful output does not remove the need to understand its failure conditions.
Ama’s face should be the last approval
Petrov’s decision in 1983 depended on a boundary the warning system could not enforce for itself. The machine detected and reported. A human judged whether the signal justified escalation.
Ama’s avatar needs that division written down before her likeness gives its answers borrowed authority.
On Monday, I would put three documents beside the demo: the prohibited-decision list, the escalation routes, and the transcript from adversarial tests. Then I would ask the team to approve those before reviewing the face again.
Comments
No comments yet.