An AI-generated pull request that passes the test suite is ready for review, not automatically ready for customers. Tests can confirm the cases you anticipated; Monday’s users will bring the cases you did not.
At 4:47 PM, Kwaku was standing beside a desk in Osu, laptop open, helmet still clipped to his backpack. The AI-generated pull request had cleared the suite after two earlier failures, and the small green check beside it had changed the room’s mood.
The patch handled a customer intake step that had been blocking a Monday rollout. One developer had written the prompt, reviewed the suggested changes, and pushed the branch. The tests passed. The office was already thinning out.
Kwaku could merge it, tell the pilot customer the feature was ready, and spend the weekend on the next problem. Or he could hold the release and risk looking slow after a week of saying Monday mattered.
The bad ending was ordinary and expensive: a customer opens the workflow on Monday morning, enters data the team never used in testing, and gets the wrong result with no clear way back. A green build would then become evidence that the team had moved too quickly.
A passing test suite only reflects the questions you asked
The pull request had tests for the happy path, blank fields, and a few malformed inputs. It did not have a test for a repeated submission from a weak connection. It did not test an account with old records created before the new workflow existed. It did not test what a customer should see when the model could not produce a useful answer.
Those gaps are easy to miss when the generated code is legible and the tests are green. AI makes it faster to produce both implementation and confidence. The second part is the risk.
A test suite is a map of the product team’s current imagination. It tells you whether the code behaves in the scenarios someone wrote down. It cannot certify the workflow surrounding the code: who owns a bad output, what data reached the model, what the customer can undo, or how support will reconstruct a confusing decision.
That distinction matters most when a small team is under runway pressure. The temptation is understandable. The code arrived quickly, the deadline is real, and a manual review can feel like choosing delay over progress. But a customer-facing AI feature has a wider surface area than the diff.
The review starts where the generated patch ends
Kwaku pulled the pull request into a call with the developer instead of merging it. They stopped reading it as code for a few minutes and traced one Monday customer through it.
What enters the workflow? Which fields are sent to the model? What happens if the answer is empty, wrong, or delayed? Can the customer correct it without creating another bad record? Which person sees the failure first?
The important question was not whether the AI had written clean code. It was whether the team could explain the resulting customer decision on Tuesday.
That is where practical review differs from asking an AI tool to produce another explanation of its own output. The team needs checks outside the generated patch:
- Run the workflow with realistic old records, incomplete records, duplicates, and inputs that should be rejected.
- Read every customer-visible message, especially the error states and any text that implies certainty.
- Confirm the rollback path before release, including who can disable the feature and what happens to records already created.
- Name the person responsible for investigating a bad result on Monday.
These checks are especially important when AI has lowered the cost of creating code. It also lowers the cost for attackers to inspect weak assumptions and generate ways to exploit them. Treat generated output as untrusted until a human has connected it to the actual data, permissions, and failure modes of the product.
An earlier launch-delay story about data access and accountability reaches the same uncomfortable point from the other side: a feature can work technically while nobody owns the decision that makes it safe to use.
Monday needs an owner, a boundary, and a way back
By 5:26 PM, Kwaku had not merged the full change. The team cut the release to a narrower group, added a visible fallback for uncertain results, and wrote down who would watch the first customer runs on Monday.
It was less satisfying than a clean Friday merge. It was also a real release decision, rather than a test result wearing the clothes of one.
The first Monday customer in this composite scene should not become a test case without knowing it. If the feature changes a record, recommends an action, or influences a business decision, the customer needs a clear boundary around what it can do and a way to correct the outcome.
For founders, the useful habit is small: before you merge an AI-generated pull request, name the customer action it can affect, the failure that would matter most, and the person who will own that failure. If those answers are vague, the green check has arrived before the product is ready.
The green check can wait five more minutes
Kwaku left the office with the release still staged, not merged. On Monday, he would open the dashboard before the first customer message arrived, with a narrower feature live and a short list of signals the team had agreed to watch.
That is the work a test suite cannot do for you. It turns a passing build into a release the team can stand behind.
Comments
No comments yet.