A payment provider outage becomes existential when the team has no tested way to complete an order without its API. The immediate problem is lost checkout volume; the deeper problem is that one external service has quietly become the company’s entire operating procedure.
At 9:30 a.m. on August 1, 2012, Knight Capital began sending erroneous orders into US equity markets. Its engineers had deployed new trading software to seven servers, but one of eight servers still held dormant code from an older system called Power Peg. When the market opened, that code activated.
Knight did not yet know whether it could stop the orders before the damage overwhelmed the firm.
The dependency nobody had isolated
The US Securities and Exchange Commission documented what followed in its 2013 order against Knight Capital Americas. Over roughly 45 minutes, Knight’s systems sent millions of orders into the market and accumulated positions worth billions of dollars. The company lost more than $460 million.
The failure involved one server, but the server was only the visible fault. Knight’s deployment process allowed an engineer to update machines manually, with no second person required to review the release. The firm also lacked an automatic process that confirmed every server was running the intended code.
One missed deployment remained invisible until live traffic reached it.
Kojo’s checkout failure has the same mechanism at a smaller scale. His payment provider can disappear because of an outage, an account review, a regional restriction, or a breaking API change. None of those events needs to destroy his morning. The dangerous part is discovering at 8:12 a.m. that the team has never defined an order outside the provider’s successful callback.
Can support record the customer’s request? Can finance verify payment later? Can the customer receive a clear status instead of trying three more times? Can the team reconcile manually created orders without charging someone twice?
If those questions first appear during the outage, the provider has become more than infrastructure. It has become undocumented business logic.
A checkout is a chain of decisions
Early teams often describe checkout as one action: the customer pays. The working system contains several separate decisions.
The product identifies the customer and the items requested. The payment service attempts the charge. The application records what happened. Inventory, access, or fulfilment changes state. The customer receives confirmation. Someone can later explain and reconcile the transaction.
Collapsing those decisions into one API response makes the first version faster to ship. I understand the choice. When runway is short, adding a second provider or building a complete payments abstraction can consume weeks without proving demand.
The mistake is treating expensive redundancy as the only alternative.
Kojo does not need to reproduce a payment network before breakfast. He needs a degraded path that preserves intent and evidence. During an outage, the checkout could stop taking payment while still recording the customer, cart, currency, quoted total, timestamp, and consent to be contacted. The interface should say plainly that payment is unavailable and that no charge has been completed.
That record gives the team something it can inspect and recover. A spinning button gives it nothing.
The same distinction matters in AI products. If an external model API fails, can the product retain the user’s request without pretending the work completed? If an automation stops between two systems, can an operator see the last confirmed step? If nobody can answer, the dependency owns the workflow.
Design the manual path before buying redundancy
The first useful exercise is small enough to run this week. Disable the payment provider in a staging environment and ask one person who did not build the checkout to complete an order.
Watch where they stop. Do not guide them.
Then write the minimum recovery procedure in the order an operator would use it. Record the failed attempt. Confirm whether money moved. Preserve the order details. Tell the customer what happened. Resume or cancel the order through an auditable action. Reconcile the result when the provider returns.
Test the awkward cases too: the provider accepted payment but the callback never arrived; the customer retried; support created a second order; the price changed before recovery. These are product decisions, not edge cases to leave inside an engineer’s head.
This is the same reason a launch test should isolate what it can actually prove. [Kofi’s US launch deadline](\/blog\/kofi-s-us-launch-deadline-pushing-ahead-risked-proving-nothing-b9d79d39\/) was a roadmap problem shaped by uncertainty. A provider outage exposes another kind of uncertainty: whether the company can separate customer demand from the machinery used to collect it.
The next outage should be boring
Knight Capital survived only after outside investors arranged a rescue. The company’s experience is an extreme example, but the engineering lesson does not require a trading system or a $460 million loss.
A dependency should fail inside a boundary the team has already drawn.
For Kojo, that boundary begins with one timed drill. Turn off the provider connection, place a test order, and follow the written recovery path from the customer’s first click through reconciliation. Record every step that depends on one person remembering what to do.
At 8:12 tomorrow morning, the checkout may still go dark. The order should not disappear with it.
Comments
No comments yet.