A production database marked for replacement is a stop signal, even at 4:47 p.m. before launch. The deadline can wait until someone can explain why Terraform wants to destroy the database, what data would survive, and how recovery has been tested.
In 2012, engineers at Knight Capital were preparing software for the New York Stock Exchange’s new Retail Liquidity Program. The deployment reached seven of the firm’s eight servers. One server kept old code.
The system went live anyway.
The server that missed the deployment
On August 1, 2012, Knight Capital began receiving customer orders after the market opened in New York. The server with the old software started sending millions of unintended orders into the market.
Knight tried to identify the cause while the system continued trading. According to the US Securities and Exchange Commission’s administrative order, the company’s controls failed to stop the erroneous activity automatically. By the time Knight halted it, roughly 45 minutes had passed and the firm had lost more than $460 million.
The deployment problem was small enough to describe in one sentence: one of eight servers did not receive the new code. Its consequences threatened the company.
That is the useful connection to a Terraform plan showing `-/+` beside a production database. The warning may appear as one line among dozens, but infrastructure tools act on the state they can see. If the tool believes replacement is required, your confidence about what you intended will not protect the data.
What the deadline changes
At 4:47 p.m., the founder has two competing facts.
The launch matters. A customer may be waiting. The team may have spent the week promising Friday. Someone may already have prepared the announcement.
The database also matters. It may contain accounts, transaction records, product configurations, or the only clean copy of data collected during a pilot. Replacing it could mean downtime. It could also mean permanent loss if backups are missing, stale, or impossible to restore.
Time pressure makes the plan tempting to interpret rather than investigate. The founder may think Terraform is being conservative, the database will probably be recreated with the same data, or the cloud provider will surely retain a snapshot.
“Probably” has no place in a destructive production change.
I would stop the apply and reduce the question until the team can answer it in plain language: Which configuration change forced replacement? Is Terraform reading the correct state? Did a resource name, region, provider setting, or immutable property change? What exactly happens to the existing database when the apply runs?
If nobody can answer those questions before the deadline, the deadline moves.
A safe plan needs evidence
The next step is not a long meeting. It is a short investigation that produces evidence.
First, save the plan output and identify the attribute marked as forcing replacement. Check the configuration change against the current production resource and the state Terraform is using. A surprising replacement can come from a genuine infrastructure change, but it can also reveal drift, an incorrect workspace, a state problem, or a resource imported under the wrong assumptions.
Then verify recovery independently. A dashboard saying a backup exists is weaker than a restore completed in an isolated environment. The team needs to know what the backup contains, when it was created, and whether the application can connect to the restored database.
Finally, separate the launch from the dangerous change. If the product release can ship without applying the database replacement, remove that dependency. If it cannot, reschedule the release with a precise explanation: production infrastructure requires investigation before deployment.
This is the same discipline behind the server Knight Capital missed. Deployment completion must be verified against the actual production target, not inferred from a script finishing successfully somewhere else.
Make replacement difficult to approve casually
After the immediate incident, change the system so the next founder does not face the same decision at 4:47.
Protect stateful production resources against deletion where the tooling allows it. Require a reviewed plan for production changes. Keep database changes separate from routine application deployments. Add an explicit approval step when a plan contains `destroy` or `replace`. Test restores on a schedule the team can maintain.
Those controls will add friction. That is their purpose.
Knight Capital’s eight servers looked like one deployment target until the missed server proved otherwise. A production database can look like one Terraform resource until a replacement plan exposes the difference between the configuration, the state file, and the live system.
At 4:47, save the plan. Cancel the apply. Write down the exact replacement trigger before anyone leaves for the evening.
Comments
No comments yet.