I would pause onboarding for Tuesday’s launch and rebuild the retention controls before accepting new customer data. Once a policy covers historical backups, deleting records from the live database solves only the visible part of the problem.
On January 31, 2017, GitLab was dealing with a database incident that had already gone badly. During troubleshooting, an engineer deleted production data. The recovery options then exposed a second problem: several backup procedures had failed or were incomplete.
GitLab documented the incident publicly, including the deletion, the failed recovery paths and the work required to restore service. Cofounder and CEO Sid Sijbrandij later described the company’s response in GitLab’s published postmortem. The uncomfortable part was clear. A backup policy can look adequate until the moment someone needs the backup.
That same uncertainty sits inside the fintech founder’s Tuesday decision. The policy has changed, onboarding is scheduled, and the backups now carry obligations the team designed only for production data.
The live database is only one copy
The first retention control usually lives where engineers can see it. A customer closes an account, a scheduled job marks the record, and the production database deletes or anonymises it after the required period.
Then someone asks what happens inside yesterday’s snapshot.
A fintech product may hold the same customer information in database backups, exported reports, object storage, test environments and logs. If the new policy applies to historical backups, the founder needs an answer for each copy. “Production deletes it” no longer covers the system.
This is where Tuesday’s launch becomes a real decision rather than a compliance checkbox. Pausing onboarding costs momentum and may disappoint customers who were expecting access. Continuing creates more data under controls the team already knows are incomplete.
I would choose the smaller reversible loss. Delay new onboarding while keeping the existing service stable. A missed launch window can be recovered. New customer records copied into an unresolved backup process create additional cleanup work and a larger set of people affected by any mistake.
Rebuild the deletion path around restore events
Deleting individual records from every immutable backup may be technically difficult or may weaken the recovery system itself. The practical control can sit at the point where archived data becomes active again.
The team should write down what happens when a backup is restored:
- The restored environment remains isolated from normal product traffic.
- A current deletion ledger is applied before customers or internal teams can use the data.
- Records beyond the approved retention period are removed or anonymised.
- Access to the restore is logged and limited.
- The team tests the procedure with a real backup and records the result.
The exact design depends on the storage system and the policy. The important part is that “we can restore” and “we can restore safely under today’s rules” become separate tests.
GitLab’s 2017 incident matters here because the company discovered that having backup jobs did not guarantee a usable recovery path. For the fintech founder, written retention rules provide the same false comfort if nobody has tested them against a restored historical copy.
A restore drill would tell me more than another afternoon spent reviewing configuration screens.
Give the launch a narrow restart condition
“Fix retention” is too broad for a team with one afternoon. It invites scattered work across logs, databases and infrastructure, followed by a late decision based on fatigue.
I would define a short restart condition before touching the system. Onboarding resumes when the team can identify every backup containing regulated customer data, show how expired records are handled after a restore, and complete one observed recovery test without exposing deleted records.
This follows the same logic as deciding what to automate when onboarding fails with six weeks of runway. The constraint determines the next useful piece of work. With one afternoon, the goal is a verified control around the risky path, followed by a documented plan for the remaining stores.
I would also keep the founder out of low-level implementation once the decision is made. Their work is to name the risk owner, protect the pause from commercial pressure and decide what evidence is sufficient to restart. An engineer can build the control. Someone must still make the call when the evidence is incomplete.
Tuesday should end with evidence
By the end of the afternoon, I would want a simple record: which systems were checked, which backups were tested, what happened after restoration, who observed the test and what remains unresolved.
If the restore test passes, onboarding can resume under a control the team has seen work. If it fails, the pause continues. That answer may be commercially awkward, but it is clearer than launching with a promise the system cannot yet keep.
In 2017, GitLab’s recovery problem became visible only after production data was already gone. The fintech founder has a cheaper warning. The policy review exposed the gap before Tuesday’s new customers entered the backup chain.
Use the afternoon to test that warning. Restore one backup, apply the retention process, and inspect the result before reopening onboarding.
Comments
No comments yet.