Nobody Wants a Backup
Everybody has a backup. Almost nobody has a restore. The difference is the whole story.
A backup you have never restored from is not really a backup. It is a wish you wrote down and filed somewhere safe. You feel covered by it the way a man with a folded parachute feels covered, reassured by an object whose cord he has never once pulled. Nobody actually wants a backup. On the worst morning of their working life what they want is the thing on the far side of it, which is the data coming back, and that turns out to be a different thing entirely, because in between the two sits a step that hardly anybody ever tests. The space between "we have backups" and "we got our data back" is where companies quietly die.
A backup is a noun people collect. Restoring is something almost nobody actually does. And the noun is what we measure - jobs completed, green ticks, gigabytes shipped off to somewhere far away - and none of that proves anything except that some bytes left the building. It is a promise about the future that has never been checked against the future. The only honest test is to throw away the original and try to rebuild everything from the copy, and that test frightens people so badly that almost nobody puts it in the calendar, so almost nobody knows what they have got. A folded parachute and a folded bedsheet look the same right up until you jump.
This is the trap and it is a cruel one, because the reassurance keeps growing while the actual safety stays flat. Add a second backup and you feel twice as covered. Add a fifth and you feel safe from every direction. But until one of them has been restored from, all five are sitting in the same unknown state, and writing another one does not change that state. It just adds another coat to the same body. The people who run these systems have a grim saying about it: you do not know the condition of a backup until you have tried a restore. Five untested backups are not five times safer than one. They are a single guess, repeated five times in a confident voice.
On the last night of January 2017 an engineer at GitLab found out what that costs. He was already deep in some other incident, late, tired, the database struggling under a flood of load and replication falling behind. He was trying to get a stuck replica to re-sync, which meant clearing out its data directory so it could copy the primary fresh. He ran the command against what he thought was the broken replica. It was the primary. Around three hundred gigabytes of live production data started to disappear, and he noticed and killed the command with a few gigabytes left. That was the part they survived. Then they went to the backups.
GitLab had five ways of backing up that database and that night not a single one of them worked. The nightly dumps to cloud storage were producing files only a few bytes long, because the dump tool happened to be the wrong version for the database it was pointed at, and it was failing without saying anything, so the place the files were meant to land was just empty. The alerts that should have been screaming about this for months had been bouncing, because the failure emails were being silently rejected, and so nobody ever saw them. The disk snapshots in the cloud were switched on for the machines nearby but had never been switched on for these. And the replica had already thrown away the very logs it would have needed to catch up. Five mechanisms, five coats on one body, and the body underneath was naked. What actually saved the company was an accident. One engineer had taken a single snapshot by hand about six hours earlier, to seed a copy onto staging, and they restored from that. They lost six hours of real people's work, thousands of projects and comments and new accounts that just stopped having existed, and they watched the recovery crawl for most of a day because the copy was choking on a slow disk. They had five backups and no restore.
And here is the part that should keep you up at night, because even the backups that did exist would not have saved them, and not because of bad luck. They all shared a fate with the thing they were guarding. Same script. Same credentials, the same silent version bug, the same blind alerting, the same assumption that someone else had it covered. There is a lovely idea from the design of the early internet called fate-sharing, where you keep the record of a thing in the same place as the thing, so the only way you lose the record is by losing what it described. On a network that is a virtue. In your backups it is the entire disease. Copies that ride the same job in the same region behind the same broken alarm are not independent copies at all. They are one copy that has learned to count to five. Real redundancy was never about how many spares you are holding. It is about how many separate, unrelated things would all have to fail at the same moment, and if that answer is one, then it does not matter how many copies you made, because you made one. Five spare keys cut from the same blank and hung on the same hook by the same door are not five chances. They are one.
So the only backup worth trusting is one you have actually watched come back to life, recently, the whole way through, ideally onto a machine that has never heard of the original. Which means running the drill you keep putting off. You wipe a copy, you rebuild it from cold, you put a stopwatch on it, and you find out your recovery is bottlenecked on a slow disk on some calm afternoon instead of in the middle of a real disaster with the whole company watching. Time-to-restore is a real number and it has a unit. "We have backups" is a feeling. The restore was always the thing you needed. The backup is just the box it arrived in, and on the worst morning of his life, nobody has ever once wanted the box.
In the manifesto, this is tenets (XXIV) and (XIII).
Sources
- [Clark 1988] David D. Clark, "The Design Philosophy of the DARPA Internet Protocols". Proc. ACM SIGCOMM, Computer Communication Review 18(4), 1988. https://dl.acm.org/doi/10.1145/52324.52336. The origin of 'fate-sharing': co-locate state with the thing it describes, the network virtue that becomes the backup anti-pattern when copies share a fate; tenets XIII, XX.
- [GitLab 2017] GitLab, "Postmortem of database outage of January 31". GitLab, 2017. https://about.gitlab.com/blog/postmortem-of-database-outage-of-january-31/. The primary wiped instead of a replica, every backup and replication path broken or disabled, a stray manual snapshot the only restore; tenets XIII, XXIV.
- [Nygard 2007] Michael T. Nygard, "Release It! Design and Deploy Production-Ready Software". Pragmatic Bookshelf, 2007/2018. https://pragprog.com/titles/mnee2/release-it-second-edition/. Untested recovery and silent failure as stability anti-patterns behind 'a backup you never restore is not a backup'; tenet XIII.
- [Vogels 2006] Werner Vogels, "A Conversation with Werner Vogels". ACM Queue 4(4), 2006. https://queue.acm.org/detail.cfm?id=1142065. You build it, you run it: time-to-restore is a property someone must own and rehearse, not assume; tenet XXIV.
One of a series of field notes on building software for the way minds actually work: tired, distractible, ordinary, and now partly machine. They all lead back to the manifesto behind them, The Shape of the System.