No Single Slice
One lock is not security. It is a single point of failure you have not yet watched fail.
Picture the man on the door who also does the cellar, keeps the books, and counts the till at the end of the night. One person doing four jobs, and a nice warm feeling that the place is run well. It isn't run well. It has one employee and four ways for him to rob you, because the only thing checking each job is the same pair of hands that did it, and those hands make mistakes. We do this to systems constantly. There's a disaster we want to keep out, so we drop one check in front of it, and there's that satisfying click as it shuts, and then we go around saying the building is safe. A single check isn't a defence though. It only feels like one. Which, when you think about it, is the exact thing a single point of failure is good at. The question was never whether your one good lock holds. The question is what happens on the day it doesn't, and the honest answer most of the time is nothing happens, nothing at all, because there was never anything behind it.
We trust the one barrier because we can see it and it nearly always works. A password. A code review. A validation step. A firewall. Each of these feels like the single thing standing between you and the bad outcome. But "nearly always works" is precisely the property a single point of failure has, right up until the morning it doesn't. A check you have never once seen fail looks exactly the same as a check that cannot fail, and you only find out which one you had the once, at the worst time it could possibly happen. A barrier never failing is not proof it can't. All it proves is that the day which would have told you hasn't come round yet.
There's a psychologist, James Reason, who gave us the picture that actually works here, and the picture is a slice of cheese. Real defences aren't solid walls. They're slices of Swiss cheese, and every slice is full of holes you never meant to leave. The patch nobody ever got round to applying. The rule that turned out to have an exception in it. An alert that goes off into a mailbox no one has opened in a year. The login half the team shares. Holes like that are just the ordinary state of any real control sitting inside a real organisation that's full of busy people with too much on. They aren't a sign that someone didn't care. No slice is whole. Where it goes wrong is believing yours is the one solid slice, the rare one, and then stacking only that. Put several slices up instead, each one holed in its own way, and most days a hole in one slice has solid cheese behind it in the next. For a disaster you need the holes in every single slice to line up at the same moment, all the way through, far enough that you can stand on one side and see daylight out the other. Safety isn't a slice with no holes. There's no such slice. Safety is keeping the holes from ever lining up.
In 2017 they lined up, all the way through, at a company that was holding the financial lives of getting on for half of America. Equifax. The first slice was a flaw in a widely used bit of web software, the sort of thing that gets found and announced and patched by everyone who's paying attention, and Equifax just didn't apply the patch. For months. So the front door was standing open with a public notice taped next to it telling you how to get in. The attackers got in. Now the second slice should have stopped them right there, except the inside of the network was flat and open, so from that one cracked web server they could get across to the databases, dozens of them, and take what they wanted. Then the third slice, the one that was supposed to catch them hauling the data back out, was a device that inspected the encrypted traffic leaving the building. Only the certificate that let it read that traffic had expired. Expired months earlier. So the device sat there doing its job over a stream it could no longer see into. For seventy-six days the data of a hundred and forty-seven million people walked out past a guard who'd been blindfolded without anyone noticing. Somebody renewed the certificate, the guard could see again, and the alarm went off straight away. Three slices. A hole in each one. And the holes were in a perfect line.
So the job can't be to go and find the one perfect check, because there isn't one. Every slice you'll ever cut has holes in it somewhere. The job is to assume each layer fails and put another one behind it, a different one, something that fails in a different way for different reasons, so the thing that sails clean through the first layer gets caught by the second. Defence in depth isn't paranoia and it isn't belt-and-braces fussiness. It's just owning up to the fact that you can't see the holes in your own cheese. The patch you're certain went on didn't. The alert you're certain fires has been dropping into a dead inbox since March. You don't get to inspect your slices and spot the holes ahead of time. Read any postmortem ever written and you'll see people guessed wrong about exactly the holes that ended up mattering. What you do get to do is refuse to bet everything on one slice. One lock on the door is a decision to find out, on the worst day of your life, whether you had the solid slice or the one full of holes. Two locks that fail for different reasons is a decision to not have that day.
In the manifesto, this is tenets (XIX), (IV) and (XVI).
Sources
- [GAO 2018] U.S. Government Accountability Office, "Data Protection: Actions Taken by Equifax and Federal Agencies in Response to the 2017 Breach" (GAO-18-559). 2018. https://www.gao.gov/products/gao-18-559. The 76-day undetected exfiltration, the expired traffic-inspection certificate, and the unsegmented internal network; tenets XIX, IV, XVI.
- [House Oversight 2018] U.S. House Committee on Oversight and Government Reform, "The Equifax Data Breach" (majority staff report). 2018. https://oversight.house.gov/wp-content/uploads/2018/12/Equifax-Report.pdf. The unpatched Apache Struts flaw (CVE-2017-5638) and the breach of around 148 million people as a preventable chain of failures; tenets IV, XIX.
- [Reason 2000] James Reason, "Human error: models and management". BMJ 320(7237):768-770, 2000. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1070929/. The Swiss cheese model: layered defences as holed slices whose holes occasionally line up; tenet XIX.
- [Saltzer & Schroeder 1975] Jerome H. Saltzer & Michael D. Schroeder, "The Protection of Information in Computer Systems". Proc. IEEE 63(9), 1975. https://web.mit.edu/Saltzer/www/publications/protection/. Defence in depth, fail-safe defaults, and least privilege as designed-in containment; tenets IV, XVI, XIX.
One of a series of field notes on building software for the way minds actually work: tired, distractible, ordinary, and now partly machine. They all lead back to the manifesto behind them, The Shape of the System.