Break It on Purpose
You do not have a failover. You have a failover you have never triggered, which is a different thing with the same name.
The fire drill nobody runs is the one that kills people. Every office has that laminated sign by the lift, in the event of fire do not use, and most of us have walked past it for years without once timing the walk down to the car park in the dark. Nobody knows the back stairwell is chained shut. You find that out on the day there is smoke in it, which is too late. Software is the same building. The sign just reads failover is automatic, and nobody has ever pulled the alarm to check it works. So one team went and did a thing that sounds completely mad right up until the moment it sounds obvious. They wrote a program whose whole job is to walk into the live data centre, on an ordinary Tuesday afternoon, and start switching things off while everyone is sat there watching.
You believe your system survives a server dying, or a whole region going dark, or a dependency that hangs on the line and never answers. Belief is free. It is also untested, and the only thing that turns it into actual knowledge is the failure happening for real. But by then it isn't a test, it's an incident, with real customers attached to it and someone's afternoon ruined. So your confidence is sitting on top of a pile of events you have never once watched take place. Put it bluntly. Every resilience feature you haven't deliberately set off is a guess that the team has quietly agreed, between themselves, to start calling a guarantee.
Netflix learned this the expensive way, which is how everyone learns it. Back in 2008 they pushed an update to a bit of storage hardware, and it corrupted the database the business ran on, and for three days they couldn't ship a single DVD to anyone. The lesson they took from that wasn't be more careful next time. It was that one big precious thing will fall over eventually no matter how carefully you guard it, and when it does, every other thing bolted to it goes down with it. So they stopped having one big precious thing. They moved onto a fleet of cheap, anonymous servers, the kind you assume will die without warning, instead of a handful of expensive ones you sit and pray over. And that handed them a brand new problem, which was a whole fleet of things that die without warning.
Here is the bit that turns it round. If your servers are going to disappear at random anyway, then the worst possible moment for one to go is three in the morning on a bank holiday weekend with nobody awake. So in 2011 Netflix built a thing called Chaos Monkey. It reaches into the live system and kills running servers, at random, on purpose, but only during office hours. Not because killing servers is a good thing to do. They did it because killing them while everyone is at their desk with a coffee, watching what happens, is the only honest way to find out whether you actually survive the same thing happening at three in the morning. What was radical about it was the timing rather than the destruction.
This is the line the whole idea has to hold or it is just vandalism with a budget behind it. Chaos Monkey isn't an arsonist, it's a controlled experiment. You start with some belief about how the system behaves when it's healthy, you introduce one real failure, a specific one, and you watch closely to see whether it stays healthy. And the thing you never let go of is keeping the damage small enough that you can stop it the second you need to. You find the weak floorboard by stepping on it gently with the lights on, not by going through it in the dark with the house full of people. The chaos was never the point. A disaster you book in for a quiet Tuesday afternoon is cheap, and that exact same disaster at two in the morning with nobody watching the screens is anything but.
One dead server is a small truth and Netflix wanted bigger ones, so they built bigger experiments. There was a monkey that made a healthy dependency answer slowly instead of dying outright, which is often the worse case, because a dead neighbour you route around straight away but a slow one quietly jams up everyone waiting on it. There was a bigger beast that switched off a whole zone of the data centre. And a bigger one again that evacuated an entire region of the world, run against the real production system, so that on the day a region actually went bad the traffic had already practised its escape and just did the same thing over again. The earlier you stage the disaster, the smaller and duller and more survivable it turns out to be.
None of this was really new. Netflix inherited it. Years before, over at Amazon, a man called Jesse Robbins, a trained firefighter whose actual job title was Master of Disaster, ran what he called GameDays. These were scheduled, announced exercises where teams deliberately broke their most important systems to drag out the dependencies nobody had ever bothered to write down. A firefighter doesn't wait for a real fire to learn the layout of the building. And this is the part that matters for you, sitting there without a fleet or a monkey or a service the size of a planet. You don't need any of it. Pick the one failure you are quietly most frightened of, the one you hope to god never happens, book it in for a Tuesday, tell everyone it's coming, and go and find out. Confidence you have never tested is just a nice feeling you have about yourself. The only sort worth anything is the sort you went looking for on purpose, half hoping it would let you down, so that it could never catch you out later.
In the manifesto, this is tenets (XIX) and (XXIV).
Sources
- [Basiri et al. 2016] Ali Basiri, Niosha Behnam, Ruud de Rooij, Lorin Hochstein, Luke Kosewski, Justin Reynolds & Casey Rosenthal, "Chaos Engineering". IEEE Software 33(3), 2016. https://doi.org/10.1109/MS.2016.60. Chaos engineering defined: experiments on a steady-state hypothesis with a minimised blast radius; tenets XIX, XXIV.
- [Bennett 2012] Tom Bennett (with Jesse Robbins, John Allspaw et al.), "Resilience Engineering: Learning to Embrace Failure". ACM Queue 10(9), 2012. https://queue.acm.org/detail.cfm?id=2371297. Robbins, a trained firefighter and Amazon's 'Master of Disaster', on GameDay exercises that break critical systems on purpose; tenet XXIV.
- [Izrailevsky & Tseitlin 2011] Yury Izrailevsky & Ariel Tseitlin, "The Netflix Simian Army". Netflix TechBlog, 2011. https://netflixtechblog.com/the-netflix-simian-army-16e57fbab116. Latency Monkey degrades a dependency and Chaos Gorilla drops an availability zone: the larger experiments beyond a single killed instance; tenet XIX.
- [Netflix 2016] Yury Izrailevsky, Stevan Vlaovic & Ruslan Meshenberg, "Completing the Netflix Cloud Migration". Netflix, 2016. https://about.netflix.com/en/news/completing-the-netflix-cloud-migration. The 2008 database-corruption outage as the catalyst for abandoning vertically scaled single points of failure; tenet XIX.
- [Principles of Chaos 2015] "Principles of Chaos Engineering". principlesofchaos.org, 2015. https://principlesofchaos.org/. Rehearse failure as a deliberate experiment and minimise the blast radius so you can stop it; an untested degraded path is a latent bug; tenets XIX, XXIV.
One of a series of field notes on building software for the way minds actually work: tired, distractible, ordinary, and now partly machine. They all lead back to the manifesto behind them, The Shape of the System.