The Shape of the System

The Blame Trap

Stop asking who made the mistake. Start asking what the system let them make.

retrospective

For a long time, when a plane crashed and the crew had not lived to tell anyone what happened, the investigation could land somewhere comfortable. Pilot error. The pilot pulled the wrong lever or misread a dial or just did the wrong thing, and that was that. Case closed. The lesson, if you could call it that, came out to roughly: be a better pilot. Nothing about the aircraft had to change because nobody had asked it to. And then a different question started to take hold, and it was a harder one. Not who got it wrong but why a trained pilot who was good at the job and meant well had, on that particular day, found the wrong move to be the one that looked reasonable. The answers turned out to have nothing to do with what kind of person the pilot was. There were two switches an inch apart that looked the same. There were rosters that put worn-out people in charge of an aircraft. Training had drilled in an instinct that was exactly wrong for the moment it finally mattered. The error was real enough. But it was the last link in a chain the system had been putting together for years, and once people started asking what the cockpit had let the pilot do, flying got a lot safer. Asking who, it turned out, was the thing that had been holding all of that back.

Software has the same instinct and it goes just as deep. A migration corrupts production and the first thing anyone types in the channel isn't what happened, it's who ran it. A name turns up, the blame settles onto it, and there's that satisfying feeling of a case being closed. That feeling is the whole problem. The minute you've got someone to blame the investigation just stops, because the mind takes a named person as a full explanation and leaves it there. You've got your cause. No reason to keep looking. So you never get round to asking why it was even possible to wreck production with one command, or why there was no staging environment, or no approval gate, or no alarm that caught the damage for ninety minutes, or why the dangerous way of doing it happened to be the quick smooth one. None of that ever comes up. A person is already holding the blame and a person holding the blame feels a lot better than a system holding the fault.

The people who study how things go wrong in cockpits and operating theatres and control rooms have a name for these two versions. The first one is the story that shows up by default, the one where the operator made a bad choice and a bad outcome came out of it. Simple cause, simple effect, and a villain to go with them. The second story is the one you have to actually work for. It's the shift Sidney Dekker has spent a whole career pushing for, away from the bad apple and towards the system that set the apple up to rot. What did the person actually know at the moment they acted? What was the pressure on them, the deadline, the noise, the thing everybody on the team just did without thinking? What feedback was missing that might have told them, while there was still time, that they were heading somewhere bad? Inside a codebase the second story asks why the safe path was slower than the dangerous one. Why breaking the single most important thing in the company took no special effort and tripped no wire on the way. John Allspaw, an engineer who had run more of these than most, made a point his team ended up being known for. Human error isn't where the investigation ends. It's where it begins. A story that finishes with a person leaves you nowhere to go. Finish on the design instead and you've got something you can actually fix.

And the objection turns up straight away. So nobody's ever accountable, then? No, that isn't it, and the people who actually run safety-critical systems have a careful answer to it. A just culture still has consequences. They're just proportional and pointed at the right thing. Real recklessness, the engineer who muted the alarm because the noise was getting on his nerves, still meets one. But where you start from, the assumption you begin with, is that the people involved were competent and were trying, and that something in the setup around them made the wrong move the easiest one to reach for. Some of that is just kindness. The bigger reason is that fear is the enemy of information. When a culture goes looking for culprits, people bury their mistakes and sit on their near-misses and quietly work around the broken thing instead of reporting it, and the organisation ends up blind in the exact spot where it most needs to see. When a culture goes looking for causes instead, people bring up the near-miss that hasn't hurt anybody yet, and that near-miss is about the cheapest lesson you'll ever get handed. Go hunting for someone to blame and what you come away with is a scapegoat. The truth only turns up for the people who went looking for the cause instead, and you don't get both.

So in a real postmortem the question that matters is never why did they do that. It's what would we have had to change so that the choice wasn't even there, or so the damage got caught before it had a chance to spread. The honest answer is nearly always something dull. Better tooling. A guard rail on the dangerous path. A clearer norm about how things get done. An alarm that fires on the right thing for once. Or just the time to do the work properly that nobody ever actually gave anyone. None of that lands the way a name does. But it's the thing that makes the next incident a smaller one. The person at the sharp end, the one whose hand happened to be on the lever, is the easiest thing in the world to point at and almost never the thing that's worth fixing. They were just standing in the place where the system finally gave out. Blaming them is a decision to leave the system every bit as dangerous as you found it, and then to wait, with a clear conscience and a fresh culprit already half lined up, for the whole thing to happen over again.

In the manifesto, this is tenets (XXV) and (XIII).

Sources

  • [Allspaw 2012] John Allspaw, "Blameless PostMortems and a Just Culture". Etsy, Code as Craft, 2012. https://www.etsy.com/codeascraft/blameless-postmortems. Treat the proximate human action as the start of the investigation into a failure's mechanism, not its conclusion, and blameless does not mean unaccountable; tenets XXV, XIII.
  • [Dekker 2014] Sidney Dekker, "The Field Guide to Understanding 'Human Error'", 3rd edn. Ashgate, 2014. https://sidneydekker.com/books/. The shift from the bad-apple view to error as a symptom of systemic conditions, judged by local rationality not character; tenet XXV.
  • [Dekker 2016] Sidney Dekker, "Just Culture: Balancing Safety and Accountability", 2nd edn. Routledge, 2016. https://www.routledge.com/9781472475787. A just culture is proportional consequences aimed at the right thing, not the absence of accountability; tenet XXV.
  • [Woods et al. 2010] David D. Woods, Sidney Dekker, Richard Cook, Leila Johannesen & Nadine Sarter, "Behind Human Error", 2nd edn. Ashgate, 2010. https://www.routledge.com/9780754678342. Source of the first-story (blame, single cause) versus second-story (systemic context) distinction the post turns on; tenet XXV.

One of a series of field notes on building software for the way minds actually work: tired, distractible, ordinary, and now partly machine. They all lead back to the manifesto behind them, The Shape of the System.