The Shape of the System

The Rumour Test

An unreproducible report is a rumour with a timestamp. Triage is the work of turning the rumour into something you can actually rank.

triage investigate

On the fourth of January 2021, the first Monday back after the holidays, Slack came in to work and fell over. All those hundreds of thousands of people sitting down at their desks at the same moment put a load on it that a quiet fortnight had let everyone forget was even possible, and the service started to buckle. The engineers triaging it went for the obvious suspect first, which was the stuff that had shipped that morning. So they rolled it back. Didn't help. The deploys turned out to be innocent and the timing was just a coincidence. Then they looked at the cloud underneath. And the actual fault, once it finally showed itself, was none of the things they'd grabbed for early on - it was a bit of network plumbing that was supposed to scale up when demand rose and just hadn't, not fast enough anyway, to cope with a wall of traffic that nobody had thought to load-test for the Monday after Christmas. So triage threw up a run of confident, reasonable, wrong first guesses. That isn't bad engineering. It's just what the work normally feels like.

Here's the thing the whole discipline hangs on. What a user tells you is what they saw. It is not what happened. "The app is slow" is a symptom and the fault might be a missing index three layers down. "I got an error" is a symptom too, and the error might be the polite face of a timeout off somewhere the user will never get to see. And reports almost never turn up clean. They come wrapped in the reporter's own theory of what caused it, a guess stitched right into the words they use, so tightly that you have to do actual work to pull the two apart. The first thing you do in triage is prise apart the three things that arrive stuck together: what the user actually saw, the story they've built around it, and the fault that's still hiding under both. The first of those you act on. The second you keep at arm's length and don't trust. And it's only ever the third one you can fix.

Which is why getting it to happen again is the gate and not some nice-to-have on top. A report you can't reproduce can be perfectly true and still matter, but it stays unranked, because you can't honestly weigh up how urgent something is when you've never once watched it happen. So when a report won't reproduce, you change the questions. What changed, and when did it change? Is there a pattern to it - a particular hour, a particular load, or maybe just one kind of account? Are other people seeing the same sort of thing? A race that only shows up under contention, a failure that only turns up at scale, a bug that hits one tenant in a thousand - these are all real and some of them are nasty. But they sit at low urgency until you can either make them go off in a lab, or pin down the condition that makes them go off reliably out in the wild. Reproduction, or a clear correlation if you can't get reproduction, is what carries you across from rumour to rank. Without it you aren't really triaging at all. You're just believing whichever version of the story sounds most sure of itself.

The first guess will usually be wrong, and it's still useful even so. Triage grabs for the nearest lever, which is the thing that changed most recently, or the component with the most instrumentation on it, or whatever happens to look most like a culprit. Slack grabbed for the morning's deploys, and the deploys were a recent and plausible thing to suspect. They just weren't the cause. This is the ordinary way a first diagnosis goes wrong, and the fix for it isn't to guess better. It's to keep asking the one question that turns a rumour into a fact, which is: what would have to be true, down inside the system, for this symptom to show up at all, and can I make that condition happen on purpose? A wrong first guess that narrows the field is doing its job. A wrong first guess you've mistaken for the answer is how you end up spending a whole outage rolling back something that was never broken in the first place.

There's a cruel wrinkle here as well. Sometimes the numbers you're watching swear everything is fine while the experience is obviously broken. That Monday, most messages were still getting through, so going by the headline count the system looked mostly up, and some of the very dashboards that should have flagged the trouble were themselves sitting behind the failing plumbing and had gone dark. A report that won't reproduce, sat next to a wall of healthy-looking numbers, doesn't mean the user has got it wrong. Sometimes it means your instruments are pointed at the wrong thing. Or they're caught up in the same failure they're supposed to be watching. There's even a name for the kind of failure that wrecks the experience without shifting any of your top-line metrics. The people who study big systems call it gray failure, and the whole point of it is to be invisible to exactly the measures you trust the most.

So you rank by the things that survive contact with the system, not by how well somebody wrote the report up. Is this one user's odd configuration, or is it hundreds of people hitting it? Did it start with a deploy, or a traffic spike, or with nothing you can put a name to? Is the symptom even plausible given how the thing is actually built, or would it take three coincidences and a power cut to make it happen? A scruffy report from a hundred people all describing the same shape beats a beautifully isolated one from a single person, because the footprint is itself evidence that the condition is real and going to come back. And the decision triage is actually making is narrower than people seem to think. It isn't how do we fix this. It's whether this is even worth the cost of going to look. A fault you can set off whenever you like ranks high, because the hunt has an edge to it. A fault you can set off one time in a thousand ranks lower - unless it loses data or crosses a security line, and then the rarity buys it nothing at all. A fault you can't set off at all gets the hardest question there is: is the condition rare enough that a guard rail comes cheaper than the hunt? Triage makes that call before anybody has spent a single day in the logs, working with nothing but the shape of the symptom, the size of the footprint, and a clear head about the way systems really break.

In the manifesto, this is tenets (XVIII) and (XXIV).

Sources

One of a series of field notes on building software for the way minds actually work: tired, distractible, ordinary, and now partly machine. They all lead back to the manifesto behind them, The Shape of the System.