The Loudest Bug Is Never the Worst
The bug your angriest customer is emailing about and the bug wrecking the most lives are almost never the same bug.
For years Microsoft had a quiet superpower, and the support queue knew nothing about it. Whenever Windows fell over on one of the hundreds of millions of machines running it, the system offered to send back a little report. Where it died, what the failure looked like, a fingerprint of the crash. Those reports got collected and sorted by the fingerprint, so the crashes piled up into buckets, and the buckets said something a support inbox never could. They weren't spread out evenly at all. A small handful of bugs, sitting right at the top of the pile, were behind a huge share of every crash that every user would ever run into. Fix that handful and the day-to-day experience of millions of people got better. And the bugs that mattered most, almost always, were not the ones people were shouting about.
Nobody wants to hear this because it feels unfair, but how loudly a bug gets complained about barely tells you anything about how many people it hurts. The person who emails three times in one day, who tracks down your chief executive on LinkedIn, who writes the furious thread - that person is usually hitting something rare. Some odd combination of features nobody else uses together. An environment you've never seen. A workflow only they depend on. They're not lying, and they're not unimportant. But they are an outlier, and an outlier who shouts is still just an outlier. The bug that ruins one session in twenty for everyone, on the other hand, makes hardly any noise, because the people hitting it just sigh and retry and get on with their day, and each of them assumes it's normal, and not one of them is cross enough to write in. The silent majority is silent by definition. Which is the whole reason you can't let volume run your queue.
The confusion comes from one word doing the work of two questions. Severity asks about damage. When this bug goes off, how bad is it - does it crash, corrupt something, leak something, or just annoy you? Priority asks about time. Given everything else waiting on the list, when do we fix this one? Those aren't the same question and they don't even get asked by the same people. A data-corruption bug that fires once a year down some obscure path is high severity and low priority. A button that's out of line on the launch screen the day before launch is trivial severity and the highest priority you've got. Teams that mash the two words into a single number end up going after whichever bug feels worst right now, which is whichever bug somebody happens to be shouting about, which puts you straight back into the upside-down queue.
The way out is to let the evidence rank the work, the same way the crash buckets did it. Not how angry was the person who reported this, but how many people does it actually reach, and how badly, and is it getting worse. A bug touching ten million users goes above a bug touching a hundred, even when the hundred are louder, even when one of them happens to be important. This feels cold and it's actually the opposite. It's the only way to be fair to the people who never wrote in, the quiet majority soaking up a problem in silence because they assumed nobody was keeping count. Triage on evidence and you can look the loud customer in the eye and tell them, honestly, that their problem is real, here is exactly where it sits, and here is what's ahead of it and why. Triage on volume and all you're doing is rewarding whoever is most willing to ruin your afternoon, and you're punishing everyone too polite or too worn down to try the same trick.
The trouble is you can't suddenly come up with this discipline once the incident is already running. When the channel is full of capital letters and someone senior has forwarded a customer's email with a one-word "thoughts?", you are not going to weigh reach against urgency. You'll fix whatever stops the noise, because that's what pressure pushes you towards. So the ranking has to be there already. Decided ahead of time and written down, with severity scored on what the bug does, and priority scored on how many it reaches and by when, and the two of them kept well apart on purpose. Sort out the rules while your judgement is still your own, because the one thing you can count on is that when it matters most it won't be. You'll be reacting, and a scheme you signed up to in advance is about the only thing standing between your roadmap and whoever turns out to be loudest that day.
In the manifesto, this is tenets (XVIII) and (XIX).
Sources
- [Beyer et al. 2016] Betsy Beyer, Chris Jones, Jennifer Petoff & Niall Richard Murphy (eds.), "Site Reliability Engineering: How Google Runs Production Systems". O'Reilly, 2016. https://sre.google/sre-book/table-of-contents/. Triage and alert on evidence and user-facing impact, not on the loudest report; tenets XVIII, XIX.
- [Glerum et al. 2009] Kirk Glerum, Kinshuman Kinshumann, Steve Greenberg et al., "Debugging in the (Very) Large: Ten Years of Implementation and Experience". SOSP '09, 2009. https://www.microsoft.com/en-us/research/publication/debugging-in-the-very-large-ten-years-of-implementation-and-experience/. Windows Error Reporting buckets crashes by signature so the small set of bugs causing most reports can be ranked first; tenets XVIII, XIX.
- [IEEE 1044-2009] IEEE, "IEEE Standard Classification for Software Anomalies" (Std 1044-2009). IEEE, 2009. https://standards.ieee.org/standard/1044-2009.html. Severity, the damage an anomaly does, classified separately from priority, its ranking against everything else; tenet XIX.
One of a series of field notes on building software for the way minds actually work: tired, distractible, ordinary, and now partly machine. They all lead back to the manifesto behind them, The Shape of the System.