The Queue That Never Drains
Past a certain point a system does not slow down, it falls off a cliff, and the only kindness left is to refuse work you cannot finish.
A motorway lane can pass about two thousand cars an hour. You add some traffic and the count goes up, and it keeps going up, right until the cars are so close together that none of them can get to the speed that gave the lane its capacity in the first place. So now every driver is braking because of the brake lights in front, and the lane that managed two thousand is down to four hundred. Stop, start, stop. It's at a hundred per cent occupancy and it's carrying next to nothing. From the air a full road and an empty one look the same. None of this is really about motorways. It's about anything where work waits in a line, and your servers do exactly the same thing the M25 does on a Friday.
The mistake we all make is to think of load like a dimmer switch, where more demand just means smoothly more delay. But the real curve has a knee in it. Below the knee you can pile on work and the response time barely shifts. Above it, one more unit and the delay goes vertical. The system drops instead of sagging, and when you were strolling up the gentle slope a moment ago you couldn't see the edge of the cliff at all. The arithmetic is plain. How long a queue gets is just how fast things arrive multiplied by how long each one sits. And if things arrive even slightly faster than you can serve them, and there's no limit on the line, the wait doesn't settle down at some high but survivable number. It grows. And it carries on growing and never stops. There's no steady state. The queue never drains.
We add queues out of kindness. You want a buffer so nothing gets dropped, so a sudden burst gets soaked up instead of refused. But an unbounded queue isn't a safety margin, it's a debt that keeps compounding. A request turns up, joins the back of the line, and by the time it has shuffled to the front the person who asked for it has given up and gone home. So now you're spending your scarcest resource answering questions that nobody is waiting around to hear. A server finishes working out a perfect reply and writes it out carefully to a connection that the client closed thirty seconds ago. All that work is heat and no light. You can be handling a perfectly healthy number of requests per second while the number you handled in time for it to matter has already dropped to zero, and it's the second number that the people waiting actually feel, even though nearly everyone is measuring the first.
The internet learned this the hard way back in 1986. The link between two laboratories in Berkeley, four hundred yards apart, three hops across the network as it was then, saw its throughput fall from thirty-two thousand bits a second to forty. Not forty thousand. Forty. That's a factor of about a thousand, between two buildings close enough to shout across. The cause was the cure misfiring. A packet would be slow to turn up, so the sender assumed it was lost and sent it again, and that added load, which made everything slower, which set off yet more resending, and on it went until the network was spending itself carrying copy after copy of work it had already failed to deliver once. Van Jacobson and Mike Karels gave the disease a name, congestion collapse, and the fix they published in 1988 added no capacity whatsoever. What it did was teach every sender to slow down when the network pushed back, to read a dropped packet as a signal and not an accident and ease off the throttle. That's the whole thing, and it sits in one of the oldest and most load-bearing pieces of software you touch every single second. The answer to overload wasn't more speed. It was restraint, wired right into the source.
And here's the cruellest turn of all. Sometimes you go and find the thing that triggered the collapse, and you remove it, and the system just stays on the floor anyway. A paper in 2021 gave this a name, a metastable failure. Something tips the system into a bad state, maybe a brief spike or a slow dependency or a bad deploy, and then a second effect, the retries and the missed caches and the sheer weight of the backlog, keeps it pinned down there long after the original thing has gone. Whatever knocked it over has left the building and the system is still on the floor, because by this point it's the thing holding itself down. You can't fix it by undoing what started it, because what started it isn't what's happening any more. You have to break the loop directly. Shed the load, empty the queue, sometimes just switch the whole thing off and let it come back up cold. A system can sit in two completely different states at the very same level of demand, one healthy and one wedged, a bit like a ball resting in one of two valleys, and one hard enough knock will send it over the ridge from the good valley into the bad one with no road back up the way it came down.
The discipline that stops all of this has an ugly name and a simple meaning. It's called backpressure. It means a component that is full gets to tell the thing upstream of it to wait, and the thing upstream has to listen instead of just shovelling more on. Without it, every layer makes the choice that looks sensible right there - accept the request, it's only one more - and the queue grows everywhere at once. With it, the slowness travels back up the chain to the source, and the source is the one place where somebody can actually decide to ask for less. The motorway has a version of this and you've sat at it. It's the red light on the slip road that lets one car onto the carriageway at a time. It feels like an obstruction. It's the opposite. Letting everyone merge freely whenever they fancy is the exact thing that turns the whole road into a car park.
Backpressure tells the source to slow down. Load shedding is the harder one to stomach, which is refusing work outright, fast and cheap, rather than taking it on and then failing it slowly. A request that shows up already past its deadline ought to be dropped at the door, not carried gently to the front of the queue so it can die there instead. Saying no in a millisecond is how you protect the things you already said yes to. Every instinct tells you that dropping requests is failure, that a healthy system should never do such a thing, and the instinct has it exactly backwards. A system that can't drop anything is a system that collapses completely the first time demand outruns capacity, and demand always outruns capacity in the end. Graceful degradation isn't accepting everything and then serving all of it badly. It's deciding, on purpose, what you won't serve, so that the things you do serve still turn up while there's still somebody who cares. The kindest queue in the world is the one with a bouncer on the door.
In the manifesto, this is tenets (V), (VII) and (XIX).
Sources
- [Bronson et al. 2021] Nathan Bronson, Abutalib Aghayev, Aleksey Charapko & Timothy Zhu, "Metastable Failures in Distributed Systems". HotOS '21, 2021. https://sigops.org/s/conferences/hotos/2021/papers/hotos21-s11-bronson.pdf. Names the trigger versus the sustaining effect that pins a system in a bad state after the cause is gone; tenets V, XIX.
- [Jacobson & Karels 1988] Van Jacobson & Michael J. Karels, "Congestion Avoidance and Control". ACM SIGCOMM '88, 1988. https://ee.lbl.gov/papers/congavoid.pdf. The October 1986 Berkeley collapse (32 Kbit/s to 40 bit/s) and restraint wired into the sender as the fix; tenets V, VII.
- [Little 1961] John D. C. Little, "A Proof for the Queuing Formula: L = λW". Operations Research 9(3), 1961. https://www.jstor.org/stable/167570. Queue length is arrival rate times wait, so at saturation a deeper buffer only adds latency; tenets V, VII.
- [Nygard 2007] Michael T. Nygard, "Release It! Design and Deploy Production-Ready Software". Pragmatic Bookshelf, 2007. https://pragprog.com/titles/mnee2/release-it-second-edition/. Fail Fast and load shedding: the discipline of refusing work cheaply rather than failing it slowly; tenets VII, XIX.
- [Reactive Streams 2015] Reactive Streams initiative, "Reactive Streams". 2015. https://www.reactive-streams.org/. Backpressure: a full consumer signals the producer upstream to slow down; tenet V.
One of a series of field notes on building software for the way minds actually work: tired, distractible, ordinary, and now partly machine. They all lead back to the manifesto behind them, The Shape of the System.