The Shape of the System

What You Set Up, Tear Down

Every resource you open is a promise to close it, and the bill for a broken promise always arrives at the worst possible hour.

implementing reviewing operating

Somewhere in the building a tap has been left a quarter-turn open, in a room nobody really goes into. Nothing dramatic about it. Just a thread of water running, too quiet to hear over the air handling. And the tank that feeds it is huge, so the gauge reads full for weeks and everyone gets on with their day. Then on some ordinary Tuesday a tap upstairs coughs and spits air and the tank is empty. It hadn't run dry that morning. It had been emptying ever since the afternoon someone walked off a job they were sure was finished. That thread of water is every connection you opened and never closed. Every handle you took and forgot to hand back. You can't hear the leak, the tank is generous, and the day the whole thing finally fails is not a day you get to pick.

It all starts with an asymmetry. Opening a resource feels like you've done something - you ask for a database connection, a file, a lock, and one gets handed to you, and now you can do the interesting part. Closing it again gives you nothing. No result, no output, no satisfying tick. So it's the first thing to slip off the end of a function written in a hurry, and the easiest thing to lose down some path you didn't test, the early return, the exception you weren't expecting. Every acquire is a small debt and the language hardly ever makes you sign for it.

The cruel part is the timing. One leaked connection harms nothing. The second one doesn't either, and nor does the ten-thousandth, because you don't pay the cost in errors, you pay it in inventory, and there's a lot of inventory. So the bug ships clean. It passes review. It runs green in staging and serves real customers for weeks, filling up slowly the way a leak fills a sump under the floor. Most bugs go off the moment they run. Not this one. This one waits and gathers and then picks its own moment.

When the tank does finally empty it empties all at once. The pool runs dry and from that second nothing new can be served, because there's no connection left to serve it with. And the alarm that goes off looks nothing like the cause. You get handed a dead system at three in the morning and told to work out why, and the why turns out to be one missing close, written three releases back by somebody who's left the company. This is such a dependable way to fall over that the tools built to catch it are a whole industry on their own. Connection pools now ship with a watchdog and its entire job is to spot a connection that got borrowed and never came back, and to print out the name of whoever took it, because nobody ever believed a person could be trusted to remember.

There's a sister to this, and it's quieter again. When you fail to release something you leak it, but failing to finish is different - you lose something instead. A program can look like it shut down perfectly cleanly and still have dropped the last thing it was doing - a buffer it never flushed, a message it never acknowledged, a transaction left neither committed nor rolled back. Shutdown is the most dangerous moment of all, and it's dangerous precisely because it looks so much like everything going to plan. Scale the old version down with a thirty-second grace period and if the thing doesn't actually stop to drain its in-flight work, then every request still in the air at the cutover gets killed mid-sentence and disappears, and not one error shows up in the log. Something can look clean from the outside and not be clean at all.

It goes deeper than most people are comfortable with, because "saved" is a stack of polite lies. Your code thinks it wrote the data. What it actually did was hand the data to the operating system, which thinks it wrote the data, and the disk hasn't even been told yet. Back in 2018 the Postgres community worked through a case of exactly this, unsettling enough that it got its own name, fsyncgate, the sort of bug that makes the people who build databases stop trusting the ground under their feet. On Linux, when a delayed write to disk failed, the kernel could report that failure once and then wipe the error, so the next time the database asked "did that reach the disk?", back came the answer "yes", cheerfully, about data that was already gone. The database had done everything right and the floor it stood on had lied to it. And the lesson goes a long way past databases. If you don't drain your obligations before you exit, the last thing you did never happened, and nothing is going to tell you.

You can't fix any of this by resolving to remember, because the whole failure is forgetting, and you will forget, on the day it matters most of all. The fix is to make the release impossible to leave out. You bind it to the acquire at the very moment you take the resource, so the close gets written the instant the open does. That's the one idea behind a whole family of tools that grew up separately and then landed on the same shape. There's a thing in C++ called, with an ugliness its own inventor happily admits to, Resource Acquisition Is Initialization; there's defer in Go; try-with-resources in Java; with in Python; using in C sharp. A dozen unrelated languages reached for the same move, and that tells you the flaw underneath is something constant in all of us, not one team's bad habit. Each of them makes the cleanup ride along on every exit, including the exits you never saw coming. Then you carry the same discipline into shutdown - drain the queue, flush the buffer, settle the transaction, and only then leave. You're not trying to turn yourself into the sort of person who always remembers. You're trying to build the kind of room where the tap shuts itself off behind you.

In the manifesto, this is tenets (VII), (VIII) and (XII).

Sources

One of a series of field notes on building software for the way minds actually work: tired, distractible, ordinary, and now partly machine. They all lead back to the manifesto behind them, The Shape of the System.