The Shape of the System

Two Clocks

Every fact you store twice is a promise to keep two things equal forever, and you will not keep it.

implementing planning

There's an old line, usually called Segal's law, that a man with one watch always knows the time and a man with two is never sure. Think of the two clocks up on the wall of some old newsroom, eleven seconds apart. Nobody minds for ages, because nobody has had to choose between them. Then a deadline is coming up, or a trade has to clear, and someone puts a finger on one of them and says "that one". And look at what the second clock actually did for you. More time-keeping wasn't it. If anything it took some away. You had the time before. Now you've got a question you didn't have, and that's really the whole problem shrunk down small, and most systems are quietly full of these second clocks that nobody remembers putting up.

Here's the thing you have to be honest about first. Two copies of a fact don't stay equal just because you'd like them to. They stay equal only if every change to one gets mirrored to the other, in the right order, never once missing, forever. Which is a perfect record across millions of events. Drop a single update, or lose a message, or crash in the half-second between writing the first copy and the second, and the two of them part company and nothing tells you. Both copies still look like data, confident, well-formed. So drift isn't the accident here. Drift is just what these things do when left alone. The equality is the costly part, the thing you pay for on every write and stop paying for the moment you stop looking.

You can see it in something as dull as an order total. Say you store the total as its own number, sitting next to the list of line items it's meant to be the sum of. For a long time they agree, because every path that touches the items also remembers to fix up the total. Then a refund comes in through some path another person wrote, and it changes the line items and forgets the total, and now the order says it cost one thing while being made of another. Nobody really wrote a bug. They just had to remember a link the code never forced them to keep, and one day, on one path, somebody didn't.

There's another old line among programmers, this one Phil Karlton's, that there are only two hard things in computer science and one of them is cache invalidation. It's hard for a precise reason. A stale copy can't tell you it's gone stale. It looks the same as a fresh one, which is the whole point of it being a copy. The only way to find out if it's still true is to go and ask the original, and asking the original is exactly the trip the copy was there to save you. So every time, you're choosing between something fast that might be lying to you and something slow that's always right. Every cache anyone has ever built is a bet about how long a fact stays true, placed quietly, and sometimes lost.

On the twenty-first of October 2018, two copies of the same data went to war, and it took GitHub the better part of a day to sort out the peace. A network link between two of their data centres dropped for forty-three seconds. That was enough. The system failed over, the far coast started taking writes, and the near coast had never stopped taking them, so now there were two databases on two coasts, each one accepting changes the other would never see, and they carried on like that for nearly an hour before anyone noticed the split. When the link came back, neither side was a clean copy of the other. Each one held real work the other didn't have. One busy cluster on its own had nine hundred and fifty-four writes stranded in the gap. There was no clock to point at. You couldn't just declare one side the truth, because that meant throwing away things people had actually done. Stitching a single agreed reality back together took twenty-four hours and eleven minutes, out of a forty-three-second blink, because once two copies of a fact can both move on their own you haven't built yourself a spare. What you've built is a future argument about which copy was real.

You don't keep two clocks in step by checking them more often. You take one down and run a wire from the other. For every fact, pick the one place that's allowed to be right. Everything else that needs it doesn't get a copy. It gets a view, either worked out fresh from the source whenever it's asked, or fed from the source so it only changes when the source does. A view can't disagree with what it's derived from, no more than a shadow can argue with the hand throwing it. The unread count isn't a number you store and then fight to keep correct. It's the count of the unread things, and there's no way for it to be wrong. And the cached page is the same idea, just a derivation with an honest expiry stamped on it, not a second truth. Do it this way and two copies disagreeing stops being a thing you chase at midnight. It becomes a state the system simply has no way to get into.

So before you write a fact down for the second time, ask the one question that decides whether you sleep at night: which of these two is allowed to be wrong. If the answer is "neither, they have to always match", then you've just promised to keep two clocks in perfect step by hand, for the rest of the system's life, and that's a promise you'll break the first quiet night nobody's watching. The system that doesn't wake you up is the one where nearly every fact has a single home and everything else is honestly, visibly downstream of it. One source of truth isn't housekeeping. What it really is, is you refusing to hang a second clock that you'll only have to choose between later on.

In the manifesto, this is tenet (XIV).

Sources

One of a series of field notes on building software for the way minds actually work: tired, distractible, ordinary, and now partly machine. They all lead back to the manifesto behind them, The Shape of the System.