The Decision Index

Every fork the manifesto and its companion put you at, the named alternatives, when to choose each, and what each one costs.

This is the catalogue MANIFESTO.md and its companion THE-SHAPE-OF-THE-WHOLE.md keep circling: the recurring decisions of building software, each written as the choice you actually stand in front of. The tenets give the principles and the guides put them to work by phase; this index cuts the other way, by the decision, so when you are standing at a fork you can find the alternatives, the case for each, the cost of each, and the rule that settles it.

Every entry resolves to the same tie-break, the one the whole corpus runs on: minimise what a tired engineer must hold in their head to make a correct change, subject to a bounded blast radius for anything an attacker or an unlucky caller controls. Made concrete, that is two questions, who controls the input and how wide the blast radius runs, and one spending rule, buy coordination, machinery or complexity only where the property is real and violating it would cost more than the coupling the cure drags in. Nothing here is free; each entry tries to price both sides honestly so the choice is yours to make on your own facts.

Each entry has the same shape: when you hit it, the call, the options (with when to choose and what it costs), how to decide, what to reach for first, the pitfalls, and cross-links to the tenets, the companion's laws, the glossary, and the lifecycle phases where the fork shows up.

Boundaries & contracts - Parse at the boundary vs validate everywhere - Strict in vs tolerant reader at a boundary - Evolve a contract: additive/versioned vs breaking change - Validate on the client vs the server vs both

Consistency & concurrency - Make a check-then-act safe: atomicity vs serialisation vs commutativity - Survive at-least-once delivery: idempotent writes vs dedupe table vs nothing - One source of truth vs a derived copy/cache - Where a rule that spans services lives: coordinator vs single owner vs saga vs quorum vs reconciliation vs eventual - Saga vs distributed transaction (and orchestration vs choreography) - Strong vs eventual consistency (do you even need strong?) - Optimistic (version/CAS) vs pessimistic (locks) concurrency - Fenced lease vs naive distributed lock - Ack/commit after the durable write vs before

Time & ordering - Measure durations and expiry: monotonic clock vs wall clock - Order events across machines: logical clock/fenced sequence vs timestamps

Failure containment & load - Deadline on a cross-boundary wait vs unbounded wait - Retry: capped backoff + jitter vs naive retry vs no retry - Add a circuit breaker vs keep calling - Shared pool vs per-tenant/per-dependency bulkheads - Load shedding (fast reject) vs deep buffer; backpressure - Degrade in tiers vs fail whole - Fail fast (halt) vs degrade (stay up) - Cap caller-driven growth vs leave it (YAGNI) - Compose control loops on purpose vs tune each independently - Build a load-reducing way back vs assume it recovers - Errors as values vs exceptions caught at a boundary - Profile before optimising vs fix on sight - Batch the round-trip (join / IN / vectorise) vs per-row access (N+1) - Long but progressing work: heartbeat + cancellation vs a wall-clock deadline

Resource lifecycle - Scope-bound release (RAII/with/defer) vs manual release - Push/subscribe vs poll vs reconcile to stay in sync - Graceful drain vs hard kill on shutdown - Reference strength on a long-lived watcher: weak vs strong-with-teardown - Atomic temp-write-then-rename vs write in place

Observability & verification - Emit the proving signal vs log everything - Per-component health check vs end-to-end probe - Prove it by fault injection/game day vs trust it works - Test the contract vs the internals - Backup (tested restore) vs failover/redundancy - Pin inputs (lockfile, toolchain, seed, versioned data) vs leave unpinned

Lifecycle & rollout - Expand-contract + flag/canary/ramp vs instant cutover - Separate the decision from the effect (dry-run/apply, four-eyes) vs do it live - Rewrite vs incremental refactor (strangler) - Deprecate with a window vs hard delete - Remove the state vs guard it - Owner + sunset for an artefact vs leave it running - Soft-delete for recoverability vs hard-delete with enforced TTL

Security & trust - Narrow, short-lived credential vs broad/long-lived - Treat crossing input as hostile vs trust it (including the supply chain) - Re-derive identity and authority from a verified token vs trust the payload

Simplicity & structure - Inline/duplicate (locality) vs unify (DRY/one source) - Abstract now vs wait (rule of three) - Make illegal states unrepresentable (types) vs runtime guard - Enforce in code/structure vs in process (runbook/four-eyes) - Name to reveal the load-bearing fact vs name to label

Composition & wholes - Verify disjointness / cells vs assume independent - Defend your part from the whole vs bound your part's effect on the whole

Triage & investigation - Reproduce first vs act on the report or from memory - Rank by blast radius and evidence vs by the reporter's volume - Dispose of an issue explicitly vs leave it in the backlog - One named owner with a due date vs a shared queue or intention list - Time-box with a planned exit vs run it to ground - Drive a throwaway spike to de-risk vs commit on the assumption - Define "done" as an explicit contract vs leave it implicit - Bisect the cause space (falsifiable hypothesis) vs scan or scattershot - Mitigate the symptom now vs find the root cause first - Hand off a diagnosis and a failing test vs fuse the fix and ship

Ship & learn - Block the merge (must-fix) vs name the cost and let it ship (consider) - Verify at production-like scale and environment vs on friendly dev data - Leave a permanent regression test behind vs a one-off manual check - Advance a ramp on instrumented evidence vs on the clock or a hunch - Automated rollback wired to a signal vs a human watching the dashboard - Pre-write a runbook vs invest in observability and reason live - Aim the retro at the system vs at the person - Turn the lesson into a self-rerunning guard vs leave it in prose - Feed findings back as owned issues vs publish a write-up - Gate removal on a usage signal and guard the absence vs trust the belief nothing uses it

Boundaries & contracts

Parse at the boundary vs validate everywhere

You hit this when A request, a row, a queue message or a config file arrives as loose bytes or a loose string, and somewhere downstream your code wants to treat it as a known good value: an email, a positive quantity, a paid order. The check for "is this okay?" is either done once up front or scattered across the call sites that consume the value

The call. When untrusted or unstructured input must become a value your code trusts, do you parse it once at the boundary into a narrower typed value, re-validate the loose value at each use, or trust it unchecked?

Parse once at the boundary. At each real crossing (process edge, deserialisation point, privilege change) you transform the loose input into a narrower type that carries its own validity: an Email, a NonEmptyList, a PaidOrder. The single check produces a value the rest of the code is allowed to assume, and the unchecked form cannot be held downstream. - Choose when: The value crosses a genuine trust or structural boundary, so something hostile or malformed lands here first; Many call sites consume it, or call sites you cannot see yet (other teams, future code) will; A bad shape downstream corrupts state, reaches the database, or drives an actuator; Your language carries types, or you have one ingestion chokepoint (a schema asserted once, pydantic, pandera) that yields a value downstream may trust. - Cost: You pay for the machinery: a distinct type or constructor, plus a parser that is now your most security-critical and most-tested code; In dynamic or tabular work the compiler will not carry the marker for you, so the guarantee degrades to validate-once-at-the-edge and rests on discipline; Over-applied it becomes a cathedral of phantom types nobody can read, where the type machinery costs more than the bug it prevents; It settles shape only; you still need a separate hostility check for size, authority and identity.

Validate (re-check) the loose value at each call site. You keep the original loose type and ask "is this okay?" again wherever it matters, throwing the answer away each time. The value stays the same string or dict it arrived as, and correctness depends on every consumer remembering to ask. - Choose when: The language gives you no way to make the narrowed type stick, and you have no single ingestion point to funnel through; The value has one or two consumers under one author, where a parsed type would be ceremony; The constraint genuinely differs per call site, so there is no single "valid" to parse into; You are retrofitting a check into existing code and cannot change the type that flows through it. - Cost: The check holds at thirty-nine of forty call sites and the fortieth is the incident; nothing structural stops the omission; Every consumer must hold the rule in its head, which is precisely the load the corpus tells you to minimise; Re-validating at every internal hop is a latency tax and tenet III's anti-pattern when the hops are inside your own trust boundary; The proof is discarded each time, so a reader of any one call site cannot tell whether the value upstream was already checked.

Trust it unchecked. You take the input as given and act on it: read userId from the payload, allocate in proportion to a declared length, write the row as received. No parse, no validate. - Choose when: You produced the value yourself on this side of the boundary and no untrusted party touched it; It is already a parsed type handed to you by the boundary that owns the crossing, so re-checking is the clutter tenet IV warns against; A hard structural guarantee downstream (a schema constraint, a verified token re-derived elsewhere) already refuses the bad case; The cost of a wrong value is genuinely nil: throwaway, local, non-privileged. - Cost: Across a trust boundary this is a defect by construction: crashing on malformed input is a free denial of service, and a 2 GB body that parses fine still kills your heap; Trusting an asserted identity or authority is the classic privilege escalation; authentication tells you who, never what they may do; Bugs surface far from the boundary, in whatever downstream code assumed the value was sane, where they are hardest to trace; "It was fine in testing" hides the adversarial input that was never in the test set.

How to decide. Decide by who controls the input and how far a bad value can travel. The governing question for the whole corpus is what a tired engineer must keep in their head to make a correct change, bounded by the blast radius of anything an attacker or an unlucky caller controls. Parsing once at the door pays that off directly: past the boundary the malformed shape has no spelling, so a future call site, including ones that do not exist yet, cannot be wrong about it, and the engineer need not remember the check. That is worth real machinery (a distinct type, a constructor, a parser worth testing) exactly when the value crosses a genuine trust or structure boundary, when many call sites consume it, or when getting it wrong corrupts state or escalates privilege. Spend nothing where the property is not real: between two functions in the same module, under one author and one set of assumptions, a parsed type is clutter and re-validation is a latency tax for no reduction in threat (this is tenet IV's own tension). And keep the two questions apart. Parsing settles shape (tenet III); it says nothing about whether a well-formed value is hostile, too big, unauthorised, or lying about its identity, which is tenet IV's job and survives however good your types are. The one safe case for trusting unchecked input is a value you produced yourself on the same side of the boundary; the moment another party controls it, trust stops being free.

Reach for first. If you do not actually need the constraint, delete the need: accept a narrower input type, or push the check into the place that already owns it. A NOT NULL, a CHECK or a foreign key in the schema refuses the bad row for every writer without a line of application code. When you genuinely have a loose value to tame, parse it once at the real crossing and carry the narrowed type inward. That is the default the rest of the options have to beat.

Pitfalls. - Conflating shape with hostility: a value that parses cleanly can still be too big, unauthorised, or lying about its identity. Parsing is tenet III; bounding size and re-verifying authority is tenet IV, and a real boundary needs both. - Parsing and then continuing to pass the loose value alongside the narrowed one, so call sites can still reach the unchecked form and the guarantee leaks. - Re-validating at every internal in-process hop, buying clutter and latency for no real reduction in threat once you are inside the trust boundary. - Treating the boundary parser as boring glue. It is the one chokepoint every attack must pass through, so it earns the heaviest test coverage, not the least. - Trusting a userId or role straight from the payload because the request asserts it; identity must be re-derived from the verified token. - Building phantom-type machinery so elaborate that the types end up harder to read than the bug they prevent.

Strict in vs tolerant reader at a boundary

You hit this when Data crosses a versioned boundary (an API response, a wire schema, an event on a queue) and the two sides ship on their own clocks. A newer producer is sending a field an older consumer has never seen, or an old client is still posting last year's payload, and you have to decide what the receiving side does when the bytes don't match the shape it was built for

The call. When data arrives at a boundary whose two sides evolve independently, how forgiving should the receiver be about input it doesn't fully recognise, and how disciplined should every emitter be about what it puts on the wire?

Strict in: reject anything outside the contract. The receiver parses the input against the full declared schema and refuses the whole message on any deviation, including fields it has never seen. Unknown means malformed. - Choose when: The data carries authority or money, so a field you don't recognise might be one you were meant to honour: a permission, a price, a signature scope; You control both ends and can deploy them together, so 'unknown' really does mean a bug or an attack rather than a peer on a different version; The blast radius of silently dropping something is worse than the cost of a noisy rejection: a payment that under-charges, an ACL that loses a deny; You are at the security-critical parser and want every byte accounted for before anything downstream trusts it. - Cost: You have turned every additive change into a breaking one: the moment a producer adds a field, every stricter consumer falls over, so you are back to flag-day deployments; An old client sending last year's payload is rejected even though its request was perfectly answerable, which punishes the consumer you can't force to upgrade; Whoever owns the producer now needs your release calendar in their head before they can ship a field, which is exactly the coupling a boundary is meant to remove; Strictness about shape gets confused with strictness about invariants, so you reject safe novelty while feeling rigorous about it.

Tolerant of the unknown: accept and ignore what you don't understand. The receiver enforces the invariants it actually depends on and skips past fields it doesn't recognise rather than failing the message. This is the tolerant reader: a schema can grow additively and nobody upgrades on the same day. - Choose when: The two sides deploy independently and you must serve consumers several versions back: a wire format, a public API, an event sat in a queue for days; The unknown content is genuinely additive and orthogonal to the invariants you rely on, so ignoring it cannot corrupt a decision you make; You want producers to extend messages without a coordinated release, so a consumer on old code is a Tuesday rather than an incident; The format already has additive-only discipline (Protobuf, Avro), so 'unknown field' has a defined, safe meaning. - Cost: Tolerance can quietly swallow a field that did matter: a new 'currency' or 'is_refund' flag the old reader skips, and so it computes the wrong total; Ignored-but-observed behaviour calcifies into an accidental contract (Hyrum's Law), so the freedom you bought erodes as consumers come to depend on quirks you never declared; Bugs surface far from the boundary and late, because the message was accepted and only misbehaved three stages downstream; Tolerance is easy to over-apply to the invariants themselves, accepting a malformed value because rejecting felt unfriendly, which is the failure mode that gives Postel's principle its bad name.

Strict on what you emit, regardless of how you read. The output half of the robustness principle, kept even when you drop the liberal-input half: emit exactly the declared contract, no undeclared fields, no convenient extras, fully conformant every time. It pairs with either reading posture above. - Choose when: Always, as the constant: there is almost no case where emitting sloppy or undeclared output is the right call; You want to keep the wire honest, so the next team's tolerant reader has nothing dangerous to tolerate; You are trying to stop accidental contracts forming: every field you emit is a field someone will eventually depend on, so emit only what you mean; You depend on peers being tolerant and owe them the same discipline you would want back: clean, predictable, additively-versioned output. - Cost: It constrains you, not your callers: you carry the discipline of a pinned contract and a deprecation window while gaining nothing the moment your peers read sloppily anyway; Strict emission alone does not protect you. A strict-out, strict-in system is still a flag-day system, and a strict-out, tolerant-in peer can still drift; Holding the line costs real work: a test that fails the instant a refactor drops or renames a field, plus a versioning ritual for every breaking change.

How to decide. Split the question the way the tenet splits it: be strict about the invariants you actually depend on, and tolerant of the unknown that you merely don't recognise. The only honest way to draw that line is by who controls the input and how wide the damage spreads when you get it wrong. Walk each unrecognised thing through one test. If this field were present and I ignored it, could it change a decision I make about authority, money, identity or safety? If yes, it is an invariant rather than noise, and a stricter rule belongs there: reject, or refuse to proceed until you understand it, because the blast radius of silently dropping it is a wrong total or a lost deny. If no, it is additive novelty from a peer on a different clock, and rejecting it just manufactures a flag-day and loads your release calendar into the producer's head for no correctness gain. The asymmetry is the point. Tolerance is the default for the unknown because the boundary exists precisely to let the two sides evolve apart, yet it is bounded hard at anything an attacker or an unlucky caller controls that feeds a decision you can't afford to get wrong. Spend the strictness where the property is real and violating it costs more than the coupling; spend tolerance everywhere else, so a tired engineer can add a field without convening both teams. And emit strictly whichever way you read, because every field you put on the wire becomes a contract whether you declared it or not.

Reach for first. Before choosing a posture, pin the contract and let the serialisation format carry the rule for you. A schema with additive-only discipline (Protobuf, Avro, or a versioned JSON contract with a test that fails the instant a field is dropped or repurposed) already defines what 'unknown field' means and makes tolerance safe by construction, so you are not hand-rolling forgiveness. With that in place the default falls out: strict on the handful of invariants you depend on, tolerant of unknown additive fields, strict on everything you emit. You only reach for the heavier all-or-nothing strict-in posture at the specific fields where the blast radius justifies it.

Pitfalls. - Conflating strictness about shape with strictness about invariants: rejecting a harmless unknown field while feeling rigorous, when the real job was to reject the malformed price. - Tolerating the unknown so broadly that a field which did matter (a new flag, a new currency, a deny rule) gets silently skipped and corrupts a downstream decision. - Letting ignored-but-observed behaviour harden into an undeclared contract, so the additive freedom you bought leaks away as consumers depend on quirks you never meant to promise. - Keeping the liberal-input half of Postel while dropping the strict-output half: accepting sloppy input and emitting sloppy output, the combination that earned the robustness principle its critics. - Assuming 'unknown means attack' when you don't actually control both ends, turning every peer on an older or newer version into a false rejection. - Putting the tolerance and the invariant checks anywhere but the one boundary parser, so the most security-critical code is scattered and under-tested.

See also. Tenets (XV), (III). glossary: strict in, tolerant of the unknown, tolerant readers. phases: integrating, implementing, planning.

Evolve a contract: additive/versioned vs breaking change

You hit this when A shared interface, an API response, an event schema, a wire format or a library signature needs to change, and other parties already build on its current shape. Some of those parties you cannot force to upgrade: a mobile app three releases back, events already queued, a third-party integration, an SDK cached in someone's build

The call. When an interface that others depend on must change, do you evolve it additively, run versions side by side, gate the break behind a consumer-driven contract test, or simply break it?

Additive + tolerant. Only add optional fields and never remove or repurpose an existing one. Readers ignore what they don't recognise, so a producer can grow the message while old consumers carry on unchanged. The contract widens; it never shifts under anyone. - Choose when: The change is genuinely expressible as an addition: a new field, a new optional capability, a richer payload that older readers can skip; You cannot force consumers to upgrade and don't know who they all are: public APIs, mobile clients, events sitting in a queue, third-party integrations; Both ends already follow tolerant-reader discipline, so an unknown field is skipped rather than rejected; You want the cheapest evolution path, with no coordination and no second copy of anything to maintain. - Cost: Fields accrete. Nothing additive ever deletes, so the schema grows a sediment of optional, half-used fields that the next reader must still reason about; It buys nothing when the change is a true semantic shift: renaming, narrowing a type, changing what a field means, splitting one concept into two. Forcing those into the additive mould breeds parallel fields that say almost the same thing, and a reader who must know which one is authoritative; Tolerance is a real attack surface. Ignoring the unknown must stop at the invariants you depend on, or you wave a malformed input straight into the blast radius. The boundary parser stays strict where correctness rests on the constraint holding; It quietly defers the reckoning: the old shape lives on unless a separate deprecation effort retires it, and this option never starts that effort.

Explicit versioning (v1 and v2 side by side). Stand up the new shape as a distinct version, run both at once, migrate consumers across on their own schedule, then retire the old one on a published window. The break is real, but it is named, dated, and isolated to a version boundary. - Choose when: The change cannot be additive: the new shape contradicts the old, so one payload cannot satisfy both honestly; Consumers upgrade on their own clocks and need a window where both versions answer, rather than a flag day; You can actually retire v1: you can see who still calls it, and you have the leverage and the will to switch them off when the window closes; The contract is worth the ceremony, because a silent break would cost more than running two surfaces for a while. - Cost: Two live surfaces means two of everything: two code paths, two test suites, two sets of bugs, twice the surface an attacker probes, for the whole overlap; The window is the trap. Without a hard expiry and an owner, v1 never dies; you collect zombie versions and the temporary overlap becomes permanent load; Migration is real work pushed onto every consumer, and the slowest one sets your retirement date; It is easy to mistake versioning for free reversibility. A version boundary makes the break orderly, not cheap, and a v3 doubles the bill again.

Break it behind a consumer-driven contract test. Make the breaking change, but pin a test that encodes what each real caller actually relies on, so the build goes red the instant you break one of them. The break is allowed; breaking a consumer silently is not. You learn the blast radius before you ship rather than after. - Choose when: You control every consumer, or near enough, and can land the change across all of them in one coordinated move; Callers are internal and known, so a contract test can name them and stand in for them at build time; You want to reshape the interface without the parallel-version tax, and you will pay for it with the discipline of a test that fails on a real break; The interface is internal infrastructure where a flag day across a known set is feasible. - Cost: It only covers callers the test knows about. An unknown consumer, a mobile build in the wild, a partner you forgot, is invisible to a green build, so the safety is exactly as wide as your inventory of callers and no wider; Someone has to write the contract tests and keep them honest; a stale or thin contract gives false confidence, which is worse than none; It still demands lockstep: when the test goes red you must fix the consumer in the same change, so it scales only while every consumer is a set you can move at once; Pin the contract, not the internals, or the tests break on every refactor and get muted, taking the protection with them.

Just break it. Change the shape and ship. No additive bridge, no second version, no contract gate. Whoever was on the old shape deals with the fallout. - Choose when: You provably own and can redeploy every consumer atomically: one repo, one deploy, nothing persisted or in flight on the old shape; The contract is pre-release, or internal with a single known caller you control in the same change; The thing on the old shape genuinely has no live dependants, and you can prove that rather than hope it; Carrying any compatibility machinery would be speculative clutter for a boundary nobody crosses. - Cost: If a single consumer you can't force to upgrade is out there, this is an outage you chose; an old consumer on the old contract is normal traffic, not a fault; The failure is silent and remote: it surfaces in someone else's system, often hours or days later, when a queued event or a stale client finally arrives; It assumes a closed world you rarely have. Hyrum's Law says some caller depends on behaviour you never published; Cheapest today, dearest the day it is wrong, and the bill lands on whoever is paged, not on whoever broke it.

How to decide. Decide by who controls the consumers and how wide a silent break would reach. The expensive options here, a second live version and a maintained contract suite, are machinery; spend them only where a break is real and where breaking quietly would cost more than the machinery drags in. First test whether the change is honestly additive: if the new shape can be a strictly wider one that old readers skip, evolve additively and stay strict only about the invariants correctness rests on, because that costs nothing in coordination and keeps every existing consumer running. If it is a true semantic shift, the question becomes who is on the old shape. When every consumer is known and you can move them in one change, a consumer-driven contract test is the lighter cure: it lets you reshape freely and turns the build red on a real break, but its coverage stops exactly at the callers it knows, so it protects a closed set and nothing beyond it. When consumers are outside your control, whether mobile, public, queued or third-party, the blast radius is open and a silent break is an outage you chose; pay for explicit versioning, run both surfaces through a window with an owner and a hard expiry, and migrate them across. Reserve just-break-it for the closed world you can actually prove, not the one you hope you have. The whole fork reduces to one rule: the less you control the input and the wider the break travels, the more you must keep the old shape working while the new one ships, because the coordination you save by breaking is paid back, with interest, by whoever the break pages.

Reach for first. Ask whether the change can be expressed as an addition. The cheapest correct answer is almost always additive and tolerant: add the optional field, never remove or repurpose an existing one, keep the reader strict about invariants and quiet about the unknown. It needs no second version, no migration and no coordination, and it keeps every consumer you can't force to upgrade running unchanged. Only when the change is a genuine semantic break, where one payload can't honestly satisfy old and new at once, do you reach for the heavier options.

Pitfalls. - Calling a semantic change additive: adding a near-duplicate field instead of renaming, so every reader must now know which of two fields is authoritative. - Opening a v2 with no owner and no expiry on v1, so the temporary overlap becomes a permanent two-surface tax and a drawer of zombie versions. - Trusting a green contract test as proof of safety when an unknown or unupgradeable consumer was never in the test's inventory of callers. - Tolerating the unknown so liberally that malformed input is waved past the invariants you actually depend on, turning Postel into an attack surface. - Pinning contract tests to internal shape rather than observable behaviour, so they break on every refactor, get muted, and stop catching real breaks. - Treating everyone-will-upgrade as the deprecation plan instead of designing the server to keep serving consumers several releases back.

Validate on the client vs the server vs both

You hit this when A user-supplied value (a form field, a query parameter, an uploaded file) needs checking, and you own both a client and a server. The browser already shows a friendly red message when the field is wrong, so it is tempting to call the job done there, or to wave the value through and let the database complain later

The call. When a value crosses from the user into your system, where do you enforce that it is acceptable: in the client, on the server, or in both places?

Server only (the security boundary). The only enforced check runs server-side, where the request actually lands. The client submits whatever the user typed, and the server parses it, bounds its size, and authorises the actor before acting on it. - Choose when: The value affects anything that matters: it is persisted, it grants access, it is billed against, or it sizes an allocation; An attacker or an unlucky caller controls the input and the blast radius of a bad value is wide; You want exactly one place that decides what is acceptable, so a tired engineer changing the rule changes it once; The client is a thin form, a public API, a mobile app you do not ship, or anything that can be bypassed by talking straight to the endpoint. - Cost: The honest user only learns a field is wrong after a round trip, which feels sluggish on slow links and long forms; Each trivial mistake (an empty required field, a malformed email) costs a request and a server cycle; Error presentation has to be threaded back from server responses into the form, which is more wiring than a local check.

Both: client for feedback, server as the real gate. The client validates for immediate feedback and the server validates again as the enforced check. These are two different jobs sharing one rule: the client smooths the path for the honest user, while the server stays the gate that decides whether anything actually happens. - Choose when: The form is long or multi-step and round-trip latency would make it painful to fill in; You can keep the rule in one shared, declared place (a schema, a validator) that both sides run, so the two checks cannot quietly drift apart; The value still reaches the server gate: the client check is an optimisation on top, never a substitute; Fast feedback measurably improves completion or reduces support load. - Cost: The same rule now lives behind two runtimes; unless it has a single owner, the client and server slowly disagree and you debug a phantom; More code and more tests for the same invariant; The client check can lull a reviewer into thinking the value is safe, so the server gate gets weakened or skipped under deadline.

Client only. The browser is the only thing that checks the value. The server trusts whatever arrives and acts on it. - Choose when: Nothing crosses a trust boundary: the value never leaves the client, or it is purely cosmetic with no security, money, or persistence behind it; A local prototype, a toy, or a single-user offline tool where there is no adversary and no server worth defending; The cost of a bad value is a redraw the user can fix themselves. - Cost: Worthless as defence: an attacker skips the browser entirely and posts straight to the endpoint, so the only protective check is the one you did not run; The moment the value is persisted, grants authority, or sizes work, a free crash or a corrupted record is one curl command away; Tends to rot into a security hole, because the form looks validated and nobody notices the server never re-checks.

How to decide. Decide by who controls the input and how far a bad value can reach. The browser is the user's machine, so anything it checks is advisory: a hostile or buggy caller posts straight to the endpoint and your client check never runs. That makes the server the only real gate for anything that is persisted, grants authority, is billed against, or sizes an allocation, and the gate must re-derive identity and authority there, not just shape, because being logged in is not the same as being allowed. So the server check is non-negotiable whenever the blast radius is wide. The client check is a separate, optional good: it buys the honest user fast feedback and saves round trips, and it costs you nothing in safety as long as it stays on top of the server gate rather than in place of it. The one real trap in doing both is duplication, since the same rule behind two runtimes will drift unless it has a single owner that both sides derive from. Spend the coordination of a shared validator only where the feedback genuinely helps (long or fiddly forms); for a one-field box the round trip is cheaper than keeping two checks honest. Never spend the second check as a replacement for the first: client-only is the one answer that is wrong for anything an attacker or an unlucky caller can reach.

Reach for first. Server-side validation, always, as the one enforced check. Add a client check only when slow feedback is a real problem, and when you add it, drive it from the same declared schema the server uses so there is one owner of the rule.

Pitfalls. - Treating the client check as protection: the form turns red, so the server quietly trusts the value and acts on it. - Validating shape but not authority on the server: the value is well-formed, but the actor was never authorised to submit it for that resource. - Letting two hand-written copies of the rule drift, so a value the client accepts the server rejects (or worse, the reverse). - Re-validating the same parsed value at every internal hop after the boundary, mistaking it for the boundary check and paying latency for no reduced threat. - Bounding shape but not size: a well-formed payload of attacker-chosen length sails through both checks and then exhausts memory.

Consistency & concurrency

Make a check-then-act safe: atomicity vs serialisation vs commutativity

You hit this when You have written something shaped like if not exists: create, if balance >= amount: debit, or "reserve the last seat". It reads as one decision on one line, but at runtime it is a read and then a separate write, and two actors can slip in between them. The window is invisible in review and only turns devastating under real concurrent load

The call. When a check and the act that depends on it run on shared state that other actors can touch, how do you close the window between them: make the pair indivisible, route it through a single owner, or make the act harmless to race?

Atomicity (one indivisible step). Collapse the read and the write into a single operation the storage layer guarantees is indivisible: a unique constraint or INSERT ... ON CONFLICT, UPDATE ... WHERE balance >= amount checking rows-affected, or an optimistic compare-and-swap on a version with a bounded retry. The check becomes a condition the write itself enforces, so there is no gap to interleave in. - Choose when: The shared state lives behind a primitive that already offers atomicity: a row, a unique key, a versioned record, a hardware CAS; The contended operation is small enough to express as one conditional write, not a multi-step workflow forced to be atomic; Throughput matters and you cannot have every actor queue behind one owner; The natural outcome of a lost race is 'retry' or 'someone else won', which the caller can absorb. - Cost: Optimistic CAS pushes a retry loop onto the caller; under heavy contention it can livelock or starve, so you must back off and bound it; It only stretches to what the engine makes atomic in one statement, so the moment the invariant spans two tables or two services a single conditional write cannot hold it; The failure signal (zero rows affected, version moved) is easy to write and easy to forget to check, and a forgotten check looks exactly like success; Optimistic concurrency is cheap when conflicts are rare and expensive when they are common, so it quietly degrades as the hot row gets hotter.

Serialisation (one owner or a consistent lock order). Remove the concurrency rather than tolerate it. Funnel every operation on the contended state through a single owner (one actor, one partition, one queue consumer, one leader), or take a lock that all writers acquire in the same order. With only one actor in the window at a time, check-then-act is sequential again and correct by construction. - Choose when: The invariant spans several reads and writes that no single atomic statement can cover; You already have a natural owner: a per-key partition, a per-account actor, a leader for that resource; Correctness is worth more than raw parallelism on that slice of state, and contention on it is modest; You want the simplest thing to reason about, since sequential code under one owner needs no concurrency proof at all. - Cost: The owner is a throughput ceiling and a single point of contention; everything for that key waits behind it; A distributed lock or lease is itself a check-then-act across the network: the holder can pause, the lease can expire, and the holder writes anyway, so you need a fenced token the resource validates rather than trusting the acquisition; Multiple locks invite deadlock unless every caller takes them in the same order, which is a global discipline that is hard to enforce and easy to break in review; Routing through one owner adds a real component (a queue, a leader election, a lock service) with its own failure and operability cost.

Commutativity (make the act harmless to race). Design the operation so order and repetition do not matter, then the interleave does no damage. Key the write by a stable id and make creates upserts, model the change as a set union or a monotonic counter, or use a structure that converges regardless of the order updates arrive. There is still a race; it just has no bad outcome. - Choose when: The act can be expressed as set-add, max, or another genuinely order-independent operation; You are fighting redelivery and retries (at-least-once messaging, double-clicks, re-run jobs) as much as concurrent interleave; You want offline or multi-writer behaviour where coordination is impossible or too slow; 'Applied twice' and 'applied once' can be made to yield the same end state. - Cost: Few real operations are naturally commutative; forcing one often distorts the data model into something the rest of the system finds awkward to read; Idempotency keys, dedupe tables, and reconciliation are real machinery with storage and expiry, and the key must be stable end to end or the protection silently lapses; Counters and convergent structures buy order-independence by giving up a strict global total or a hard limit, so 'never oversell the last seat' is exactly the invariant they cannot promise; It changes the contract callers see (eventual convergence, not an immediate authoritative answer), which can push complexity outward to every reader.

How to decide. Decide by who controls the contended state and how wide the damage runs when a race is lost. The starting question is whether two actors can truly interleave; if the state has a single writer the window does not exist and any machinery is pure waste, but treat single-threadedness as something you enforce and prove, because it is the assumption that turns out wrong most often at scale. Where contention is real, scope the cure to the invariant. If the rule fits one conditional write, atomicity is the cheapest correct answer: it adds no component and keeps the guarantee in the layer that can hold it. When the invariant spans several operations or several stores, it is an invariant no single component owns, and you are choosing between paying for serialisation (a single owner, with its throughput ceiling and, across a network, a fenced lease so a paused holder cannot write through a stale lock) and paying for commutativity (machinery and a weaker contract so the race stops mattering). Let the blast radius break the tie. For money, inventory, and any hard limit an attacker or an unlucky caller can push against, prefer the option that gives a strict, immediate, authoritative answer (atomic conditional write, or serialised owner), because a converged-eventually structure cannot promise 'never oversell the last seat'. Spend the coordination only where the property is genuinely load-bearing and a violation would cost more than the coupling the cure drags in; everywhere else, the lighter answer that keeps the next tired reader from holding a concurrency model in their head is the right one.

Reach for first. First ask whether two actors can genuinely interleave here at all. If the state is touched by only one writer (one process, one partition owner, no overlapping cron), there is no window and any locking is dead weight; prove the single-writer property and move on, treating it as something you enforce rather than inherit. Where there is real contention, reach for atomicity expressed in the storage engine you already have: a unique constraint, INSERT ... ON CONFLICT, or UPDATE ... WHERE <condition> checking rows-affected. It is one statement, it adds no new component, and it pushes the guarantee down to the layer that can actually keep it. Only when the invariant outgrows a single conditional write do serialisation and commutativity earn their keep.

Pitfalls. - Treating SELECT then INSERT/UPDATE as atomic because it sits on adjacent lines; the runtime makes no such promise and the two steps can interleave. - Running an optimistic CAS or UPDATE ... WHERE and not inspecting rows-affected or the swap result, so a lost race reads as success and the violating write is silently dropped or applied anyway. - Trusting a distributed lock or lease as if holding it were a fact: a GC pause or expiry can revoke it underneath you, and without a fence token the resource will accept the stale holder's write. - Reaching for heavy locking in code that is genuinely single-writer, paying contention and operability cost to close a window that was never open. - Calling an operation idempotent without a stable end-to-end key, or letting the dedupe record expire before the retry arrives, so 'apply once' and 'apply twice' quietly diverge. - Choosing a convergent counter or CRDT for an invariant that needs a hard ceiling, then discovering it cannot refuse the overselling write.

See also. Tenets (IX), (X). the companion's third law. glossary: check-then-act is a race, compare-and-swap, fenced lease, two operations pretending to be one. phases: implementing.

Survive at-least-once delivery: idempotent writes vs dedupe table vs nothing

You hit this when A message, webhook or job runs the same logical attempt more than once: an ack was lost, a broker redelivered, a client retried a timed-out request, a worker crashed mid-batch and the scheduler re-ran it. The second arrival carries the same intent as the first, and your handler has no idea it has seen it before

The call. When the same attempt can be delivered twice, how do you stop the second delivery from charging, sending or incrementing again?

Idempotent write (key by a stable id). Carry a stable, attempt-level id through the handler and let the write itself absorb the repeat: an upsert keyed by that id, an INSERT ... ON CONFLICT DO NOTHING, an UPDATE guarded by a precondition. The second delivery hits the same key and converges to the same row instead of producing a second effect. - Choose when: The effect lands in a store that can enforce a key for you: a unique constraint, an upsert, a conditional update; The attempt has a natural stable id you can thread end to end (order id, payment intent, source event id), or the caller will supply an idempotency key; The whole effect is one write, or a set of writes you can make converge under the same key; You want the dedupe to live in the same transaction as the effect, so there is no window where one committed without the other. - Cost: You must find or invent a stable id and carry it intact through every layer; a regenerated uuid or a timestamp defeats the whole thing; Only as atomic as the underlying write: two writes plus a check are still a race (IX) unless the constraint or transaction collapses them into one step; Multi-step effects (charge a gateway, then write a row, then publish) are not one write, so a single key does not cover the external call; Schema work up front: the column, the constraint, the conflict clause, and the discipline to keep them in step as the handler grows.

Dedupe table (record processed ids, drop repeats). Keep a separate table of ids you have already handled. On each arrival, insert the id and run the effect; if the id is already present, drop the message without re-running. Rows expire after a retention window long enough to outlast any plausible redelivery. - Choose when: The effect cannot carry its own key: it is an external call, a fan-out, or several stores with no single constraint to lean on; One dedupe boundary must guard many different effects funnelled through the same consumer; You can record the id in the same transaction as the effect, or you have a clear plan for the gap when you cannot; You need an explicit, queryable record of what was processed, for audit or replay. - Cost: Real storage that grows with throughput, plus an expiry policy you have to size and run: too short and a late redelivery slips through, too long and the table bloats; The check-and-insert is itself check-then-act (IX); under concurrency it must be atomic (insert, and let the unique violation signal the duplicate), or two workers both pass the check; If the dedupe row and the effect are not in one transaction, a crash between them either re-runs the effect or drops it for good; Another moving part to operate, monitor and reason about when you are debugging a missing or doubled message.

Do nothing, assume single delivery. Run the effect on every arrival and rely on the transport never delivering twice. No key, no table, no guard. - Choose when: The effect is genuinely safe to repeat: a pure read, an overwrite to the same value, a set-to-constant that converges by nature; A repeat is provably impossible on this path, not merely unlikely: truly single-shot, single-run, with no retry layer anywhere above it; The cost of a rare double is trivial and self-correcting (a duplicate log line, a metric a later reconcile fixes); You are early enough that the machinery would be speculative, and the blast radius of a double is small and contained. - Cost: At-least-once is the default of almost every queue, retry library and load balancer, so single delivery is an assumption that tends to be wrong exactly when it hurts; The failure is silent in review and surfaces in production as a double charge, a double email, or a stock count drifting low; 'Safe to repeat' is fragile: a later caller adds a retry, or a value-overwrite quietly becomes an increment, and the path that was harmless now corrupts; No record of what happened, so the first sign is a customer complaint rather than an alert.

How to decide. Start from what the effect touches and who controls how often it fires. If the effect is a read, or an overwrite that converges to the same value, doing nothing is correct and the machinery is pure waste; do not pay for a property you already have. The moment a repeat mutates money, external state or a running total, the only open question is where the dedupe lives, and that is settled by whether the effect can carry its own key. If it lands in one store that can enforce a constraint, push the dedupe into the write: an upsert or conditional insert keyed by a stable id is the smallest thing a tired engineer has to hold in their head, because the effect and its protection are the same atomic step with nothing to drift apart. Reach for a dedupe table only when the effect cannot carry a key (an external call, a fan-out, several stores) or when one boundary must guard many effects, and then accept that you have bought a second piece of state whose check-and-insert must itself be atomic (IX) and whose retention you have to size against the longest redelivery you can suffer. The deciding tension is the gap between the dedupe record and the effect: in one transaction they are a single fact to reason about; in two they re-import the race you were trying to kill. So spend the table only where the key cannot ride with the write and the cost of a double is larger than the storage, the expiry policy and the extra moving part the table drags in. Since the transport cannot give you real exactly-once, you are faking it at the consumer; make the seam where you fake it obvious and keep its blast radius to the one effect it guards.

Reach for first. Look for a stable id you already have (order id, payment intent, source event id) and make the write itself absorb the repeat, with a unique constraint or an upsert in the same transaction as the effect. Having no id and no constraint to lean on is what justifies a dedupe table; never reach for the table while the effect can simply key itself.

Pitfalls. - Generating the idempotency key inside the handler instead of carrying the caller's: a fresh uuid per delivery makes two arrivals look like two attempts, so dedupe never fires. - Check-then-insert on the dedupe table as two statements: under concurrency both workers pass the check and both run the effect; let a unique constraint and the insert do the deduping atomically (IX). - Writing the dedupe row in a different transaction from the effect, leaving a crash window that either double-applies or silently drops. - Keying the write but leaving the external call (gateway, email, webhook) outside the key, so the row is idempotent and the side effect is not. - Sizing the retention window shorter than the longest redelivery the broker can produce, so a late duplicate sails through an empty table. - Calling the transport 'exactly-once' and trusting it, when what you actually bought is at-least-once plus whatever deduplication you remembered to add.

One source of truth vs a derived copy/cache

You hit this when A fact you already store in one place is needed somewhere else: a hot read is slow, a list query joins five tables, the UI must respond before the server replies, or a second service keeps asking you for the user's plan. The cheap move is to put the fact in a second place. The fork is whether that second place is allowed to write, and whether anyone has said how stale it may get

The call. When a fact needs to live in more than one place for speed or convenience, do you keep a single authoritative owner and derive every other copy read-only, declare a governed derived copy with an invalidation path and a staleness budget, or let a second copy quietly appear?

Single source of truth, derive everything read-only. One owner is allowed to write the fact. Every other appearance of it is computed on demand or kept by a mechanism the owner drives: a materialised view, a projection rebuilt from an event log, a read replica fed by replication. No downstream copy is writable, so by construction it cannot disagree with the owner; the worst it can be is behind. - Choose when: The fact has a clear owner and the derivation is cheap, or the platform maintains the copy for you so no hand-written sync exists; A wrong answer costs more than a slightly slow one, so you would rather pay read latency than risk two beliefs; You want the property to hold without anyone remembering anything: changing the owner updates every view by construction; Two copies disagreeing at 3am would be a real incident, not a cosmetic glitch. - Cost: You pay the derivation on every read, or you pay for the machinery (replication, view refresh) that keeps the copy honest; Read load lands on the owner, which can become the bottleneck you were trying to relieve; Some shapes genuinely cannot be derived cheaply: a cross-service aggregate, or a feed ranked by data three systems away; A replica still lags; read-your-own-writes is not free, and pretending the derived view is instantly fresh is its own bug.

Declared, owned derived copy with an invalidation path and a staleness budget. You keep a second copy on purpose and govern it. It has a named owner, a defined way to invalidate or refresh it, and an explicit limit on how far behind the source it may fall. For deliberate divergence (optimistic UI, an in-progress draft) you also fix the reconciliation point where the server confirms or you roll back. The copy is still derived; it is just maintained by a mechanism you wrote and can point to. - Choose when: The read is genuinely hot or expensive and deriving on demand will not meet the latency or load you need; A bounded amount of staleness is acceptable and you can put a number on it, so the divergence is something you can reason about; The UI must respond before the server can, and you have a confirm-or-rollback path to settle it; You can name who owns the copy and where it reconciles, and write that down so the next engineer does not have to reverse-engineer it. - Cost: Invalidation is one of the genuinely hard problems; every write path to the source now has to remember the copy, and a missed one drifts silently; You carry a staleness budget as a live contract: someone has to set the number, monitor it, and own the alert when the copy exceeds it; Optimistic divergence adds a reconciliation and rollback path, which is real code with its own failure modes (the rollback that fails, the confirm that never arrives); More moving parts for a tired engineer to hold: to change the fact correctly they must now know the copy exists, how it is invalidated, and how stale it is allowed to be.

Undeclared second copy (a bug). The same fact ends up writable in two places with nobody having decided it should be: two denormalised columns updated by different code paths, a client store mutated directly alongside the server, two services each holding a writable copy of the user's plan. There is no named owner, no invalidation path, no staleness budget, and no reconciliation point. It looks like the governed copy above and behaves like a fault waiting to fire. - Choose when: Never on purpose. It is listed only so you can recognise it: this is what the second copy decays into when you skip declaring the owner, the invalidation, and the budget. - Cost: When the two copies disagree there is no answer to which one is right or why the other exists, so every divergence becomes an investigation; The drift is silent until a user sees a contradiction, by which time you cannot tell when it started or which writes were affected; No owner means no one is responsible for reconciling it, so it is fixed by whoever is unlucky enough to be on call; It is indistinguishable at a glance from a legitimate derived copy, so it survives review precisely because it looks deliberate.

How to decide. Decide by who is allowed to write the fact and how far a stale read can spread before it does damage. If the derivation is cheap or the platform will keep the copy honest, keep one owner and derive read-only: you spend nothing on coordination and the property holds by construction, which is the smallest thing for a tired engineer to carry. Reach for a declared copy only when deriving on demand genuinely will not meet the latency or load, and then pay the full price honestly: a named owner, an invalidation path on every write to the source, and a staleness budget with a number on it, plus a reconciliation point wherever you let the copy diverge first (optimistic UI, a draft). Set the budget by blast radius. A copy that feeds a dashboard can lag minutes; a copy that gates a payment or a seat sale can lag almost nothing, and at that point the honest question is whether you should be holding a second writable copy at all rather than a single owner others read through. The same question gets sharper across a service boundary, where the same fact made writable in two services is a distributed invariant nobody owns, and the home for it is one writer others subscribe to, not two writers and a hope. The line that decides it: you may duplicate a fact only once the duplicate is declared, invalidated and reconciled, because the coordination it costs is cheaper than the 3am incident where the system holds two beliefs and no one can say which is real. Short of that, the copy is not an optimisation; it is a bug you have not hit yet.

Reach for first. Remove the need before you manage it. Often the second copy exists only because the read off the owner is slower than someone assumed: an index, a tightened query, or a join the owner already supports makes the derivation cheap enough that there is nothing to cache. Where the platform can keep the copy honest for you (a database materialised view, a read replica fed by replication, a query-cache layer with built-in invalidation), reach for that next, because it gives you the speed without a hand-written sync path to get wrong. Only when neither removes the need do you start declaring a copy of your own.

Pitfalls. - Treating an undeclared copy as if it were a cache: it has no owner, no invalidation and no budget, so it carries the cost of duplication with none of the governance that would make it safe. - Setting a cache with no staleness budget, so 'might be stale' is a vague worry no one can act on instead of a contract with a number and an alert. - Adding the second copy to one write path but forgetting one of the others; the forgotten writer is exactly where the drift starts, and it stays silent until a user sees the contradiction. - Optimistic UI with no reconciliation point: the client shows the new value, the server rejects it, and now the parallel copy you sync by hand has quietly become the source the user trusts. - Caching a fact whose blast radius is too large for any staleness at all (a balance, a permission, a seat) when what you actually needed was a single owner read through, not a copy.

See also. Tenets (XIV). the companion's third law. glossary: one source of truth, declared divergence, a parallel kingdom of truth, staleness budget. phases: implementing, operating.

Where a rule that spans services lives: coordinator vs single owner vs saga vs quorum vs reconciliation vs eventual

You hit this when A rule has to hold across three or more services and no single one of them can see far enough to enforce it: a seat sold at most once across many booking nodes, an account never overdrawn when the debit and the limit live in different stores, an order shipped or refunded but never both when two workflows act in ignorance of each other. Today the rule is upheld only by every service happening to behave, and you need to give it a home before the day they don't

The call. When a rule spans several services and none can enforce it alone, where do you put the authority that keeps it true: a single owner, a coordinator, a saga, a quorum, a reconciliation sweep, or an explicit eventual-consistency design?

Single owner. One service owns the spanning fact outright and is the only writer; every other party reads a derived view or asks the owner to decide. The cross-party rule collapses into a local rule inside one process. - Choose when: The fact can be made to live in one place without dragging half the system's data along with it; The decision is cheap and fast enough that routing it through one owner adds no meaningful latency; The other parties are content with a read replica or a request-reply, not their own writable copy; You can enforce the rule with a local transaction or a compare-and-swap once it is in one hand (tenet IX). - Cost: The owner is a single point of failure for that fact: if it is down, nobody can make the decision; Everyone now depends on it, which is the shared substrate and coupling Law I warns about; It can become a write bottleneck if the decision is hot, since all of it serialises through one node; Collapsing the fact into one place sometimes means moving data that genuinely belonged elsewhere, distorting the boundaries to suit the invariant.

Coordinator. A dedicated component serialises the decision: everyone asks it, it decides one at a time, and the rule holds because there is exactly one authority making the call across all parties. - Choose when: The fact cannot be made to belong to any one existing service, so the authority has to be its own thing; You need a strict global order of decisions and can tolerate routing them through one place; The invariant is real and unrepeatable, and you have accepted paying for a new piece of infrastructure; A short-lived fenced lease (tenet IX) through the coordinator is enough to make each decision safe. - Cost: You have built a fresh single point of failure and a new shared substrate, the cure becoming the disease (Law I); Under a partition it must choose consistency over availability: callers that cannot reach it simply wait (CAP); Every decision pays a network round trip to one place, which caps throughput and adds latency; It is a new thing to run, monitor, scale and reason about, and a tired engineer now has to hold it in their head on every change to the rule.

Saga. Implement the spanning operation as a sequence of local steps, each with a compensating action that undoes it if a later step fails, so the whole reaches either complete success or a clean, compensated rollback without one transaction holding locks across every service. - Choose when: The invariant is really a business operation wearing several services: create the order, charge the card, reserve the stock; Each step can be undone by a sane compensation (refund, release, cancel) that the business actually accepts; You want each service to keep its own store and stay independently deployable, avoiding a distributed transaction; The operation tolerates a brief window where it is partway done and visibly so. - Cost: The compensations are themselves irreversible-decision code (tenet XI) that can fail halfway, so you have traded one hard problem for a pile of undo logic that must all be correct; There is a real interval where the system is partially applied and an observer can see inconsistent state; Some steps have no honest compensation: you cannot un-send an email or un-ship a parcel, only paper over it; Getting every failure path and every retry to be idempotent (tenet X) is genuine, ongoing work that grows with each step.

Quorum / consensus. A majority of nodes must agree before anyone acts, so a single cross-party fact (who holds the lock, what value committed) has one answer that survives individual failures. Paxos or Raft turn this into a primitive you build the invariant on. - Choose when: The fact genuinely cannot tolerate two answers, ever, even for a moment: leadership, locking, a committed ledger entry; You need the authority itself to survive node failures, not just be fast in the happy path; The decision rate is low enough to absorb the cost of agreement on every write; You are prepared to refuse service during a partition rather than risk divergence. - Cost: It is the heaviest machinery here: a consensus group to run, operate and understand, far more than most invariants warrant; Every committed decision pays the latency of convincing a majority, so throughput is bounded by agreement, not by the work; Under a partition the minority side cannot act at all (CAP), so consistency is bought squarely at the price of availability; It is easy to over-reach for: reaching for consensus on a fact that would have tolerated eventual consistency is a large, permanent tax.

Reconciliation sweep. Let the parties act independently and accept that they will drift, but run a level-triggered, convergent reconciler that periodically re-reads the actual state, finds violations of the rule, and repairs them after the fact. The rule holds eventually rather than at every instant. - Choose when: The violation is detectable after the fact and cheap to repair: cancel a double booking, claw back a stray charge; A bounded window of inconsistency is genuinely acceptable to the business; You would rather keep the parties decoupled and available than serialise every decision through one authority; You want a safety net under an event-driven design anyway, since events are lossy and a sweep heals what they miss (tenet VIII). - Cost: The rule is provably violated between sweeps; you are betting nobody acts irreversibly on the bad state in that gap; Reconciliation logic is real software that must itself be correct, and a bug in the repairer can make drift worse; Some violations cannot be undone once observed, so this is no good for anything genuinely unrepeatable; If the sweep ever falls silent the drift grows unchecked, so it needs its own monitoring and a named staleness budget.

Eventual consistency (declared, no active repair). The honest admission that the fact is allowed to disagree across parties for a while, paired with the promise that copies converge once updates stop, often via convergent data types that need no coordinator at all. - Choose when: The rule is one of the many that only feels like an invariant and actually tolerates temporary divergence; The data structure can be made to converge on its own (counters, sets, last-writer-wins where that is acceptable); Availability and partition tolerance matter more than instantaneous agreement; Divergence is bounded and self-correcting, so an active sweep would be ceremony. - Cost: You must name the convergence point out loud, because eventual consistency with no declared reconciliation is just unbounded inconsistency under an optimistic name; Convergence semantics (which write wins, how conflicts merge) are subtle and easy to get quietly wrong; Callers downstream have to be written to tolerate stale and conflicting reads, which pushes complexity outward; It is the wrong tool the moment the fact is money, safety, or anything you cannot take back.

How to decide. Decide by who controls the decision and how wide the damage spreads if the rule breaks for an instant. First ask whether it is an invariant at all: most facts people reflexively protect will tolerate divergence, and for those the cheap, available answers (declared eventual consistency, or a reconciliation sweep as a safety net) are correct and the strong machinery is pure tax. Reach for serialising authority only where a violation is genuinely costly and, crucially, irreversible. If the bad state can be detected and cheaply repaired after the fact, reconcile; if it can be undone by a sane compensation, run a saga; if it can neither be undone nor tolerated for a moment, you must serialise the decision, and the only question left is where. Prefer a single owner if the fact can honestly live in one existing service, because that turns a cross-party rule into a local one and is the smallest thing a tired engineer must hold in their head. Add a standalone coordinator only when no service can own it, and reach for a quorum only when the authority itself must survive node failure. Both are a fresh single point of coupling, and both sign you up, the moment the network splits, for refusing service rather than risking two answers (CAP); that is the bounded blast radius you are buying, paid in availability. Spend that coordination only where the invariant is real and breaking it would cost more than the new shared substrate, the lost availability, and the standing load on every engineer who now has to reason about the coordinator on each change. Above all, never leave the rule implicit: an undeclared spanning invariant is the swallowed error written at the scale of the whole system.

Reach for first. Try to make the rule not span at all: pull the fact into a single owner so a cross-service invariant becomes a local one enforced with an ordinary transaction or a compare-and-swap (tenet IX). If even that is more than you need, and the fact tolerates drift, declare eventual consistency with a named reconciliation point and let a sweep heal it. Spend a coordinator, saga or quorum only after a single owner is genuinely impossible and the rule genuinely cannot tolerate the gap.

Pitfalls. - Leaving the invariant implicit, upheld only by every service happening to behave, which is the swallowed error at system scale and the failure this whole fork exists to prevent. - Reaching for consensus or a coordinator on a fact that would have tolerated eventual consistency, paying a permanent availability and operational tax for a property you never actually needed. - Calling a design 'eventually consistent' with no named reconciliation point, which is just unbounded inconsistency with a flattering label. - Writing a saga whose compensations cannot actually undo the step: you cannot un-ship a parcel or un-send an email, so a refund is a new fact, not a rollback. - Adding a coordinator to cure inconsistency and forgetting you have just created a fresh single point of failure and shared substrate (Law I). - Treating the reconciliation sweep or the coordinator as fire-and-forget: when either falls silent, drift or outage grows unwatched, so each needs its own monitoring and a staleness budget.

Saga vs distributed transaction (and orchestration vs choreography)

You hit this when A single business operation, create the order, charge the card, reserve the stock, has to span three services and complete as a unit. No service owns the rule, and a partial outcome (charged but unreserved, shipped yet refunded) is a real and expensive bug. You have already accepted that the steps must be sequenced; what is unsettled is how failure between them is made safe

The call. Do you hold a global lock to make the cross-service operation atomic, or commit each step locally and undo with compensations on failure, and if the latter, who drives the sequence?

Distributed transaction (2PC/XA). A coordinator runs a two-phase commit across every participant: it asks each to prepare (durably promise it can commit, and hold locks), and only if all vote yes does it tell them to commit. The operation is atomic and isolated, so no one observes a half-done state. - Choose when: The invariant is genuinely unrepeatable and a partial outcome corrupts money or safety, and no compensation can honestly undo a committed step; All participants are resources that actually speak a transaction protocol (one RDBMS, an XA-aware broker), not arbitrary HTTP services; Contention is low and transactions are short, so the locks held across the prepare window do not throttle throughput; You can accept the system refusing to act during a partition rather than risking two answers. - Cost: The coordinator is a fresh single point of failure and a new shared substrate (Law I); if it dies between prepare and commit, participants sit locked and in doubt until it recovers; It picks consistency over availability the moment the network splits: the CAP bargain is signed in advance here, not stumbled on in the incident; Locks held across services for the whole prepare-commit window serialise unrelated work and crater throughput under load; Most real participants (third-party gateways, REST services) cannot enlist in XA at all, so across a typical microservice estate the option is simply unavailable.

Saga (local commits + compensations). Each step commits locally and immediately, with no global lock. If a later step fails, the saga runs a compensating action for each completed step to walk the world back to a clean state. The whole reaches either full success or a fully-compensated rollback, and the system is eventually consistent in between. - Choose when: Participants are independent services that cannot share a transaction, which is the normal case once an operation crosses service boundaries; The intermediate inconsistency is observable but tolerable for a bounded window, and a reconciliation sweep can mop up anything the compensations miss; Every step has a real, honest undo: release the reservation, refund the charge, cancel the order; Throughput and availability matter more to you than instantaneous global isolation. - Cost: The compensations are themselves irreversible-decision code (tenet XI) that can fail halfway, so you now have to get every undo right, and test the undo of the undo; There is no isolation: other readers can and will observe the half-applied state, so downstream logic must cope with an order that is charged but not yet reserved; Some effects do not truly compensate (the email is sent, the missile is launched), and a refund is not the same as never having charged, which customers notice; Steps and compensations must be idempotent (tenet X), because retries and redelivery will replay them; that is real dedupe machinery you have to build and expire.

Saga orchestration. One coordinator component owns the saga: it holds the workflow as explicit state, calls each service in turn, and on failure invokes the compensations in reverse. The sequence lives in one readable place. - Choose when: The flow has branches or timeouts, or more than a couple of steps, and you need to read the whole sequence in one file to reason about it; You want a single place to query where a given order has got to, and to drive recovery, dry-run and retries; The compensation order matters and must be deliberate rather than emergent. - Cost: The orchestrator is a coordinator, a single authority and a new shared dependency; every participant now couples to it, and it can grow into a god-service that knows too much; It is another thing to run, make durable and recover, and a partial failure of the orchestrator itself needs its own answer; Centralising the flow can pull business logic out of the services and into the coordinator, blurring who owns what.

Saga choreography. No central driver. Each service reacts to the events the previous step emits and emits its own, so the sequence is implicit in who subscribes to what. Compensations are triggered by failure events flowing back through the same fabric. - Choose when: The flow is short and linear, and unlikely to grow branches; You want services maximally decoupled, with no component that knows the whole sequence; The team is comfortable tracing behaviour across an event bus rather than reading a single workflow. - Cost: The sequence exists nowhere as a written artefact; to answer what happens next, a tired engineer must hold the whole event graph in their head, which is exactly the load the corpus tells you to minimise; Cyclic event dependencies and accidental fan-out are easy to create and hard to see, so the control flow becomes emergent behaviour no one specified; No single place shows progress or drives recovery, so debugging a stuck saga means reconstructing it from logs across services.

How to decide. Decide by who controls the input to each step and how wide a half-done outcome can spread. The first question is not saga vs 2PC at all: it is whether the rule is real enough and unrepeatable enough to deserve coordination, because most things people call invariants will tolerate eventual consistency, and the cheapest correct design names a reconciliation sweep and lets the copies converge. Spend the strong machinery only on money, safety and the genuinely unrepeatable. If you do need atomicity and the participants are a small set of transaction-aware resources under your control with low contention, a distributed transaction buys true isolation, and you pay for it knowingly: a coordinator that is a single point of failure, and an availability sacrifice the instant the network partitions. But the moment the operation crosses independent services, and especially anything an attacker or an unlucky caller can drive into a partial state, 2PC is usually both unavailable (the services cannot enlist) and too coupled (locks strung across the estate are the exact cascade you were trying to avoid), so the saga becomes the honest home for the invariant: local commits, compensations written as carefully-tested decision-from-effect code (tenet XI), and every step idempotent so retries are safe (tenet X). Then choose orchestration over choreography by blast radius on comprehension rather than elegance: if the flow has any branching or timeouts, or more than a step or two, put the sequence in one place an engineer can read in one breath, because a choreography whose order lives only in the subscription graph fails the governing test, that the most you can hold in your head to make a correct change should be the worst case, not the best. Keep choreography for the genuinely short, linear, decouple-at-all-costs case. The rule across every branch is the same: pay for coordination only where the invariant is real and violating it costs more than the coupling the cure drags in.

Reach for first. Before any of it, try to remove the need: can the operation live behind a single owner, so the rule is a local transaction rather than a distributed one? Failing that, ask whether the invariant truly needs coordination at all, or whether eventual consistency plus a named reconciliation job is honest enough. Most are. If you do need sequencing across services, an orchestrated saga with idempotent steps and tested compensations is the default; reach for a distributed transaction only when no compensation can honestly undo the step and the participants actually speak the protocol.

Pitfalls. - Calling something an invariant when it would happily tolerate eventual consistency, and paying for a coordinator or 2PC you never needed. - Writing the compensation as an afterthought: a refund is not an un-charge, a release is not an un-reserve, and some effects (sent email, launched job) do not compensate at all, so the saga can only ever reach a clean-ish state, never the prior one. - Forgetting that saga steps and compensations are replayed under at-least-once delivery, so a non-idempotent step double-charges and a non-idempotent compensation double-refunds. - Treating choreography as simpler because there is no coordinator, when the sequence has merely moved from one readable file into an invisible event graph no one can see end to end. - Leaving the intermediate inconsistency undeclared, so downstream readers assume an atomicity the saga never promised and build on a state that is only half-applied. - Adding an orchestrator and then letting it absorb all the business logic, turning a decoupling tool into a god-service every participant depends on.

See also. Tenets (XI), (X). the companion's third law. glossary: saga, coordinator, the CAP bargain, eventual consistency, distributed invariant. phases: integrating.

Strong vs eventual consistency (do you even need strong?)

You hit this when You have a rule that spans more than one party: two replicas, a cache and its source, an order service and a stock service. You are reaching for coordination to keep them in lockstep, and you have not yet asked whether the rule actually breaks if those parties disagree for a few seconds

The call. Does this invariant genuinely require strong (linearisable) consistency, or will it tolerate a bounded window of disagreement that converges?

Strong / linearisable consistency. Every reader sees the latest committed write, as though there were one copy and one clock. Enforced by a single owner of the fact, a coordinator that serialises the decision, or a quorum that makes a majority agree before anyone acts. - Choose when: The fact is money, safety, or genuinely unrepeatable: a balance that must not go negative, a seat sold at most once, an irreversible one-time effect; A stale read drives a wrong action that has no clean compensation, so repairing it after the fact is not on the table; The cost of a single violation plainly exceeds the cost of the coordination that prevents it; Write traffic on the contended fact is low enough that serialising it through one owner or quorum will not become the bottleneck. - Cost: By CAP you have chosen consistency over availability for the instant the network partitions: with no majority reachable the system must refuse to answer rather than risk two truths; Latency on the write path, since every decision crosses a quorum or queues behind one serialiser; A fresh single point of failure and a new shared substrate (the coordinator or consensus group), with its own operational weight and its own ways to fail; Hot keys serialise: throughput on the contended fact is capped by one owner no matter what you spend elsewhere.

Eventual consistency plus reconciliation. Let the parties disagree for a window and promise they converge once updates stop, backed by a named reconciliation sweep that finds the divergence and repairs it after the fact. - Choose when: A short window of disagreement is harmless: a follower count, a recommendation, a cache, a search index, an analytics rollup; Availability and write latency matter more than every reader seeing the very latest value; The repair is cheap and safe: writes are idempotent (tenet X) so replaying them converges, and a wrong intermediate read costs little or can be compensated; You can write down, concretely, where the reconciliation runs and how stale a read may get before it is a bug rather than a feature. - Cost: Readers can observe stale or briefly contradictory state, and code downstream must be written to tolerate it rather than assume one truth; Without a named reconciliation point the design is just unbounded inconsistency wearing an optimistic name (tenet XIII): the sweep is the load-bearing part, and it is the easiest thing to skip under deadline; Conflicting concurrent writes need a resolution rule (last-writer-wins, a merge, a CRDT), and getting that rule wrong loses data quietly; The convergence window is a second surface to observe and bound; without a probe on staleness you are guessing how far apart the copies have drifted.

Causal / bounded-staleness consistency. A middle ground that drops full linearisability but keeps a guarantee: reads never go backwards and causally related writes are seen in order (read-your-writes, monotonic reads), or staleness is capped to a stated bound. - Choose when: The pain is not raw staleness but an ordering anomaly: a user editing their own profile and seeing the old value, a reply surfacing before the message it answers; You want most of the availability and latency of eventual consistency while ruling out the disagreements that actually confuse people; The store or session layer already offers these guarantees (session consistency, bounded-staleness reads) so you are configuring rather than building; A stated staleness bound (at most N seconds, or N versions behind) is enough for the rule to hold. - Cost: It is weaker than strong consistency. It constrains the order and recency of what a reader sees but does not stop a true cross-party invariant being violated, so it will not keep a balance non-negative; It is easy to over-trust: 'causal' sounds like a safety net, and engineers quietly lean on it for invariants it was never meant to carry; Tracking causality (vector clocks, session tokens, dependency metadata) adds machinery and metadata that travels with every request; The guarantees are subtle and per-operation, so reasoning about exactly which anomalies are excluded becomes its own cognitive load on whoever reads the code.

How to decide. Decide by who controls the divergence and what a single violation actually costs, not by which model sounds safest. Coordination is often the most expensive thing in the building, so spend it only where the invariant is real and breaking it would cost more than the coordination drags in. Ask first whether the fact can have one owner: if it can, you have removed the question rather than answered it, and a tired engineer reasons about one writer instead of a quorum. If the rule genuinely spans parties no one owner can see, weigh the blast radius of a stale read against the blast radius of refusing to answer. When a wrong read spends money, oversells a seat, or fires an irreversible effect with no compensation, the violation is unbounded, so you pay for strong consistency on purpose and accept that you have chosen to be unavailable the moment the network splits. When a stale read costs only a slightly old follower count or a recommendation a beat behind, the disagreement is small and self-healing, and the honest answer is eventual consistency with a named reconciliation sweep; strong machinery here is coupling bought to guard an invariant that does not exist. Reserve causal or bounded staleness for the case where the harm is an ordering anomaly a person will notice rather than a true cross-party rule. The default is the cheapest correct one: keep strong consistency for money, safety and the genuinely unrepeatable, and let everything else converge.

Reach for first. Try to make the question disappear before you answer it. Most rules people reflexively call invariants are not. Give the fact a single owner (tenet XIV) so there is nothing to keep in lockstep, and the contention vanishes with the coordination. If the fact must be derived in more than one place, subscribe for the low-latency copy and run a reconciliation sweep for correctness, so disagreement is bounded and named rather than coordinated away. Only when a single owner cannot carry the load, or the rule genuinely spans parties that no one owner can see, do you reach for the heavier options below.

Pitfalls. - Calling something an invariant out of reflex and paying for a quorum to protect a counter no one would miss if it lagged a second. - Shipping 'eventual consistency' with no reconciliation sweep and no stated staleness bound, which is unbounded inconsistency given a reassuring name. - Adding a coordinator to enforce one rule and quietly creating a new single point of failure and shared substrate that can take down everything, not just the contended fact. - Trusting causal or session consistency to uphold a true cross-party invariant such as a non-negative balance, which it was never designed to guarantee. - Resolving concurrent writes with an unexamined last-writer-wins and silently dropping the loser's update. - Choosing strong consistency on a hot key and discovering that the single serialiser, not the database, is now your throughput ceiling.

See also. Tenets (X), (XIII), (XIV), (XX). the companion's third law. glossary: the CAP bargain, eventual consistency, distributed invariant, coordinator, reconciler, one source of truth. phases: integrating, planning.

Optimistic (version/CAS) vs pessimistic (locks) concurrency

You hit this when Two writers can touch the same record in the gap between one reading it and writing it back, and the second write silently clobbers the first. You have a check-then-act on shared state, and under real concurrency the value you read is already stale by the time you act on it

The call. When concurrent writers may collide on the same record, do you detect the collision after the fact (read a version, write conditional on it being unchanged, retry on conflict) or prevent it up front (take a lock for the duration so others wait)?

Optimistic (version / compare-and-swap). Read the row with a version (a counter, a timestamp, or the old value itself). Compute the new value. Write it only on the condition that the version is unchanged, typically UPDATE ... SET v = v+1 WHERE id = ? AND v = ?, then check rows-affected. Zero rows means someone got there first, so you re-read and retry. - Choose when: Conflicts are rare: most writes touch disjoint rows, so the common path pays nothing and only the loser of a genuine race retries; The read-compute-write window is short and the work is safe to redo, so a retry costs little; Writers are spread across requests, processes or machines, where holding a row lock open for the whole think-time would park a connection or cross a boundary you cannot trust; You want the racy check-then-act collapsed into one indivisible conditional write rather than leaning on a held lock you might silently lose. - Cost: Under contention the retry rate climbs and throughput falls: every loser redoes its work, and a hot row can livelock with most attempts failing; The caller must own the retry loop, with a bound and a backoff. Omit it and one conflict surfaces as a hard error, or an unbounded loop hammers the row; A version guards one row. An update that must stay consistent across several rows needs a version on each, or a single transaction wrapping them, and the simplicity is gone; It rejects late, after the work is done. For an expensive computation, or one whose side effects are already in flight, discovering the conflict at write time wastes the whole attempt.

Pessimistic (locks). Before reading, take a lock on the row or key (SELECT ... FOR UPDATE, an advisory lock, a lease) and hold it across read, compute and write. Other writers block until you commit or release. The window in which anyone could interleave never opens. - Choose when: Conflicts are common: on a hot row, queuing writers once is cheaper than a crowd of optimistic retries thrashing; The protected section is long, expensive, or has side effects you must not run twice, so paying to wait beats paying to redo; The update spans several rows or statements and you want one consistent lock order held for the whole sequence rather than versioning each piece; A single writer at a time is itself the invariant the domain requires, not merely a means to avoid lost updates. - Cost: Throughput is capped by the lock: writers serialise on it even when most would not actually have collided, so you pay for contention you do not have; Locks compose into deadlock. Two sequences taking the same rows in different orders will wedge, and you now owe a consistent lock order and a timeout for every path; A held lock is a liability across a boundary: a crash, a GC pause or a network partition can leave the holder believing it still owns the lock after the lease has moved on, so a distributed lock needs a fence token the resource validates, not trust in the acquisition; Held connections and parked transactions are finite. Long critical sections drain the pool and turn one slow holder into a queue that stalls unrelated work.

How to decide. Decide by who contends and how much a wrong outcome costs, not by taste. The governing question is what a tired engineer must hold in their head to make a correct change here, with a bounded blast radius for the case where writers actually collide. First settle whether the window is even real: if writes for one entity already funnel through a single owner, or fit in one atomic statement, neither scheme is needed and adding one is pure coupling. Where the window is real, let contention pick the mechanism. Rare collisions favour optimistic, since the common path stays lock-free and only the loser retries, and that cost is honest provided the caller actually carries a bounded retry loop. Frequent collisions on a hot row, or a critical section that is long, expensive or has side effects you dare not repeat, favour pessimistic: you pay to wait once instead of paying to redo many times. Then weigh the blast radius of being wrong. Optimistic fails loud and local, a rejected write you re-drive. Pessimistic fails in ways that spread: deadlock and pool exhaustion that reach code which never touched the contended row, and across a process boundary a believed-but-lost lock that needs a fence token or it corrupts silently. Spend the heavier machinery only where the collision is genuine and a lost update would cost more than the contention and deadlock risk the lock drags in. If you cannot say roughly how often two writers really meet on this record, you are not yet ready to choose. Measure or reason out the contention first, because the whole decision turns on it.

Reach for first. Remove the collision before you arbitrate it. Often there is no real shared write at all: route writes for one entity to a single owner (one partition, one queue consumer keyed by id) so they serialise naturally, or fold the update into one atomic statement the database already makes indivisible, such as INSERT ... ON CONFLICT or UPDATE ... WHERE balance >= amount with a rows-affected check. If you genuinely never have two interleaving writers, neither version nor lock buys you anything, and the machinery is just cost with nothing to show for it. Reach for explicit concurrency control only once you have confirmed the window is real.

Pitfalls. - Read-modify-write in application code with no version and no lock: the plain SELECT then UPDATE that looks atomic on one line but is two operations, so the second writer's work overwrites the first and nobody notices. - Optimistic without a real retry loop: catching the conflict and either throwing it straight at the user, or retrying forever with no bound and no backoff, turning a rare race into either errors or a livelock. - Trusting a distributed lock as if held means held: a lease can expire under a GC pause or a partition while you still believe you own it, so without a fence token the resource validates, your stale write lands anyway. - Inconsistent lock order across paths, which compiles and passes review and then deadlocks only under the exact interleaving production eventually produces. - Holding a pessimistic lock across a network call or user think-time, parking a database connection for seconds and draining the pool so unrelated requests stall. - Versioning one row of a multi-row invariant: each row's compare-and-swap succeeds independently while the set they form is left inconsistent.

See also. Tenets (IX), (X). the companion's third law. glossary: the check is a guess, compare-and-swap, two operations pretending to be one, fenced lease. phases: implementing.

Fenced lease vs naive distributed lock

You hit this when Two or more machines must take turns on one resource, and the way you have arranged it, a worker can acquire the lock, stall, and carry on writing as though nothing happened. The stall is real: a long GC pause, a scheduler eviction, a network partition that hides the renewal. While it sleeps the lease times out and is granted to a second worker. Then the first wakes and writes, and now two holders are live at once. The damage shows up downstream as a double-charge, a corrupted file, or a job processed twice, far from the lock code that allowed it

The call. When you need mutual exclusion across machines and the holder can pause and wake believing it still owns the lock, do you hand out a fenced lease, run a naive TTL lock without fencing, or remove the shared lock entirely?

Fenced lease. The lock service hands out a monotonically increasing token (the fence) alongside a time-bounded lease. Every write to the protected resource carries its token, and the resource rejects any write whose token is lower than the highest it has already accepted. A stale holder that wakes up late carries an old number and is refused at the point of the write. - Choose when: A paused or partitioned holder writing after its lease expired would corrupt data or cause a real-money or safety-critical error; The protected resource can be taught to check a token and refuse stale writes, or already supports conditional writes (a version column, an If-Match precondition, a compare-and-swap); Correctness must survive the holder being wrong about whether it still holds the lock, which across a distributed boundary it eventually will be; You are ordering or expiring against time and want the guard to hold even when a clock steps backwards. - Cost: Two parties must cooperate: the lock service issues and increments the token, and the resource enforces it; a lease without enforcement at the resource buys nothing; Every protected write path has to thread the token through and validate it, which couples the resource to the locking scheme; The resource needs durable, monotonic state for the highest token seen, so a resource that cannot store or compare it (a fire-and-forget side effect, a third-party API with no conditional write) cannot be fenced at all; More moving parts to reason about and test than a single SETNX call.

Naive TTL lock with no fencing. A single key in a shared store (Redis SETNX with an expiry, a row with a lease-until timestamp) grants the lock; the holder renews before the TTL lapses and deletes the key when done. There is no token, so the resource trusts whoever claims to hold the lock at write time. - Choose when: The work under the lock is advisory or best-effort, where an occasional double-run is harmless and self-correcting; The protected operation is already idempotent (X), so a second holder repeating it does no damage; You genuinely cannot fence the resource and the lock is there to reduce duplicate work, not to guarantee single ownership; Contention is low and the blast radius of a rare overlap is trivial and observable. - Cost: It is a TOCTOU waiting to happen: the gap between checking that you hold the lock and acting on it can be arbitrarily long, and the check becomes a guess the moment a pause or partition slips in; The TTL is typically compared against the wall clock, so an NTP step or a lying peer can expire or extend it out from under you; It reads as correct in review and fails only under the exact production conditions (GC, eviction, partition) that are hardest to reproduce; Renewal is itself a deadline across an uncontrolled boundary; a missed renewal you never notice is silent split-brain.

Avoid the shared lock: single owner or partitioned work. Remove the contention instead of arbitrating it. Route every operation on a given key to one owner (a partition, a consistent-hash shard, a single-writer queue consumer), or claim units of work by an atomic primitive (an atomic rename, INSERT ... ON CONFLICT, UPDATE ... WHERE checking rows-affected) so the store itself grants single ownership with no separate lock. - Choose when: The work partitions cleanly by a key, so each partition has exactly one owner and cross-machine exclusion is never needed; You can express the claim as one atomic step at the store you already trust, collapsing check-and-act into a single operation; You would rather delete the distributed-lock problem than carry the machinery to make it safe; Throughput per key is within what one owner can handle. - Cost: Partitioning is a design commitment: rebalancing on failover or scale-out reintroduces a brief two-owner window you must handle (typically with fencing at the seam); A single owner per key is a throughput ceiling and a failure domain; if that owner stalls, its keys stall with it; Some problems do not partition: a genuinely shared external resource with one global invariant still needs cross-machine exclusion; Atomic-claim primitives exist at the database or filesystem but not for every side effect you might need to guard.

How to decide. Decide by who controls the timing of the write and how wide the damage spreads when two holders are briefly live at once. The dangerous input here is not an attacker's payload but your own holder's stale belief: across a distributed boundary you will eventually be wrong about whether you still hold the lock, so the question is what one extra write costs at the resource. If an overlapping write corrupts data, double-charges, or breaks a safety invariant, the blast radius is unbounded and you must pay for fencing, because the property (at most one effective writer) is real and a naive lock cannot enforce it: a TTL guards intent, whereas a fence token guards the write itself. Spend the coupling, threading the token through every protected path, only where that is true. If the operation is idempotent or the overlap is self-correcting, the property is not real, fencing is machinery guarding nothing, and a naive lock (or no lock) is the honest, smaller thing to hold in your head. Before either, ask whether the lock should exist at all: a single owner per key or an atomic claim at the store you already trust removes the window rather than policing it, and leaves a tired engineer with one fewer distributed invariant to keep true.

Reach for first. Try to delete the need for the lock: partition the work by key so each key has one owner, or claim each unit with a single atomic step at the store you already have (an atomic rename, INSERT ... ON CONFLICT, UPDATE ... WHERE with a rows-affected check). If exclusion genuinely cannot be designed away, fence the lease; reach for a naive TTL lock only when the guarded operation is already idempotent.

Pitfalls. - Treating a lease as if it were a lock: the TTL bounds how long you may hold it, not how long you actually do, and a paused holder keeps writing past expiry. - Generating a fence token but never enforcing it at the resource, which leaves you with all the machinery and none of the protection. - Comparing the lease expiry against the wall clock, so an NTP correction or a clock-skewed peer revokes or extends it silently; measure the lease against a monotonic base. - Renewing the lease on the same thread that does the long blocking work, so the very pause that should lose you the lock also stops you noticing you lost it. - Assuming the lock service's own availability is free: when it partitions, every holder must decide whether to keep writing or stop, and 'keep writing' is split-brain. - Picking a single global owner for throughput you cannot actually serve through one process, then bolting a lock back on to shard it.

See also. Tenets (IX), (XXIII). the companion's first law, the companion's third law. glossary: fenced lease, TOCTOU. phases: implementing, integrating.

Ack/commit after the durable write vs before

You hit this when You consume a message or a stream record, do something with it, and must tell the source you are finished so it stops offering you that record. The only real choice is when you send that signal: after the work is durably written down, or before. The source treats your ack as proof the record is handled, and it will not send it again

The call. When do you acknowledge the message or commit the offset: after the record is durably persisted, or on receipt before the write?

Ack/commit only after the durable write. Persist the record first (to your store, or to a WAL you can replay), confirm the write landed, then send the ack or advance the committed offset. The source keeps offering the record until it hears back, so a crash anywhere before the ack means redelivery, not loss. - Choose when: The record carries anything you would miss: a payment, an order, a customer event, a row in a ledger; The source supports at-least-once redelivery, which is almost all of them; you are choosing whether to use the safety it already offers; You can make the write idempotent, so the redelivery this ordering invites lands once and not twice; A crash between receipt and ack is plausible, which on a long-running consumer means always. - Cost: Redelivery is now normal traffic rather than an edge case, so the consumer must dedupe by message id or upsert by a stable key; that dedupe table or idempotency key is real storage with real expiry to manage; Throughput drops a little: you hold the record uncommitted across the write, the source's in-flight window fills sooner, and a slow store backs pressure up to the broker; A consumer that crashes mid-batch reprocesses the whole uncommitted batch on restart, which is wasted work even when it is correct.

Ack/commit on receipt, before persisting. Tell the source you have the record as soon as it arrives or is enqueued in memory, then write it down afterwards. The source marks it consumed and moves on immediately, before you have any durable proof you kept it. - Choose when: The record is genuinely disposable: a metric sample, a cache-warm hint, a log line you would shrug at losing; You have measured that the durable write is the throughput ceiling and that early ack buys back something you actually need; Loss on restart is acceptable and you have written down, somewhere a reader will see, that this stream is lossy by design. - Cost: A crash in the window between ack and write loses the record outright, and the source never resends it because it believes the record was handled; this is ack-before-persist eating data on restart, and it surfaces with no error, because every component reckons it did its part; The loss stays invisible until someone reconciles totals weeks later and finds a gap nobody can date or explain; You have quietly converted the source's at-least-once delivery into at-most-once, usually without saying so, so the next reader assumes the stronger guarantee that is no longer there; The faster path only helps if persistence was truly the bottleneck; if it was not, you took the loss risk and bought nothing.

How to decide. The input here is supplied by the source, and the blast radius of getting it wrong is every record in the crash window, gone with no trace. That settles it for anything you would miss: persist, then ack. The signal you send the source is a promise about durability, and a promise made before the write is one you cannot keep across a crash. Early ack is not a general optimisation. It is a deliberate trade of durability for a little latency, and it pays only when persistence was genuinely the ceiling and the records were genuinely disposable. The honest test is what a reconciliation would find: if a gap in this stream would be a defect someone has to chase, ack after the write. The cost of doing it right is that redelivery becomes ordinary, so you owe the consumer idempotency; that coupling is cheap and local, and it is what turns at-least-once into the effect you wanted, applied once where it counts. The coupling you avoid by acking early comes back as silent loss, which is the most expensive kind.

Reach for first. Persist then ack, and make the write idempotent so the redelivery this invites lands once. That is the correct default for every stream whose loss you would notice, and it is no more machinery than a dedupe table or an upsert keyed by message id.

Pitfalls. - Auto-commit left on: many client libraries commit offsets on a timer in the background, which silently acks records you have not finished persisting. Turn it off and commit explicitly after the write. - Acking the whole batch after persisting only the first record. The ack must cover exactly what is durably down, no more. - Treating an in-memory enqueue or a buffered writer as durable. The record is not persisted until the write is flushed and confirmed, not when it enters a buffer a crash discards. - Persisting then acking but with a non-idempotent write, so redelivery double-applies. The safe ordering and idempotency are a pair; the ordering without idempotency just trades silent loss for silent duplication. - Shrinking the loss window instead of closing it. Acking 'almost immediately' before the write is still ack-before-persist; the window is smaller, not gone. - Choosing early ack for latency without measuring that persistence was ever the bottleneck, so you take the data-loss risk and gain nothing.

See also. Tenets (XII), (X). glossary: ack-before-persist silently eats data on restart, a WAL, exactly-once where it counts. phases: implementing, operating.

Time & ordering

Measure durations and expiry: monotonic clock vs wall clock

You hit this when You are coding a timeout, a retry backoff, an elapsed-time measurement, a request deadline, or a TTL on a token, lease or cache entry. The obvious tool is whatever now() your language hands you, and on most platforms that default is the wall clock

The call. When you need an interval, a deadline or an expiry, do you measure it against a monotonic clock or a wall clock?

Monotonic clock for every interval and deadline. Read a clock that only ever moves forward (CLOCK_MONOTONIC, System.nanoTime, performance.now) at the start, read it again later, and act on the difference. A deadline or TTL becomes a duration added to a monotonic base rather than a target wall-clock instant. - Choose when: You are measuring elapsed time: a timeout, a backoff, a latency figure, a rate-limit window; You are firing after a delay relative to now: a retry, a watchdog, a control-loop deadline; An expiry must hold even if NTP steps the clock, a leap second lands, or a client lies about the time; The interval lives within one process or one machine, where a single monotonic source is available. - Cost: The reading is meaningless across reboots, and usually across machines, so you cannot persist it or compare it against another host's reading; It carries no calendar meaning, so you cannot log it as a human time or relate it to an external schedule unless you also capture a wall-clock reading; Some monotonic sources stop counting while the machine is suspended, so on those platforms a long sleep can make a timeout fire late or never; Two clocks now exist in the code, and an engineer must know which one each value came from before comparing them.

Wall clock, only to show or hit a calendar moment. Use the wall clock for what it is actually for: displaying a time to a human, stamping a record's created_at, or firing at a named calendar instant such as midnight UTC or a billing boundary. - Choose when: The output is a human-facing timestamp, a log line, or an audit field someone will read as a date; The trigger is a genuine calendar event ("run at 02:00", "expire at the end of the quarter"), not "in N seconds"; You need to coordinate with an external system that itself speaks in wall-clock instants; You accept that the value can jump, and the jump is harmless or even wanted (a displayed time should track corrections). - Cost: The value can step backwards or forwards under NTP, a leap second, a manual change, or a daylight-saving transition, so any code that treats it as monotonic is wrong; Across machines it skews, so comparing two hosts' timestamps to decide order is a race in disguise; see clock skew is a race wearing a timestamp; Calendar arithmetic drags in time zones and DST, a large source of bugs in their own right; If you reuse it for an interval out of convenience, you have silently chosen the broken option below.

Wall clock for an interval (the bug). Compute elapsed time, a timeout, or an expiry by subtracting two wall-clock readings, or by storing an absolute wall-clock deadline and comparing now() against it. The tempting default, and the one tenet XXIII exists to stop. - Choose when: Never as a deliberate choice. It is listed only because it is what you get by reaching for the default now() without thinking about which clock it is. - Cost: An NTP correction or leap second can step the clock backwards mid-interval, yielding a negative duration, a timeout that fires instantly, or one that never fires; A TTL or token expiry checked this way can be moved by a clock jump, or by anyone who controls the clock, so the security property evaporates exactly when it matters; On a machine whose RTC resyncs mid-loop, the timing maths corrupts silently and the code cannot tell; The failure is intermittent and environment-dependent, so it survives testing and surfaces in production months later.

How to decide. Pick the clock by the question you are asking, not by which now() is shortest to type. If the answer is a duration (how long has this taken, how long until this fires, how long is this valid for), the only correct source is a monotonic clock, because the wall clock can run backwards and a duration must not. If the answer is a calendar instant a human will read or an external system expects, use the wall clock and accept that it moves. For expiry in particular the governing tie-break is who controls the input. A token, lease or TTL is checked against a clock; if that clock is the wall clock then anyone who can step it (an NTP misconfiguration, a lying peer, an attacker on a machine they control) can move your expiry. That is an unbounded blast radius on an input you do not own, and the cure is nearly free: measure the lifetime as a duration on a monotonic base. Spend the small cost of carrying two readings, a monotonic one for the interval and a wall-clock one only where a human or an external schedule needs the date, wherever an interval or expiry is load-bearing. The one thing a tired engineer has to hold in their head is the question from the tenet: am I measuring a duration or naming a moment? Get that right at the call site and the rest follows.

Reach for first. Reach for the monotonic clock for anything that is a duration, deadline, timeout, backoff or TTL, and reserve the wall clock for displaying a time or firing at a named calendar instant. This is the default the language should have given you; on most platforms you have to ask for it by name (CLOCK_MONOTONIC, nanoTime, performance.now, a steady_clock). Make that the habit and the buggy third option never gets written.

Pitfalls. - Storing a monotonic reading in a database or sending it over the wire: it is only meaningful within the process that read it, so cross-process or cross-reboot comparisons are nonsense. - Ordering events across hosts by comparing created_at timestamps: that is the check-then-act race of tenet IX wearing a timestamp; order with a logical clock or a fenced sequence instead. - Assuming the monotonic clock advances during sleep or suspend: on some platforms it pauses, so a timeout set before a laptop sleeps can behave oddly on wake. Know your platform's CLOCK_MONOTONIC vs CLOCK_BOOTTIME distinction. - Setting a token or lease expiry as an absolute wall-clock instant, then trusting it on a machine whose clock a client or attacker can move. - Mixing the two clocks in one calculation (subtracting a wall-clock reading from a monotonic one), which produces a garbage interval that may look plausible in testing. - Treating a negative measured duration as impossible and dividing or indexing by it, turning a clock step into a crash or a wild value.

Order events across machines: logical clock/fenced sequence vs timestamps

You hit this when Two events happened on two different machines (a write on host A, a write on host B; a "created" and an "updated"), and downstream code needs to know which came first. The obvious move is to compare their created_at fields, and in a demo it always works

The call. When two events occur on different machines, how do you establish which one happened first?

Compare wall-clock timestamps across hosts. Read created_at (or now()) on each host and order by whichever value is smaller. The order is inferred from whatever clocks the two machines happened to be carrying. - Choose when: The two events never have a causal relationship and the order genuinely does not matter; you only need a rough display ordering, and a wrong answer is cosmetic; The machines share a single trusted time source with bounded, measured uncertainty, and you actually wait out that uncertainty window before committing to an order (the TrueTime discipline, not merely calling now()); You are ordering events on one host against one monotonic source, which is not really the cross-machine case at all. - Cost: Clocks on separate machines drift relative to one another, so the order you read off created_at is a guess rather than a fact: the same race as a check-then-act on shared state, dressed up as a time comparison; The window stays invisible in review and on a developer laptop with tight NTP, then reorders events silently in production where the skew is wider; An NTP step, a leap second, or a host whose clock simply runs fast can move an event backwards in time relative to its own cause, and a lying or compromised client can set its timestamp to whatever it likes; The failure is silent corruption. There is no exception and no log line, only two records that disagree with reality, which is about the worst thing to be debugging after the fact.

Logical ordering: happened-before, version vectors, a fenced sequence. Order by causality rather than by clocks. A happened-before relation, a Lamport or vector clock, or a monotonic fenced sequence number records that A could have influenced B, independent of any wall clock. - Choose when: You need to assert true ordering across machines, and getting it wrong corrupts data or breaks an invariant downstream; Events have real causal links (a reply after a message, an update after the create it amends) that the order must respect; A fenced token is already in play for the same resource, so the sequence number you need is sitting right there; Multiple writers can each advance independently and you must merge their histories without losing causality (version vectors). - Cost: You must thread a clock or sequence through the events and persist it; the metadata then travels with every record and every message forever; Vector clocks grow with the number of writers and need pruning, which is its own state to own and expire; Logical clocks give you a partial order rather than a total one. Genuinely concurrent events stay incomparable, so you still need a deliberate tie-break rule for them; It is more to carry in your head than created_at, and the cost is only justified where the ordering property is actually load-bearing.

A single owner that assigns the order. Route the events that must be ordered through one component (a partition leader, a sequencer, a single database sequence) that stamps each with a monotonically increasing number as it arrives. The order is read off that number. - Choose when: The events you must order can be funnelled through one place without crippling throughput (per-key or per-partition, not the whole system); You want a total order with a single, obvious source of truth for it, rather than reasoning about partial orders; You already have a natural owner: a database sequence, a per-partition log, a leader that the writes go through anyway. - Cost: The owner is a serialisation point and a bottleneck; everything that must be ordered together waits on it, capping concurrency for that key; It becomes a failure and availability concern. If the sequencer is down, ordered writes stall, and failover must fence the old owner or two sequencers will hand out clashing numbers; Cross-partition ordering is back to square one; the single owner only totally orders what flows through it; You have traded a distributed-ordering problem for a leader-election and fencing problem, which is real work in its own right.

How to decide. Decide by who controls the clock and how much a wrong order costs. If either machine's timestamp is set by a client, an untrusted peer, or any host whose NTP discipline you do not own, comparing wall-clock times hands an attacker or an unlucky caller a lever to reorder your events at will, with a blast radius as wide as whatever consumes that order; the cure is logical ordering, and the coupling it drags in (a clock or sequence threaded through every event) is cheap set against silent corruption. If the order is genuinely cosmetic and a swap costs nothing, spend no machinery at all: read created_at and move on. The interesting middle is real ordering between writers you do control. Reach for the cheapest mechanism that makes the order a fact rather than a guess, which is usually a sequence number from an owner you already have, or a fenced token already in play, before reaching for vector clocks. Spend the coordination of a single sequencer only where you need a total order and the throughput cap is tolerable, and spend the bookkeeping of version vectors only where independent writers must merge without losing causality. The governing question stays the same throughout: a tired engineer reading created_at < created_at must be able to trust it without holding the two machines' clock histories in their head, and nothing an attacker controls should be able to make that comparison lie.

Reach for first. Ask whether the order has to be established across machines at all. Very often the events you are comparing already share one owner (the same partition, the same row, the same fenced resource), and a single monotonic sequence number from that owner settles the order with no distributed machinery. If they do not share an owner, check whether the order is actually load-bearing before building anything: where a swap is harmless, order by created_at and document that it is approximate. Only when the order is both cross-machine and load-bearing do you reach for logical clocks or a sequencer.

Pitfalls. - Ordering by created_at across hosts and calling it correct because it passed on a laptop with tight NTP; the skew that reorders events only shows up at production scale. - Trusting a timestamp that a client or untrusted peer set, so the order ends up being whatever the caller decided to claim. - Assuming a logical clock gives a total order. Concurrent events are incomparable, so code that sorts on them needs an explicit, stable tie-break or it will reorder nondeterministically. - Reaching for vector clocks where a single sequence number would do, then carrying unbounded per-writer metadata you never prune. - Adding a single sequencer for ordering and forgetting to fence it on failover, so a stale leader and a new one both hand out sequence numbers. - Using the monotonic clock you measure durations with to also order events across machines, a job it cannot do.

Failure containment & load

Deadline on a cross-boundary wait vs unbounded wait

You hit this when A piece of your code is about to block on something you do not own: a network call, a lock held by who-knows, a queue read, a connection acquired from a pool. The happy path returns in milliseconds, so the wait is invisible right up until the day the other side stops answering and your thread sits there forever

The call. When your code waits on a boundary you do not control, do you bound the wait with a deadline that frees the resource and propagates, or let it wait until the dependency returns?

Deadline that frees the resource and propagates. Bound the cross-boundary wait with an absolute cutoff, ideally a single budget pushed through the whole call chain rather than a fresh fixed timeout per hop, and on expiry cancel the underlying work and release the thread, connection or lock. - Choose when: The wait crosses a boundary you do not control: a network, an IPC, a lock you do not own; The work behind the boundary is cancellable, or idempotent so a retry does no harm; The resource held during the wait (a pool slot, a thread) is scarce and shared, so a hang would starve other callers; You can carry one budget down the chain, so a 5 s edge cutoff is not spent waiting on a hop that itself waits 30 s. - Cost: A budget is only honoured if it propagates: per-hop fixed timeouts compose into a total far longer than any single number suggests; A timeout that walks away without cancelling orphans the slow work, and a retry on top can fire a duplicate, so this is unsafe on non-idempotent, non-cancellable operations; The number is a guess about the tail: too tight and you kill healthy-but-slow calls under load, too loose and the protection arrives too late; Threading a deadline through every layer is real coupling and real code, and the retry policy that usually follows it is a further decision, not a freebie.

Unbounded wait. Block on the dependency with no cutoff and trust it to return. The default you get for free from a synchronous call with no timeout set. - Choose when: Genuinely never for a boundary you do not control: this is the failure mode the deadline exists to prevent; The only defensible case is a boundary you fully own with a hard, proven latency bound, where adding a cutoff is noise; You are prototyping throwaway code where a hang is an acceptable, visible failure and nothing else depends on the thread. - Cost: A hung dependency does not stall in one place; it eats this thread, then the pool, then the caller's pool, until one slow service takes the fleet down; The failure stays invisible until the worst day, because the happy path returns fast and hides the missing cutoff in review; Nothing is observable: the system looks dead instead of reporting a bounded error you can alert on and shed against; Recovery means a restart or a manual kill rather than a clean error, and the reader at the call site has no way to know the worst-case wait.

Long-job exception: progress, heartbeat and cancellation. For work that is legitimately long and progressing, drop the wall-clock deadline and instead require a heartbeat or progress signal, with cooperative cancellation, so you can tell a stalled job from a slow one and stop it on demand. - Choose when: The work is correct long work: a multi-hour query, a large batch transcode, a training run, where any fixed cutoff would kill healthy work; You can emit progress or a liveness heartbeat, so a stall is distinguishable from steady advance; The job is cancellable cooperatively, so an operator or a supervisor can stop it without a kill -9. - Cost: More machinery than a deadline: you must instrument progress, run a watchdog on the heartbeat, and wire a cancellation path; A missing or coarse heartbeat lets a truly stuck job masquerade as a slow one, so the stall protection only works if the signal is honest and frequent; Misapplied to a short cross-boundary call it is over-engineering, and a heartbeat across an uncontrolled network can itself hang, so the boundary question does not disappear; Cancellation that is not actually cooperative leaves the job running after you have given up on it, the same orphan problem in a new costume.

How to decide. Decide on who controls when the wait ends and how far the damage travels if it never does. If the other side of the boundary is a party you do not control (a network, an IPC peer, a lock you did not take), you do not own the clock, and a wait with no cutoff is a bet that the stranger is always fast and always answers. The blast radius is what makes that bet expensive: a hung call holds the thread it is on, then the pool, then the caller's pool, so one slow dependency stops being one slow request and spreads outward until the fleet melts. That is genuine liability over something someone else controls, so it earns the machinery, because a deadline turns the unbounded hang into a bounded, observable, local error. Spend the deadline only where the boundary is genuinely uncontrolled. An in-process call whose latency you do own does not need one, and a wall-clock cutoff on correct long work (a six-hour training run, a 50 GB sort) just murders it at hour five, which is the case for the heartbeat option instead. And a deadline is only safe once the work it bounds can be cancelled or is idempotent on retry, otherwise the cure orphans the slow work and a retry doubles the load. The tie-break that settles it: a tired engineer reading this call site should be able to see the worst-case wait without reasoning about a stranger's uptime, and the cost of a missing deadline (a fleet-wide hang) dwarfs the small coupling of carrying a budget through the call chain.

Reach for first. Before reaching for a hand-rolled timer, check whether the client or driver in front of you already has a deadline knob: an HTTP request timeout, a database statement_timeout plus a connection-acquisition timeout, a lock try-acquire with an expiry. Set the number there, write it down next to the call, and you have the cheapest correct answer. Hand-built deadline plumbing is only worth it where no such knob exists.

Pitfalls. - A timeout that does not cancel and propagate: it fires, the caller walks away, the slow work keeps running orphaned, and a retry quietly doubles the load. - Per-hop fixed timeouts that do not compose: 5 s set at three layers can wait 15 s in total, so set a shared deadline that shrinks as the budget is spent. - Putting a deadline on a non-idempotent, non-cancellable operation, so firing it on a write that already partly committed leaves duplicate or half-done state. - Killing correct long work with an arbitrary wall-clock cutoff because it crossed a boundary, when the right tool was progress plus heartbeat plus cancellation. - Forgetting the resource held during the wait: the deadline must release the connection or pool slot, not just abandon the call, or you have bounded the wait but not the contention.

Retry: capped backoff + jitter vs naive retry vs no retry

You hit this when A call to a dependency has just failed or timed out, and the obvious move is to try it again. The failure may be a genuine blip (a dropped packet, a brief overload) or the leading edge of an overload you are about to make worse. You have to decide whether to retry at all, and if so, how hard

The call. When a call fails or times out, do you retry with capped exponential backoff and jitter, decline to retry and surface the failure, or retry immediately in a tight loop?

Capped exponential backoff with jitter (idempotent only). On failure, wait before retrying; roughly double the wait each attempt up to a ceiling, add randomness to each delay, and stop after a fixed budget of attempts. Applied only to an operation that is safe to repeat. - Choose when: The operation is idempotent (X), so a repeat can never double-charge or corrupt; The failure is plausibly transient and a short, bounded wait gives the dependency room to recover; Many clients can fail at the same instant, so their retries must be scattered in time rather than fired in lockstep; The original call was deadlined (VI), so the retry budget is bounded by a clock the caller actually owns. - Cost: Real machinery to build and, harder, to tune: a budget, a backoff schedule, a jitter source, often a breaker, set near the edge of the bistable region rather than the comfortable middle; Added latency on the slow path, since every backoff interval is wait the caller absorbs before it learns the call has truly failed; Correctness rests entirely on the idempotency precondition; the moment that is false, every retry is fresh corruption and the discipline saves nothing; An untested retry-plus-fallback path is a second bug waiting for the worst moment (XIX): the storm is exactly when it first runs for real.

No retry: surface the failure. Do not try again. Return or propagate the error to the caller, or to an outer layer that owns retry policy, and let it decide. - Choose when: The operation is not safe to repeat and you have not yet made it idempotent (X): a repeat would charge twice, send twice, or corrupt state; An outer layer already retries, so adding your own stacks loops and multiplies attempts you cannot see; The failure is plainly not transient (a 4xx, a validation error, a permission denial), so trying again just fails again, slower; You would rather fail fast and shed load than add yourself to a queue piling onto a struggling dependency. - Cost: You give up recovery from genuine blips that a single jittered retry would have absorbed, pushing transient noise up to the user or caller; The judgement moves outward: every caller must now decide whether and how to retry, and some will do it worse, or not at all; Surfaced failures still need a fallback or a degraded tier (XIX), or you have only moved the white screen one level up; If nobody upstream retries, an operation that would have succeeded on a second attempt is recorded as a hard failure.

Naive tight retry loop. On failure, immediately try again, typically in a fixed loop with no wait, no ceiling worth the name, and no randomness. - Choose when: Honestly, almost never against a shared dependency; it is listed because it is the default people reach for and the one to recognise and remove; Defensible only in-process against a local, contention-style failure (a compare-and-swap that lost a race) where the retry is cheap and the contended resource is not a remote service; Acceptable as a stopgap only where load is provably tiny and single-caller, and even then a small cap costs nothing to add. - Cost: This is the retry storm generator: synchronised clients hammer a recovering service in lockstep and knock it back down, the textbook metastable failure Law II describes; With no backoff, the retries become the load that sustains the outage, so removing the original fault changes nothing; With no jitter, every client's clock stays aligned, turning the moment of recovery into one coordinated spike; An unbounded retry is the queue that melts the fleet under a helpful name; it is the same unbounded-resource liability as any other (VII), only disguised as resilience.

How to decide. Decide by who controls how often this fires and how wide the damage spreads when it does. A retry is a queue with no admission control under a friendlier name, so it falls under the rule for anything a caller or an unlucky upstream can drive in a loop (VII): if the trigger is external (a flaky dependency, synchronised clients, a redelivered message) it gets a ceiling, a backoff, and jitter, or it does not ship. The first gate is the blast radius of a wrong repeat. If the operation is not idempotent (X), a retry does not merely add load, it corrupts, so the correct strategy is no retry until you have made the write safe to repeat with a stable key or an upsert. Past that gate the question turns from correctness to load: capped backoff with jitter is the right spend precisely when many callers can fail together and a naive loop would synchronise them into a thundering herd that re-buries a recovering service, the metastable loop Law II warns will not let go on its own. Spend the machinery (a budget, a breaker, a jitter source) only where that loop is real; for a single caller hitting a dependency that either works or does not, a tight loop buys nothing, and a couple of jittered attempts or a clean surfaced failure is the smaller thing for the next engineer to hold in their head. The tie-break is the deadline: bound the retry to the clock the caller already owns (VI), because an attempt whose result no one is still waiting for is pure load with no upside.

Reach for first. First ask whether the retry can be deleted rather than tuned. The cheapest correct answer is usually to not generate the failure in the first place: remove the fragile hop, make the call local, or let a layer that already retries (the message broker, the load balancer, the client SDK) own it, so you do not stack a second retry loop on top of a first one you cannot see. If a retry genuinely belongs to you, the cheapest correct shape is a small fixed budget with capped backoff and jitter, gated on the operation being idempotent (X) and bounded by the deadline the caller already owns (VI). Anything heavier (per-dependency budgets, a circuit breaker, adaptive backoff) is earned by evidence, not adopted by default.

Pitfalls. - Retrying a non-idempotent write. Reaching for backoff and jitter without first checking that a repeat is harmless (X), so the polish lands on the load problem while the correctness problem goes untouched. - Stacked retries. Your loop wraps an SDK that already retries, which wraps a load balancer that already retries; three attempts become twenty-seven, and the budget you reasoned about is fiction. - Backoff without jitter. Capping the delay but firing all clients on the same schedule still synchronises them into a thundering herd; jitter, not the cap, is the part that scatters the herd. - Retrying past the deadline. Continuing to attempt a call whose result no caller is still waiting for (VI), so the queue fills with already-dead work that re-buries the service on recovery. - No fallback behind the retry. Treating retry as the whole answer, so when the budget is exhausted there is no degraded tier (XIX) and the failure surfaces raw. - Shipping the degraded path untested. The retry-and-fallback logic first executes during the storm it was meant to survive, which is the worst possible time to find out it is wrong.

See also. Tenets (VI), (VII), (X), (XIX). the companion's second law. glossary: retry storm, jitter, thundering herd, unbounded retry wearing a helpful name. phases: integrating, operating.

Add a circuit breaker vs keep calling

You hit this when A downstream dependency is failing or has gone slow, and every call you make times out, ties up a thread or a connection while it waits, then lands as fresh load on something already on its knees. Your own retries are now part of what is keeping it down

The call. When a dependency is failing, do you wrap calls in a circuit breaker that trips open and fails fast, lean on retries with backoff and jitter, or leave the call path unprotected?

Circuit breaker. A small state machine in front of the call. It watches the recent failure rate, and once failures stay high it trips open: calls return immediately with an error or fallback instead of going to the wire. After a cooldown it lets a probe or two through, and re-closes only if they succeed. - Choose when: The downstream can be overwhelmed by the very traffic you are sending, so backing off as a fleet lets it recover instead of being held flat; A failing call is expensive to make: it ties up a thread, a connection, or a deadline budget while it waits to time out; You have a sane thing to return when open (a cached value, stale-but-serviceable data, a degraded response), so failing fast loses a feature rather than the page; The operation is idempotent and already deadlined, so anything the breaker sheds can be retried later without harm. - Cost: A second state machine with thresholds, cooldowns and probe logic that a tired engineer must understand before they can reason about why a healthy-looking call returned an error; Tuned wrong it is its own outage: too twitchy and it trips on a blip and sheds good traffic, too sluggish and it never opens when it matters; The open path runs almost never, so it is the classic untested degraded path that fires for the first time during the real incident; Per-instance breakers see only their own slice of traffic, so trip decisions can be noisy and disagree across the fleet.

Retries with backoff and jitter only. No breaker. Each failed call is retried a bounded number of times, with exponentially growing delays and randomness added to the delay so callers that failed together do not all retry on the same clock. - Choose when: Failures are genuinely transient and short: a brief blip, a leader election, a quick GC pause that the next attempt sails through; The downstream has ample headroom, so a modest retry multiplier cannot push it into collapse; You want the simplest thing that recovers from one-off errors without standing up extra machinery; Calls are cheap and fast to fail, so a retry costs little even when it is wasted. - Cost: Backoff slows the storm but does not stop it: against a sustained failure, retries are net-new load on something already down, and a flaky upstream becomes a self-inflicted DDoS; No fast path home. When the dependency is hard down, every request still pays the full timeout before giving up, burning threads and deadline budget; Backoff alone does not break the sustaining loop, so a latency blip can tip into a metastable retry storm that outlives the original cause; Without jitter (and even partly with it) synchronised retries re-converge into a thundering herd that re-floors the service the instant it tries to come up.

No protection. Call the dependency directly. On failure, surface the error or let it propagate. No retry, no breaker, no fallback. - Choose when: The call is on a non-critical, low-volume path where a raw error is acceptable and the blast radius is one user, one request; A layer above already owns containment (an upstream breaker, a queue, a sidecar or mesh policy), and duplicating it here would only confuse; The dependency is in-process or so reliable that the machinery would be speculative complexity you cannot yet justify; You are prototyping, and a real failure should be loud and visible rather than quietly absorbed. - Cost: Nothing dampens a feedback loop, so a single slow dependency can saturate your thread or connection pool and cascade an outage outward; A failure downstream becomes a failure everywhere upstream: with no fallback the whole page or job dies, not one feature; You have no fast path, so latency under failure is bounded only by your timeout, multiplied by every waiting caller; The cheapest option to write is the most expensive to operate the day the dependency has a bad hour.

How to decide. Decide by who controls the load on the failing dependency and how far its failure is allowed to spread. The breaker earns its keep precisely when you are the load: when the traffic you send can hold a struggling downstream flat, failing fast is the cooperative move, because it gives the downstream room to heal instead of hammering it through its own recovery, and it stops a blip turning into a self-sustaining retry storm that no amount of backoff will lift you out of. That is the whole case for the extra state machine, and it is real only when three things hold together: the call is expensive enough to fail that tying up threads on dead requests hurts, the operation is idempotent and deadlined so anything you shed can be safely retried later, and you have a fallback worth serving so an open breaker loses a feature and not the system. Where those do not hold, the breaker is just a second thing to tune and a degraded path you will never exercise until it misfires in the night, and bounded retries with jitter are the honest, cheaper answer. The governing question is the corpus-wide one: spend the coordination and machinery only where the property (the downstream's ability to recover, your pool's survival) is genuinely at risk, and where violating it would cost more than the breaker's complexity drags into every engineer's head. A breaker on a path that cannot start a feedback loop is clutter; its absence on the path that can is an unbounded liability dressed up as simplicity.

Reach for first. First make the call cheap to fail and bounded: a tight deadline so a hung dependency cannot pin a caller, a small bounded retry budget with exponential backoff and jitter, and a wait-or-reject pool so a slow downstream cannot consume unbounded threads. That removes most transient pain on its own. Add the breaker only once you can name a concrete fallback to serve when it is open and you have evidence (or a clear mechanism) that your own retries can keep the downstream from recovering.

Pitfalls. - Adding the breaker but having nothing to return when it opens, so a tripped breaker just converts slow failures into fast failures and still white-screens the page. - Never exercising the open path, so the fallback is silently broken and the first real trip is also the first time the code runs. - Tuning the threshold to the comfortable middle rather than the edge of collapse: too sensitive and it sheds good traffic on a blip, too lax and it never opens when it counts. - Wrapping a non-idempotent or undeadlined call, so the work the breaker sheds cannot be safely retried later and you trade an outage for corruption. - Retrying through a breaker, or stacking retries at several layers, so the multipliers compound into the storm the breaker was meant to prevent. - Trips and probes that fire silently, hiding the metastable state instead of escaping it; every shed and trip must announce itself or you are blind to the loop.

Shared pool vs per-tenant/per-dependency bulkheads

You hit this when Many callers, or many downstream dependencies, draw on one finite resource: a thread pool, a connection pool, a queue, a worker fleet. Right now everything shares it. The day one downstream goes slow, or one tenant sends a flood, every thread is parked waiting on that one thing and unrelated traffic starts timing out for a reason it cannot see

The call. Do you keep one shared pool of the finite resource, or partition it into bulkheads (per dependency, per tenant, or per cell) so one greedy caller or one stalled dependency cannot starve the rest?

One shared pool. Every caller and every dependency competes for the same finite resource: one thread pool, one connection pool, one queue. Capacity is fungible, so any request can use any free slot. - Choose when: The workload is genuinely uniform: callers and dependencies have similar latency and failure profiles, so no single one can monopolise the pool while the others sit idle; You are early and the traffic mix is still unknown; partitioning now would be guessing where the variation lands, and a guessed split is usually wrong; Utilisation matters more than containment: one pool packs load tightly and you have no spare headroom to strand in idle compartments; The blast radius of the whole resource stalling is already acceptable, for example a single-tenant internal tool whose one bad day affects only its one owner. - Cost: No containment: the slowest dependency sets the failure mode for everything, parking every thread until unrelated, healthy paths start failing too (tenet XIX); One greedy or hostile tenant can consume the whole resource; if any of that load is caller- or attacker-controlled, the blast radius is the entire system, which is exactly the wide-blast case the objective says to pay down now; Failures are correlated by construction: the pool is the shared substrate (Law I), so independence between callers is a claim the single pool quietly refutes; Recovery is all-or-nothing: you cannot shed or restart one tenant's work without touching everyone's.

Bulkheads / cells (per dependency, per tenant, or per cell). The finite resource is partitioned into compartments, one per downstream dependency, per tenant, or per cell, each with its own slice of threads, connections, or queue. A flood or a stall is confined to its own compartment. - Choose when: Load is caller- or attacker-controlled, so one tenant or one upstream could otherwise drain the shared pool and take everyone with it; One or more dependencies are slow or flaky enough that their stall would park a shared pool; a per-dependency pool exhausts only itself and stalls only its own callers (tenet XIX); Tenants have an isolation contract: a noisy or compromised one must not degrade the others, which is least privilege applied to a runtime resource (tenet XVI); You need a blast radius you can name in advance, so a poison input or a bad deploy takes one cell's fraction of customers rather than the whole fleet. - Cost: Stranded capacity: each compartment must be sized for its own peak, so the fleet runs at lower average utilisation than one shared pool at the same headroom, and idle slots in one bulkhead cannot rescue a starving neighbour; Real, ongoing complexity: routing, per-compartment sizing, and monitoring are all more to build, hold in your head, and get wrong (tenet XIX names bulkheads as genuine complexity, not free safety); False isolation is the trap: compartments that share a database connection limit, a host, or a zone are one pool in disguise (Law I), so the wall has to be verified disjoint or it contains nothing; An untested compartment is a second bug waiting for the worst moment: rehearse the day one fills up, or the degraded path runs for the first time exactly when you need it; Per-tenant pools multiply: thousands of tenants cannot each get a dedicated pool, so you fall back to cells (a shared pool per group), which dilutes isolation back towards the shared-pool failure mode inside a cell.

How to decide. Decide by who controls the load that can exhaust the pool, and how wide the harm spreads when it does. If the load is your own and uniform, a single pool is the cheaper correct answer: it packs capacity tightly and spares you the idle headroom and the routing logic a partition drags in. The case for a bulkhead is the second term of the objective rather than the first: you are not making the system easier to reason about, you are buying a bounded blast radius for a resource that some caller or some dependency can otherwise take down for everyone. So spend the partition where the demand is caller- or attacker-controlled (a multi-tenant fleet, an untrusted upstream), or where one dependency is slow or flaky enough that its stall would park the shared pool and the joint failure would be wide. Spend it nowhere else. An isolated pool carved out of self-controlled, uniform, well-behaved load is just stranded capacity, plus one more thing the tired engineer has to hold in their head when they touch the routing. And the partition only contains anything if the compartments are genuinely disjoint (Law I): two pools backed by the same database connection limit, the same host, or the same zone are one pool wearing two names, and the bulkhead is a comforting drawing rather than a real wall. Size each compartment for its own peak, confirm the substrate underneath is not itself shared, and rehearse the day one fills up, or you have bought the cost of isolation without the containment you paid for.

Reach for first. Before partitioning anything, bound the demands on the pool you already have: a deadline on every call so a slow dependency releases its thread instead of holding it, a per-call concurrency cap, a circuit breaker that fails fast when a downstream is sick, and a per-tenant rate limit at the edge. A deadline plus a breaker stops one stalled dependency from parking the whole pool, and it costs you a handful of config values rather than a partitioned resource and the idle headroom every compartment then needs. Reach for a true bulkhead only once you have a named resource that one caller or one dependency can still monopolise after the demand on it is already bounded.

Pitfalls. - Drawing the bulkhead in boxes but backing both compartments with the same connection limit, host, or availability zone, so the partition contains nothing the moment the shared substrate fails (Law I). - Sizing every compartment for its global peak, then finding the summed headroom costs more than the shared pool ever did, with most slots idle most of the time. - Partitioning self-controlled, uniform load where no caller can monopolise the pool: pure cost, no containment bought, and extra routing for the next engineer to misread. - Building the bulkhead and never exercising it, so the first time a compartment fills up in production is the first time the degraded path runs at all. - Giving every tenant its own pool until the pool count itself becomes the scaling problem, instead of grouping tenants into a bounded number of cells. - Adding bulkheads while leaving the calls inside them unbounded: with no deadline, a stalled dependency still parks its whole compartment, it just takes the rest of the system a little longer to notice.

Load shedding (fast reject) vs deep buffer; backpressure

You hit this when Arrivals are outrunning service. The queue is growing, tail latency is climbing, and you have to decide what happens to the work you cannot serve in time: refuse it at the door, push the pressure back to whoever is sending it, or hold it and hope the burst passes

The call. When work arrives faster than you can serve it, do you fast-reject past capacity, signal the producer to slow down, or buffer the overflow and serve it later?

Shed at the door (deadline-aware fast reject). Past a capacity threshold you return an explicit "full" (a 503, a "queue full") immediately, and you drop work whose deadline will expire before you could serve it. Rejection is a first-class response the caller has to handle. - Choose when: The work has a deadline and a stale result is worth nothing: a request the user has already navigated away from, a frame past its budget, a quote that has expired; You are at or past the edge of the bistable region and need a way back that reduces the load that caused the collapse rather than adding to it; You would rather give a fast honest "no" to some callers than a slow "maybe" to all of them; The caller controls arrival rate and you do not, so the only lever you own is what you accept. - Cost: Shedding too eagerly leaves throughput on the floor: a system that rejects at the first tremor never reaches the capacity it was bought for, so the trigger has to be tuned to the edge of the bistable region and not the comfortable middle; Every reject has to announce itself with a metric or a log, or you have hidden the overload rather than survived it; Picking what to drop well needs a per-item deadline to read; without one you are shedding blind and may drop work that was still serviceable; The rejected caller now owns the retry decision, and if its backoff is naive your shed bounces straight back as a retry storm.

Backpressure (slow the producer). Instead of dropping work, you propagate a "full" or a "not ready" signal upstream so the producer stops producing until you can take more. The bound is still real; the response to fullness is to throttle the source rather than to refuse the item. - Choose when: You control the producer, or it is a cooperative client that will genuinely slow down when told to: a streaming consumer, an internal pipeline stage, a pull-based reader; The work cannot simply be discarded, because each item matters and there is no acceptable dropped outcome; The producer can do something useful, or at least harmless, while throttled rather than spinning; Arrival rate is elastic at the source, so slowing the tap actually reduces inflow rather than just moving the queue upstream. - Cost: It only works if the producer obeys. An adversary or an indifferent client ignores the signal, and then backpressure silently degrades into an unbounded buffer or a hang; Slowing one producer can stall a whole pipeline behind it, turning local fullness into a system-wide freeze if you have not bounded the wait; The pressure has to be propagated honestly through every hop; one stage that absorbs the signal instead of passing it on hides the overload from the source that could act on it; For a public, caller-controlled edge it is often not available at all, because you cannot make a stranger's client throttle, which is exactly when you fall back to shedding.

Deep buffer / queue. You make the queue bigger and serve everything eventually, betting that a larger buffer absorbs the overload. It absorbs bursts well; under sustained overload it only postpones the rejection that was always coming. - Choose when: The overload is genuinely bursty rather than sustained: arrivals spike above service rate briefly and then fall back below it, which is the one case a buffer is for; Latency under the burst does not matter and no item has a deadline you can blow; You have measured that the burst integral fits the buffer and drains before the next one, so the queue returns to empty between spikes. - Cost: Under sustained overload it does nothing but delay the rejection, because a queue smooths bursts and does not add throughput, so arrivals outpace service no matter how deep the buffer goes; It fills with work already past its deadline, so capacity is spent producing answers nobody can use, and recovery has to fight a backlog of the dead; A large in-flight queue is itself a memory liability and a latency multiplier, hurting exactly the callers it was meant to protect; It is the most comfortable wrong answer. It looks like it is coping right up to the moment it melts, so it tempts you away from the bound you actually needed.

How to decide. Decide by who controls the arrival rate and how wide the damage spreads when you cannot keep up. A buffer is the right tool for one narrow case, a burst that fits and then drains, so reach for it only when you have measured that the queue returns to empty between spikes. Under sustained overload a deep buffer just delays the rejection and fills with work already past its deadline, so by the time you reach an item the result is dead and serving it is wasted capacity, which is precisely the exhaust that feeds a metastable collapse. The real choice is between shedding and backpressure, and it turns on control of the input. If you own the producer, or it will cooperate, push the pressure back: that preserves every item and is cheaper than throwing work away. If the arrival rate is controlled by a caller you do not own, a stranger's client or an unlucky retry loop, the only lever you hold is what you accept, so shed at the door with an explicit "full" and prefer freshest-first, because that bounds the blast radius of someone else's load to a fast honest rejection instead of letting it freeze the thread a user is watching. Spend the deadline-tracking machinery only where work actually expires and where violating the deadline costs more than the bookkeeping; for work that never goes stale, a plain bounded reject is enough.

Reach for first. A bounded queue with a wait-or-reject policy and an explicit "full" response. That single mechanism gives you a buffer for bursts and a fast rejection under sustained load, without committing to deep buffering or to a producer that may never throttle. Add deadline-aware shedding on top of it only once you have work that genuinely expires.

Pitfalls. - Treating a bigger buffer as the cure for overload: it raises the queue to where stale work piles up and makes recovery fight a backlog of the already-dead, instead of capping the load. - Shedding silently, with no metric or log on the reject path, so the overload is hidden rather than survived and you cannot see the bistable state you are in. - Tuning the shed trigger to the comfortable middle of the load curve, so you reject traffic you could have served and never reach the throughput the system was sized for. - Calling something backpressure when the producer is a caller you do not control: the signal is ignored and the bound quietly becomes unbounded. - Shedding without bounding the caller's retries, so every rejection returns immediately as new load and your defence becomes the engine of a retry storm. - Draining the queue oldest-first under overload, burning scarce capacity on requests that have already blown their deadline.

Degrade in tiers vs fail whole

You hit this when A non-critical dependency has just gone dark: the recommendations service, the avatar host, the feature store. The request that needed it can still produce most of its answer. You are deciding, ideally before the pager goes off, whether a request like this returns a usable page with one thing missing, or a 500

The call. When a non-critical dependency dies mid-request, do you shed the nice-to-have and serve the core, or fail the whole request?

Degrade in tiers: drop the nice-to-have, serve the core. You decide in advance what each request can lose and lose it gracefully: a cached or stale value, a default, an empty widget behind an error boundary, a slightly worse prediction. The core of the response still ships. The tier is chosen, built, and rehearsed before the incident rather than improvised during it. - Choose when: The dependency is genuinely non-critical: a recommendations strip, an avatar, an enrichment feature, a personalisation layer the user can do without; Staying up with less is the lesser harm, and the degraded answer misleads nobody, being stale-but-labelled or visibly absent; You run a long-lived service where halting is an outage, so partial availability beats going dark; You will actually exercise the fallback, through drills or fault injection, so it is not dead code. - Cost: It is the costliest term of the objective to pay for: a fallback path is real code and real state you carry forever, in service of a failure that is rare by design; Every tier you build but never run is an untested degraded path, a second bug primed to fire at the worst moment, once the primary has already failed; More branches in the request handler means more for a tired engineer to hold in their head: the happy path, the cached path, the empty path, and how they interact; Silent degradation hides the very failure it survives; without a counter and an alert you ship a worse product for weeks and nobody notices.

Fail the whole request. Treat the dependency as load-bearing: if it is unavailable, the request returns an error rather than a partial answer. The classic fail-fast stance, leaning on tenet XIII. - Choose when: Carrying on without the dependency would produce a wrong answer that other people trust, or quietly corrupt downstream state; The request is a write, a transaction step, or a pipeline stage where a partial result is worse than no result; The 'non-critical' part is not actually optional once you trace what depends on it, such as a missing field that later code assumes is present; The blast radius of a wrong-but-served answer exceeds the blast radius of an honest error. - Cost: You convert a feature outage into a request outage; the user loses everything, not the one thing that broke; A single flaky downstream can take the whole endpoint's availability with it, coupling your uptime to its worst day; On a long-running service this is often the larger harm: a degradable page returned as a 500 is availability you threw away; It can mask the cheaper truth that the dependency was never critical, so you fail hard out of caution rather than from a real corruption risk.

Untested fallback. A degraded path exists in the code but has never been run in anger: no drill, no fault-injection test, no production exercise. It looks like tier-one degradation on the org chart and behaves like a coin flip in an incident. - Choose when: Never as a deliberate choice. It is listed because it is where 'degrade in tiers' silently decays to if you stop rehearsing; If you find yourself here, the honest move is to either wire up the rehearsal or delete the path and fail whole. - Cost: You pay the full complexity of the fallback and get none of the safety: it is unverified code wearing a safety-net costume; It fails at the exact moment it is invoked, because that moment is also when the primary failed, so both paths are exercised for the first time together under load; It defeats whole-path probing (law V): the route reports a graceful-degradation design while the actual degraded route is broken, green parts over a red whole; Discovering it is dead during an incident is worse than having no fallback, because you planned around safety that was never there.

How to decide. Settle it by who controls the consequence and how far a wrong answer travels, rather than by a standing preference for staying up. Degrade where serving less is the lesser harm and the cut-down answer misleads nobody: the user loses one widget, downstream state is untouched, and the missing piece really was optional once you traced what reads it. Fail whole where carrying on would emit a wrong result that someone trusts, or would quietly corrupt state downstream, because a loud stop is cheaper than corruption nobody notices. That is the abort-versus-degrade call decided by blast radius, the same tie-break the rest of the corpus resolves to. Then weigh the second term honestly: degradation is one of the few places you deliberately add code and runtime state to contain a failure, so spend it only where the property is real. A tier earns its weight when the dependency genuinely is non-critical, the failure is plausible, and the fallback is one you will rehearse. It is dead weight, more branches for a tired engineer and a latent second bug, when the part was never optional or the path will never be exercised. The decision is not finished when you pick a tier. It is finished when that tier is wired to a counter, surfaced so its use is visible rather than silent, and run end to end at least once before the incident that needs it.

Reach for first. Remove the dependency from the critical path entirely, so the question never arises. If the recommendations, the avatar, or the enriched feature is genuinely not needed to answer the request, make the call asynchronous, precompute it, or render the core without ever blocking on it. A dependency that cannot fail your request needs no tier and no fallback. Only once the call is unavoidably in-band do you owe it a degradation decision.

Pitfalls. - Calling a dependency non-critical without tracing what reads its output: the missing field a later stage assumes is present turns graceful degradation into a null-pointer crash two frames down. - Building the tier and never running it, so the first execution is in production during the incident the primary already caused. - Degrading silently with no counter or alert: you serve a worse product for weeks and the dashboards stay green, because each component is honestly healthy (law V). - Defaulting to fail-whole 'to be safe' when the part was genuinely optional, throwing away availability you could have kept. - Letting the fallback itself depend on the thing that just died (a cache fronting the same store, a default fetched from the same service), so it fails in lockstep with the primary. - Treating degradation as licence to ship a confidently wrong answer: stale data served as fresh, or a default prediction with no signal that it is a default.

See also. Tenets (XIX), (XIII). the companion's fifth law. glossary: degrade in tiers, an untested degraded path, choose by blast radius. phases: operating, integrating.

Fail fast (halt) vs degrade (stay up)

You hit this when A step has failed: a non-zero exit, a thrown exception, a dependency timing out, a record that won't parse. The code beneath you is now in a state you didn't plan for, and you have to choose in that frame whether to stop the world or carry on with less. Halting might be an outage nobody can clear until morning; carrying on might write bad data that surfaces a week later, miles from here

The call. When something goes wrong, do you halt the whole operation loudly, keep running in a reduced mode, or let the failure pass quietly?

Fail fast (halt). On the first failure, stop the whole operation immediately and loudly, before any further work runs. The classic mechanism for glue is set -euo pipefail at the top of a script; the equivalent in code is to let the error propagate to a top-level boundary that aborts the run with a non-zero exit and a legible message. - Choose when: The work is a chain where each step trusts the one before: a batch job, a migration, a data pipeline, a glue script. Carrying on past a bad step corrupts everything downstream; A wrong result would propagate silently into state other people trust, and a loud stop is cheaper than corruption nobody notices for a week; The run is short-lived and re-runnable, so aborting costs a retry rather than an outage; You can't yet name a safe reduced answer, so 'less' would actually mean 'wrong'. - Cost: A halt is a hard stop for everyone on this path. If the failing thing sits on a hot request route, fail-fast is itself the outage; Partial work already done may need cleaning up or making idempotent before the retry, or the re-run doubles it; It tunes you toward stopping on failures that were survivable; an over-eager set -e can abort on a grep that legitimately found nothing; A noisy halt that fires often trains operators to rerun on autopilot without reading why it stopped.

Degrade (stay up). Catch the failure at a boundary and continue in a reduced mode: serve a cached or stale value, a default, or a page with one widget missing rather than a 500. This is the retry-and-fallback discipline of tenet XIX, built before you need it and rehearsed. - Choose when: The thing is long-running and shared: a service, a UI, a request handler where halting is the outage and a reduced answer is the lesser harm; You can name a fallback that is honest: stale-but-serviceable data, a default feature vector, a widget that quietly disappears, none of which mislead the caller; One dependency's death should cost one feature rather than the system, and you have a bulkhead or breaker to keep it contained; The reduced path is one you'll actually exercise, not a branch that only runs at 3am. - Cost: Fallbacks and bulkheads are real machinery that has to be written, owned and tested. An untested degraded path is a second bug waiting for the worst moment; A degraded answer that looks normal can mislead: serving stale data without saying so quietly lies to whoever trusts it; Staying up on bad input can let corruption accumulate behind a green status, the failure XIII exists to stop; More states for the next reader to hold: every fallback is another branch and another 'what is true right now' to reason about.

Swallow the error (the sin). Catch the failure and do nothing: an empty catch, a bare except, a script that ignores a non-zero exit and ploughs on. This is less a third stance than the trap both real stances are defined against. It is listed because it is the default you fall into when you pick neither on purpose. - Choose when: There is no good case. It is named only so it can be ruled out: whenever halting feels too drastic and degrading feels like too much work, this is the tempting middle, and it is wrong. - Cost: The failure doesn't go away; a swallowed error is a wrong state that has learned to hide, and it resurfaces far from where it started, far harder to trace; The happy path becomes a lie: code that appears to succeed is concealing a failure underneath, so success no longer means success; Nobody is told and nothing is logged, so the one signal that could have caught it early is gone; It is the silent swallow that is the sin, not any keyword: the same empty handler is the defect whether it's a catch, an except, or an ignored exit code.

How to decide. Decide by blast radius and by who controls what happens next. Ask where a wrong result goes if you keep running: if it propagates silently into state other people trust, or into the next step of a chain that assumes the last one was clean, abort, because a loud stop is cheaper than corruption nobody notices. If staying up with less is genuinely the lesser harm, meaning you can serve a stale or cut-down answer without misleading anyone, degrade. The split tracks the shape of the work: a glue script, batch or migration should halt on the first bad step, because carrying on wrecks everything downstream; a long-running service or UI should degrade, because for it the halt is the outage. Spend the machinery of fallbacks and bulkheads only where the reduced answer is both real and honest, and where you'll actually run that path, since an untested degraded branch buys you a second failure at the worst moment rather than resilience. And whichever way you go, the failure stays visible: the one option ruled out in every case is the silent swallow, because it leaves the next tired engineer reading a happy path that is lying to them. That is the tie-break here. Keep failure legible, stop it where a wrong result would silently spread, and pay for a fallback only where the property it buys is worth more than the branch it adds.

Reach for first. First try to not have the failure mode at all (XXI): a stateless, idempotent step can't be corrupted by a re-run, so fail-fast-then-retry becomes free and you need no fallback. Failing that, the cheapest correct default is to make the failure loud and let it propagate with minimal ceremony to one deliberate boundary: set -euo pipefail for a script, a single catch or Result at the top of the call for code. Halt-and-be-legible is the right baseline; reach for a degraded mode only once you can name the fallback and the path you'll rehearse to keep it honest.

Pitfalls. - Defaulting to degrade on a batch job or migration, so a failed step is swallowed and the run finishes 'green' having corrupted everything after the bad row. - Defaulting to fail-fast on a hot request path, where the halt you reached for is itself the outage you were trying to avoid. - Building a degraded path and never running it, so the fallback is a second untested branch that fires for the first time during the incident. - A degraded answer that looks identical to a healthy one: serving stale data with no marker, quietly lying to whoever trusts it. - Reaching for the empty catch because halting feels too drastic and a real fallback feels like too much work; that middle is the silent swallow, not a compromise. - An over-broad set -e that aborts on commands whose non-zero exit was expected, such as a grep with no match, turning fail-fast into flakiness.

Cap caller-driven growth vs leave it (YAGNI)

You hit this when You are writing code where something can grow: a list that accumulates, a payload you accept, a retry loop, a recursion, a count of open connections or in-flight jobs. Nothing in the current tests goes anywhere near a dangerous size, so the ceiling feels like make-work, and a colleague has already said the magic word, YAGNI

The call. When something can grow, do you put a ceiling on it now, or leave it unbounded and add a limit only if it ever turns out to matter?

Cap it (for caller-, input-, or time-driven growth). Give the thing an explicit maximum: a bounded queue or pool, a cap on retries with backoff, a payload size limit, a recursion depth guard, a ceiling on connections or batch size. When the ceiling is hit you do something defined and survivable, such as reject with a 503, shed load, truncate, or fail the one request rather than the whole process. - Choose when: The growth is driven by a caller, by external input, or by elapsed time: anything where you do not control how big it gets; Hitting the ceiling is a survivable event you can name (reject, shed, truncate, back off) rather than a process death; An attacker or an unlucky caller could push it, so the blast radius of no limit is an OOM or a DoS that takes down everything sharing the host; Your own code is a possible source: a retry storm against a flaky upstream is a self-inflicted flood, not just an external threat. - Cost: You must pick a number, and a wrong ceiling pages you at 3am for traffic that was legitimate; the limit needs a rationale and sometimes a config knob; A reject-or-shed path is real behaviour you now have to design, test, and explain to callers, which is more surface than doing nothing; Too tight a cap converts a transient spike into visible errors for real users who would otherwise have been served; The ceiling adds a branch and a failure mode to read, genuine complexity you are spending against the simplicity budget (XXI).

Leave it unbounded (YAGNI). Add no limit. The list grows as far as input takes it, the retry loops until it succeeds, the pool spawns a connection per caller. You rely on the observation that nothing realistic gets near a dangerous size today, and you keep the code free of a guard nobody is exercising. - Choose when: The set is genuinely fixed and you own its size: the twelve months, the suits in a deck, an enum you control at compile time; Growth is bounded upstream by something you can point to, so the ceiling already exists and re-stating it would be a second source of truth; No caller, no external input, and no clock can push the size; only your own code, in a path that itself has a known bound, ever appends; The thing is short-lived and scoped so tightly that it cannot accumulate across requests. - Cost: The moment any caller or input drives the growth, this stops being simplicity and becomes a latent denial of service or out-of-memory shipped to production; The failure lands far from the missing bound, in time and in the stack, so the eventual incident is baffling to trace back to its cause; "It never gets big in practice" holds right up until real traffic, accumulated data, or hostile input proves otherwise, and then it goes all at once; What looks like one fewer thing to build is a safety requirement quietly skipped, dressed up to look like restraint.

How to decide. Settle it by asking who controls the growth, not by guessing how big it gets today. If a caller, an external input, or the clock drives the size, the blast radius of leaving it unbounded is the whole process: one skewed key, one retry storm, one hostile payload, and you are out of memory or serving a self-inflicted DoS. That is a real property being violated, and the violation costs far more than the branch a cap drags in, so you spend the complexity and add the ceiling. Where you own the size and it is genuinely fixed (the months of the year, an enum you compile in) there is no property to protect, and a limit would only be a second, drift-prone copy of a bound that already exists, so leave it. The YAGNI objection is sound but mis-scoped: it governs features, the speculative flexibility you can always bolt on later, whereas a missing limit is a safety requirement quietly skipped. The whole thing reduces to one line a tired engineer can apply without holding the system in their head: caller-, input-, or time-driven growth always gets a ceiling, and the fixed set you own does not.

Reach for first. Before reaching for a configurable maximum or a load-shedding subsystem, check whether the growth is already bounded by something you own. If it is, write nothing and note why. If it is not, use the cheapest ceiling the language hands you: a bounded queue or pool with a wait-or-reject policy, a fixed retry count with backoff, a size check that rejects the request. A small constant cap with a clear reject path is enough; you do not need an elaborate adaptive limiter yet, and an unbounded list is not an option.

Pitfalls. - Treating the limit as a feature and deferring it under YAGNI, when a missing ceiling on caller-driven growth is a liability rather than a deferred nice-to-have. - Capping the size but letting the over-limit path crash the process, so the ceiling exists yet the blast radius is unchanged. - Forgetting that your own retries are caller-driven growth too: uncapped retries against a wobbling upstream amplify the wobble into a flood, the metastable loop that does not let go even once the original blip has passed. - Setting the cap from dev traffic, which never came near it, so the number is arbitrary and either useless or a 3am page when real load arrives. - Picking such a tight limit that a normal spike becomes user-visible errors, then concluding limits are bad rather than that this one was wrong. - Adding a config knob for every cap reflexively, multiplying the states a reader must reason about when a sensible constant would do.

See also. Tenets (VII), (XXI). the companion's second law. glossary: a missing limit is a liability, unbounded just fails later, YAGNI. phases: implementing.

Compose control loops on purpose vs tune each independently

You hit this when An autoscaler, a circuit breaker, a cache, and a rate limiter all act on the same running system, and each was built and tuned in its own room. In an incident they begin to flap: capacity is removed just as the breaker sheds load, retries beat against the limiter's window, and the correction deepens the fault it was meant to fix. Every controller is doing precisely its job, yet the system as a whole hunts where it was meant to settle

The call. When several feedback loops act on overlapping signals in one system, do you design their composition deliberately, or tune each loop in isolation and trust that stable parts make a stable whole?

Compose the loops on purpose. Treat the set of controllers as one system. Write down which signal each loop reads and which it writes, keep two loops that touch the same variable on different timescales so the fast one settles before the slow one stirs, and where they genuinely must move the same lever, give one loop authority and make the others defer. - Choose when: Two or more loops demonstrably read or move the same variable: the autoscaler and the load-shedder on the same latency number, the retry policy and the breaker on the same error stream; Their timescales are close enough to line up, so a correction by one arrives while the other is still chasing the signal it left behind; The shared variable is something a caller or a fault can swing hard, so a fight between loops widens the blast radius rather than narrowing it; An oscillation has already shown up in a game day or an incident, or one in production would be an outage you cannot afford. - Cost: You must model the coupled system, which is the cognitive load you were trying to avoid by keeping the controllers separate; Damping and timescale separation cost responsiveness: a loop slowed enough never to oscillate may be too slow to catch a real fault when one arrives; Naming one loop the authority over a shared lever pulls against locality and against the independent deployment that made these separate controllers in the first place; The composition becomes a thing someone owns and must re-check whenever any single loop is retuned, so a local change is no longer safely local.

Tune each loop independently. Prove and tune each controller on its own. Each is stable in isolation, ships and scales on its own schedule, and nobody has to hold the interaction between them in their head. - Choose when: The loops act on genuinely disjoint signals and levers, so there is no shared variable for them to fight over: most controllers in most systems are like this; The loops sit on timescales far enough apart that the fast one has fully settled before the slow one notices, by construction rather than by luck; You have a level-triggered reconciler behind the events (VIII) that converges the system regardless of the order corrections land in; Independent ownership and independent deployment matter more here than squeezing out the last of an interaction you have evidence is absent. - Cost: A controller proven stable alone tells you almost nothing about the whole: two loops each stable in isolation can compose into one unstable loop the instant their timescales align; The failure lives in the interaction, which no single loop specifies and no single dashboard can see, so it surfaces in production rather than in review; Each loop is an honest witness to the wrong question, and you can tune one in its own room while it drives another into a rhythm it never meant to set up; When it does oscillate the symptom is systemic flapping with no obvious culprit, the hardest kind of fault to attribute and the slowest to settle.

How to decide. Decide by the shared variable and who can swing it, not by how many loops you have. For each pair of controllers ask one concrete question: do they read or write the same signal, and are their timescales close enough to overlap? If the answer is no, leave them independent, because the coupling a deliberate composition drags in (a modelled whole, a named authority, a local change that is no longer local) costs more than an interaction that is not real. If the answer is yes, the property you must protect is that the loops settle rather than fight, and here that property is genuinely violated, so spend the coordination: separate their timescales, or give one loop authority over the lever and make the rest defer. Spend it most where the shared variable is something an attacker or an unlucky caller controls, because there an oscillation is not merely slow recovery but a blast radius that widens under exactly the load you least want to amplify. Pay the damping only on the pairs that demonstrably couple, and keep a level-triggered reconciler (VIII) behind the whole set so that whatever order corrections arrive in, the system still converges. The tired engineer should have to reason about two loops interacting only where two loops actually interact.

Reach for first. Before modelling anything, check whether the loops even share a variable. Most do not, and the cheapest correct answer is to keep them independent and confirm by construction that their timescales cannot line up. If exactly two do couple, the next cheapest fix is timescale separation alone (slow one loop until the other has settled) before you reach for naming an authority over a shared lever. And put a level-triggered reconciler (VIII) behind the set so correctness does not depend on the order corrections arrive in.

Pitfalls. - Treating "each loop is proven stable" as evidence the system is stable; isolation stability is the one fact that tells you almost nothing about the composition. - Tuning a retry policy and a circuit breaker in separate rooms, so the breaker opens and closes in a rhythm the retries themselves drive, flapping where it was meant to heal. - Two loops moving the same lever with no agreed authority, each erasing the other's correction, so the system hunts forever and no single dashboard shows why. - Damping a loop so heavily that it never oscillates and never corrects a real fault either, trading a rare oscillation for a permanent loss of responsiveness. - Modelling the coupled system everywhere out of caution, paying the cognitive load the parent exists to spare you on loops that never actually interact.

See also. Tenets (VIII). the companion's fourth law. glossary: control loop, coupled-loop oscillation. phases: integrating, operating.

Build a load-reducing way back vs assume it recovers

You hit this when A latency blip, a brief dependency wobble, or a capacity spike has passed, yet the system is still on the floor. You have fixed or removed the original cause and nothing has improved. Throughput stays low, queues stay full, and the only thing keeping the outage alive is the system's own response to being overloaded

The call. When a trigger can tip the system into a collapsed state held open by its own feedback, do you engineer a response that actively reduces the load it caused, or do you rely on removing the trigger to restore health?

Build a load-reducing way back. Treat recovery as a behaviour you design rather than a state the system drifts back to. The response to overload must subtract load: shed work that is already past its deadline, coalesce duplicate work into one shared unit, cap and de-synchronise retries, and trip a breaker, so the sustaining feedback loop runs at a gain below one and each turn produces less load than the last. - Choose when: The system has a sustaining feedback loop, where its own response under stress (retries, regeneration, reconnection) becomes the load that keeps the stress going; Load is driven by callers or external input you do not control, so a backlog of already-dead work can pile up faster than you serve it; The path down and the path up differ: removing the trigger demonstrably does not lift the system back out, so recovery needs an active push; You run near a known tipping point, where a transient spike can flip you into a collapse you will not climb out of unaided. - Cost: The way back is itself behaviour, so it has to announce every shed, every coalesce, and every breaker trip, or you have hidden the collapsed state rather than escaped it; A trigger tuned too eagerly sheds at the first tremor, and the system never reaches the throughput it was bought for; More moving parts on the hot path: a shedding policy, a freshness order, a coalescing key, a breaker. Each is one more thing for a tired engineer to reason about and get wrong; The mechanisms interact. A breaker plus aggressive shedding plus synchronised backoff can compose into a fresh oscillation unless you tune them as one control loop.

Assume removing the trigger restores health. Treat the outage as caused by an external fault and recoverable by clearing that fault. Keep the edges polite (timeouts, sane defaults) and trust that once the dependency is back, the slow database is unstuck, or the spike has passed, the system returns to its healthy operating point on its own. - Choose when: The failure is genuinely monostable: there is no feedback loop in which the system's response feeds its own overload, so clearing the cause is sufficient; Load is bounded and self-limiting (a fixed set of callers, no caller-driven retry amplification, no regeneration on a miss); You operate with comfortable headroom, well clear of any bistable region, so a transient never reaches the tipping point; The blast radius of being wrong is small, and a manual restart or a hand-drained queue is a cheap, acceptable recovery. - Cost: Once the failure is metastable, this is simply incorrect: hysteresis means the push that knocked you down will not lift you back up, so you wait for a recovery that never comes; The congestion sustaining the collapse usually also blinds the monitoring you would reach for, so you hunt in the dark for a cause that has already swapped places with the symptom; Recovery becomes a manual, heroic act, performed under pressure at 3am and repeatable only by whoever happens to know the runbook; It is the cheapest design right up until the first collapse, and then the cost lands all at once, in production, during the worst possible window.

How to decide. Decide by who controls the load and how far a collapse spreads. The governing question is not whether any single edge has a deadline but whether the system as a whole has a way back: if the trigger vanished this second, would it climb out on its own, or is something it is still doing holding it under? Where the load is caller-driven or external-input-driven, and the response under stress feeds its own overload (retries that become load, misses that trigger the regeneration that causes the next miss), the failure is metastable and the recovery property is real. You must spend the machinery, because hysteresis means no amount of fixing the trigger buys recovery back, and the blast radius is the whole fleet rather than one request. Spend it exactly where the loop is, and no wider: the smallest correct cure is to push the loop's gain below one (cap and jitter retries, shed work already past its deadline, coalesce duplicate regeneration) and to make every such action observable, since a way back you cannot see is a way back you cannot trust. Where load is self-limiting and you run well clear of the bistable region, the loop does not exist, the property is not real, and elaborate shedding only adds hot-path complexity a tired engineer must reason through for a danger that cannot occur. The honest tie-break is the cost of being wrong: assuming recovery is cheap until the day it is catastrophically and silently incorrect, so reserve that assumption for systems you can prove are monostable, and design the load-reducing way back wherever a sustaining feedback loop is even plausible.

Reach for first. First remove the loop instead of taming it. The cheapest correct answer is a response that cannot amplify: cap retries with a budget, add jitter to de-synchronise clients, coalesce duplicate work behind one shared computation, and trip a breaker when a dependency is down, so there is no positive feedback to sustain a collapse in the first place. These are local manners (tenets V, VI, VII, XIX) and they are necessary; only once the edges are disciplined do you ask the whole-system question of whether a backlog of already-dead work still needs shedding, and whether your trigger sits at the edge of the bistable region rather than in the comfortable middle.

Pitfalls. - Hunting the trigger: chasing what started the outage when cause and symptom have swapped places, so the fix you ship has no effect because the system is now its own cause. - A deep buffer mistaken for a cure: a bigger queue just delays the rejection and fills with work that is already past its deadline, so recovery has to fight a queue full of the dead. - Shedding silently: a breaker or load-shed that fires without announcing it hides the metastable state instead of escaping it, and the next engineer cannot tell recovery from outage. - Tuning to the comfortable middle: setting the shed trigger far from the bistable edge, so the system sheds at the first tremor and never delivers the throughput it was sized for. - Synchronised recovery: clients that all back off on the same clock re-converge into a fresh thundering herd the moment the dependency returns, restarting the storm.

Errors as values vs exceptions caught at a boundary

You hit this when A function can fail, and you have to choose how that failure reaches the caller who holds enough context to decide what to do about it. The choice is not a moral one between keywords. It is a choice about where the handling decision is forced to happen, and how loudly the language complains when nobody makes it

The call. When this call fails, how does the failure travel to the one frame that can actually decide what to do, and what stops it being ignored on the way?

Errors as values (Result / (value, err)). Failure is an ordinary return: a Result, an Either, a (value, err) pair. The caller cannot reach the success value without first naming the failure, and the type system or linter nags at the call site until it does. Handling lives exactly where the call lives. - Choose when: You are in a language that gives you sum types or the (value, err) convention as idiom: Rust, Go, Swift, anything with a real Result. Working with the grain costs nothing extra; The failure is expected and local: a parse that may not parse, a lookup that may miss, a write that may conflict. The caller one frame up usually has the context to decide; You want the set of failure modes legible from the signature, so a tired reviewer can see every way the call fails without running it; You are crossing a trust boundary (IV) and want the unhappy case to be as typed and as in-your-face as the happy one. - Cost: Ceremony at every frame. If the deciding context is ten frames up, ten frames have to thread or wrap the error, and each wrap is a place to lose information or relabel it wrongly; Without sugar (?, try, map_err) the threading is loud enough that people reach for the unwrap or the discard, which is just a silent swallow wearing a type; A Result you ignore is still ignorable in many languages: the compiler permits dropping it unless the type is marked must-use. The nag is real only if the tooling enforces it; Pushes the decision to the call site even when the call site is the wrong place to decide, encouraging premature handling of an error that should have travelled further.

Typed exceptions caught at one deliberate boundary. Failure propagates as a typed exception with little ceremony in the middle, and is caught at one boundary you chose on purpose: the request handler, the job runner, the top of the unit of work, where the context to log, retry, or surface lives. In a language where exceptions are the idiom this is the primary error-handling path, not a last resort. - Choose when: You are where exceptions are the grain: Python's EAFP, Ruby, Java with checked or documented types. Fighting it with hand-rolled Results buys you noise, not safety; The deciding context is genuinely far from the failure, and the frames in between have nothing useful to say. Propagation keeps them clean; Many call sites share one sensible policy (abort this request, roll back this transaction, return a 500 with a trace id), and one boundary expresses it once; You want the middle of the code to read as the happy path while still failing safely, with the catch concentrated where someone can act. - Cost: The failure modes are invisible in the signature. A reader in a middle frame cannot see what blows up beneath them, so the happy path is a lie until proven otherwise; The boundary must actually exist and actually be deliberate. A catch that is too broad turns into a swallow; a boundary placed by accident drops the very context it was meant to hold; Control flow leaps. Resources opened in the skipped frames must be released by the language (RAII, with, ensure, defer) or they leak on the way past; Untyped or over-typed catches erase which failure happened, so the boundary handles all of them the same and the specific recovery is lost.

Error codes (embedded / C style). The function returns an integer or sets errno; the caller checks it against known values. No allocation and no unwinding, just a convention the reviewer has to police. The discipline lives entirely in that convention. - Choose when: You are in a constrained environment (embedded, a kernel path, an allocation-free hot loop) where exceptions and heap-backed Results are off the table; You need a stable ABI across a language boundary, where an integer code travels and a rich type does not; The set of failures is small, fixed, and documented, and every caller really does check. - Cost: Nothing forces the check. An unchecked code is the cheapest silent swallow there is, and the compiler says nothing; The code carries no context: which file, which row, which call. You reconstruct the story by hand later; Easy to confuse with a valid return (the classic read() returning -1, or 0 meaning two different things), so the failure hides inside the success channel.

Bare except / empty catch (the silent swallow). The failure is caught and discarded. No log, no rethrow, no value that says it happened. The code carries on as if nothing went wrong. This is not an option you choose so much as one you slide into; it is named here so it can be refused. - Choose when: Effectively never as a deliberate design. The only honest use is catch-narrow-log-and-continue where continuing is genuinely correct, and that is no longer a swallow; If you truly mean to ignore a specific, expected, harmless failure, say so in code: catch that one type, comment why, and leave a trail. - Cost: The failure does not go away; it becomes a wrong state that has learned to hide, surfacing later and far from where it began; It makes every frame above it untrustworthy: the happy path now lies, because success no longer means success; The worst ones are in distributed code, where a swallowed partial failure leaves no stack trace at all and shows up only as a latency cliff or a creeping error rate; Whichever keyword got you here, the empty catch is the bug; the language is innocent.

How to decide. Start from who controls the input and how wide the failure spreads, not from which keyword your tribe prefers. The invariant is the same in every language: a failure must stay visible and impossible to ignore between the point it happens and the point someone can decide. So pick the mechanism that your language nags hardest about, and place the decision where the context to make it actually lives. If the deciding frame is one or two hops up and the failure is expected and local, errors as values keep the unhappy case in the signature where a tired reader sees it for free; that is usually the cheaper correctness. If the deciding frame is far away and the frames between have nothing to add, a typed exception caught at one deliberate boundary keeps the middle clean without hiding anything, provided the boundary is real and the catch is narrow. The tie-break bites on blast radius: the wider the radius of a missed failure, and the more of the input an attacker or an unlucky caller controls, the less you can tolerate any path that lets the failure go quiet, so spend the ceremony of values or the discipline of a typed boundary precisely there. Where the radius is tiny and the failure obvious at the call site, do not drag in machinery to prove a property that was never in doubt. And in every language the same line holds: the sin is the silent swallow, not the keyword, so the only mechanism you may never reach for is the empty catch.

Reach for first. Use whatever your language makes idiomatic, with the failure modes named in the signature or the catch type, and turn on the tooling that nags: must-use Results, an unhandled-rejection check, a linter that flags bare except. The cheapest correct move is the one that makes ignoring a failure cost a compiler error rather than a production incident.

Pitfalls. - Reinventing Result by hand in a language built around exceptions (or throwing for control flow in one built around values): you pay the ceremony of both idioms and get the safety of neither. - A catch (Exception) or except: at the boundary that is too wide to be a boundary. It catches the failure you planned for and the bug you did not, and handles both as if they were the same. - unwrap, .get(), or ignoring a (value, err) because threading it was tedious. That is a silent swallow with a type annotation on it. - Catching to log and then carrying on as if it succeeded, when continuing corrupts state. A logged swallow is still a swallow if the next line trusts the value that never arrived. - Returning an error value but leaking the resource opened just before it, because the early return skipped the cleanup the happy path relied on. - Expecting a stack trace from a distributed partial failure. The interesting production failures are statistical and silent; instrument rates and trends, do not sit waiting for a throw that never comes. - Placing the catch boundary by habit (top of main, a global handler) rather than where the context to decide lives, so the one frame that knew how to recover never got the chance.

Profile before optimising vs fix on sight

You hit this when You are looking at code that might be slow, and the question is whether you reach for a profiler or reach for a fix. The honest default is the profiler: your intuition about where the time goes is usually wrong, and every optimisation made on a guess spends clarity you may not get back. Two shapes are the exception, fixed on sight with no profile, because neither is really a performance question. The first is cost that grows super-linearly in a size the caller or attacker picks, a denial-of-service bug wearing a performance costume. The second is a small set of antipatterns reliable enough to recognise by eye. Outside those, measure first

The call. Is this a tuning question I can only answer with numbers, or a structural bug I can already name?

Measure first, then make the proven common case fast. Run a profiler under realistic load, find where the time actually goes, and optimise only the hot path the measurement proved. Leave the cold 95% legible. A performance claim you have not measured is a vibe, not a property of the running system. - Choose when: The code is self-controlled in size and the path is cold or unfamiliar, where your guess about the bottleneck is least trustworthy; Making it faster would cost clarity, and you cannot yet point to a number that says the clarity is worth losing; The slow-looking thing is on a path no realistic input drives hard, so the suspected cost may never materialise; You are tempted to optimise three things at once and have measured none of them. - Cost: A profiler run under realistic load is real work: you need representative data, a warm system, and the patience to read the result instead of your priors; Discipline tax. ‘Measure first’ feels slow next to a fix you can see, and on a tight deadline the measurement is the step that gets skipped; It can lull you into deferring a genuine blowup because the dev-data profile came back clean: the smallness of the test set is exactly what hid the cliff.

Fix on sight: super-linear cost in caller- or attacker-controlled size. Treat complexity that grows faster than linearly in input someone else controls as a correctness and DoS bug, and bound or rewrite it now. No profiler. The accidental O(n squared) on a request body, the unbounded allocation proportional to a field the caller sets: these are tenet IV and tenet VII failures, not tuning. - Choose when: The input size is set by a caller or an attacker, not by you, so they choose how bad the day gets; The blast radius is the process or the fleet: one crafted request exhausts the heap or pins a core; The failure is structural, a matter of the exponent and not the constant factor, so a profiler would only confirm what the shape already tells you; Small dev datasets pass it clean and would keep passing it clean until production traffic arrives. - Cost: Bounding untrusted size adds a cap, a rejection path, and the error handling around it: machinery that the happy path never exercises; A cap you set too low rejects legitimate large inputs, so you owe a defensible limit and a clear error, not a magic number; Rewriting the algorithm can drag in coupling or a data structure the cold path did not need, spending some of the simplicity tenet XVII is otherwise protecting.

Fix on sight: the known antipatterns. Fix the handful of shapes reliable enough to recognise without measuring: the N+1 query, the per-row loop over a vectorisable operation, the per-frame allocation in a hot loop. Batch the N+1 into one join or IN; push the filter into the engine; hoist the allocation out of the loop. The shape is the evidence. - Choose when: You can name the pattern on sight: a loop that lazily loads one association per row, a per-element call where a batched one exists; The cure is the obvious one and well understood, so you are not trading legibility for a clever trick; The cost stays hidden at dev scale and only bites once the row count or frame rate climbs, so waiting for a profiler means waiting for production to find it; The fix is local and leaves the surrounding code as readable as before, or more so. - Cost: Recognition can misfire: a loop that looks like an N+1 may run twice in practice, and you will have spent effort batching a thing that was never hot; Batching changes the shape of the code and sometimes the transaction or fetch semantics, so the cure has to be checked, not just applied; The list is short on purpose. Treat ‘I recognise this’ as a licence to optimise anything and you are back to guessing, which is what measure-first exists to stop.

How to decide. Decide by who controls the size and how far the damage reaches, not by how slow the code looks. If the input is caller- or attacker-controlled and the cost grows super-linearly in it, the blast radius is the whole process, so the fix is not optional: bound it now, the way you would any tenet IV or VII liability, because a profiler would only report the exponent you can already read off the loop. If the shape is one of the named antipatterns, the recognition is the measurement and you fix it. Everything else is genuinely a tuning question, and there the honest move is to spend nothing until a profiler under realistic load proves the cost is real; clarity is what you are about to trade, and you should trade it only for a number. The trap sits in the middle: in-process work whose cost scales with input you do not bound, say a client-side sort over user data, is attacker-shaped even though it is "yours". When in doubt about who controls the size, measure it at realistic scale before you decide which side of the line it sits on.

Reach for first. Ask one question before touching anything: who picks the size? If a caller or attacker does and the cost grows faster than linearly, bound it now and move on. If not, do not optimise yet. Reach for the profiler under realistic load, and let the measurement, not your intuition, tell you whether there is anything to fix at all.

Pitfalls. - Optimising on a guess: you rewrite the function you suspected, ship the lost clarity, and the profiler later shows the time was somewhere else entirely. - ‘Dev data was fine’: the O(n squared) is instant on a hundred rows and lethal on a million, and the small test set is precisely what hid the cliff until real traffic hit it. - Filing an unbounded blowup as a perf ticket to revisit ‘once we have numbers’, when it is a DoS that ships the moment a caller sends a large input. - Calling caller-controlled in-process cost ‘yours’: a sort or regex over user data is attacker-shaped latency even with no network in sight. - Over-reading the antipattern list: treating every loop near a query as an N+1 and batching things that were never hot, trading legibility for a speed-up nobody needed. - Capping untrusted size with a number plucked from the air, so legitimate large inputs get rejected and the error tells the caller nothing about the real limit.

Batch the round-trip (join / IN / vectorise) vs per-row access (N+1)

You hit this when You need data for many items, and the loop that reads them one at a time is the natural way to write it. The question is whether you fetch in one shot or pay a round-trip per item

The call. You have a collection of items and need related data for each. Do you collapse the fetch into one batched round-trip, or let each iteration of the loop fetch its own?

Batch the round-trip (join / IN / vectorise). Fetch everything in one go: a join or an IN list instead of a query per row, a filter pushed into the engine instead of a Python loop, a vectorised op instead of element-by-element. One coarse call where the loop wanted many fine ones. - Choose when: The number of items is set by the caller or by data volume, so the per-row version is super-linear in a size you do not control and grows without a ceiling; The per-item cost carries fixed overhead that dwarfs the payload: a network or database round-trip, an RPC hop, an interpreter dispatch per element; The work is a recognised antipattern on sight, the N+1 SELECT or the per-row loop over a vectorisable op, so it is a correctness bug you fix without a profiler; Throughput is money here: the fan-out shows up as instance hours, egress or per-request charges, not just latency. - Cost: The batched query is harder to read than the obvious loop; a join or a window function asks more of the next person than a foreach. You spend clarity (I, XXI) to buy the speed; An IN list with thousands of items, or an unchunked join, swaps an N+1 problem for one giant query that strains the engine or blows a parameter limit. Batching is not free of its own unbounded edge; Vectorising, or pushing the filter down, moves logic into a different idiom (SQL, an array library) where the rules and the failure modes differ from the surrounding code; Loading everything up front can fetch rows the caller never touches, trading round-trips for memory and wasted reads.

Per-row access (N+1). Keep the loop as written: one round-trip per item, each iteration lazily loading what it needs. The chatty RPC and the N+1 SELECT are this shape. - Choose when: The collection has a small, known ceiling the caller cannot inflate: a fixed config list, a handful of enum rows, a bounded set you control; Each item is genuinely independent and you want per-item failure isolation, where one bad fetch fails just its own iteration rather than the whole batch; The access is rare and cold, off any hot or caller-driven path, and the legibility of the plain loop is worth more than collapsing a cost nobody pays at scale; The data source has no batch primitive, so a join or IN is not on offer and faking one would cost more than the round-trips. - Cost: It is the most common latency-and-cost disaster in software, and it stays invisible until the data grows: ‘dev data was fine’ is exactly how it ships; The cost is super-linear in a size the caller often controls, so a list that was ten in testing becomes ten thousand in production and the loop becomes 10,000 round-trips per request; Each round-trip pays fixed latency and the bill pays per request, so the waste compounds on both clocks and money even when each individual call looks cheap; It hides from a profiler on the cold path you tested it on, then dominates the trace on the live one, which is the worst time to discover it.

How to decide. Decide by who sets the count and whether it has a ceiling. If the number of items is fixed and you control it, the plain per-row loop is the more legible choice and you keep it; the cost it carries is bounded because the size is. The moment the count is set by the caller or by data volume, the per-row loop stops being a tuning question. It becomes a correctness bug, because its cost is super-linear in a size you do not control, and that is precisely the carve-out tenet XVII lets you fix on sight without a profiler. The N+1 SELECT and the per-row loop over a vectorisable op are recognisable enough to fix on recognition, so batch them. Batching has its own unbounded edge, though: an IN list or a join with no ceiling just relocates the blowup into one enormous query, so cap and chunk the batch so neither shape grows without bound. The tie-break is whether a tired engineer can read the loop and know its cost is bounded. A small count that is yours says that plainly on the page; a count that belongs to the caller stays bounded only behind the batched, chunked fetch, and that is worth the clarity it costs.

Reach for first. If the item count is fixed and yours, keep the plain loop. The instant the count is caller- or data-driven, collapse it: one join or one IN for a database, the filter pushed into the engine or a vectorised op for data work, and chunk the batch so it cannot itself grow without a ceiling.

Pitfalls. - Testing the loop on dev data, watching it pass, and shipping it: the smallness of the test set is the very thing that hides the N+1 until real traffic arrives. - Curing N+1 by loading an IN list of every id at once, then hitting a parameter limit or a query that locks the table. You swapped one unbounded shape for another instead of chunking. - Eager-loading associations everywhere as a blanket fix, fetching mountains of rows no request reads, and trading round-trips for memory and cost. - Assuming a profiler will catch it: the per-row cost lives on the cold path you measured and dominates the hot path you did not. - Vectorising or pushing a filter into SQL and silently changing semantics (null handling, ordering, type coercion) that the row-by-row version got right by accident. - Treating the chatty RPC as merely slow and adding a cache in front, when the fix is to collapse the fan-out into one coarse call.

Long but progressing work: heartbeat + cancellation vs a wall-clock deadline

You hit this when You have a unit of work that is genuinely long: a six-hour batch, a 50 GB sort, a multi-hour query, a training run. It is not stuck, it is working. You want a bound on it so a wedged instance cannot sit there forever eating a worker, but the obvious bound, a wall-clock cutoff, would also kill the runs that are merely slow. The question is what kind of bound this work actually needs

The call. This wait does not cross a boundary you do not control; it is your own long computation. So the deadline rule from tenet VI does not apply unchanged. What stops a wedged run without killing a correct one?

Progress + heartbeat + cancellation. The work emits a progress signal at a known cadence (rows done, batches done, a monotonic counter), a watcher treats a missed beat as the failure rather than treating elapsed time as the failure, and there is a cancellation path that actually tears the work down. You bound advancement, not duration. A run that keeps beating is allowed to take as long as it honestly needs; a run that goes quiet is stopped. - Choose when: The work is legitimately long and you cannot put a true upper bound on its correct duration: input size varies, the query plan varies, the model varies; The work can report that it is advancing without lying, so a stalled heartbeat genuinely means stalled work and not just a quiet phase; You can cancel cooperatively: the work checks a token, releases what it holds, and stops, so stopping it is real rather than abandoning an orphan that keeps running; An operator needs to stop a specific run by hand, mid-flight, without taking the whole job down. - Cost: You now own a protocol, not a number: where the beat is emitted, what counts as a beat, what cadence is healthy, and a watcher that reads it. A heartbeat nobody watches is a silent fallback with a counter for a disguise; Cancellation has to be plumbed through every layer that holds a resource, and a token nobody checks inside the inner loop is decoration; Phased work fools a naive watcher: a long quiet load phase before the first batch reads as a stall, so the cadence has to know the shape of the work; Heartbeat itself can lie. A loop that beats but makes no real progress, spinning on a poisoned row, passes a liveness check while doing nothing useful, so the signal should track work done and not merely that the loop turned over.

A wall-clock deadline. A single duration: if the work is not done in N hours, it is killed. The right tool for a wait across a boundary you do not control, where silence is indistinguishable from death and the only safe assumption is that it has hung. Wrong here, because this work is not waiting on a stranger, it is computing, and the clock cannot tell six correct hours from a hang. - Choose when: The wait genuinely crosses an uncontrolled boundary (a network call, an IPC, a lock you do not own) where you have no progress signal and silence is the only information you get; There is a real, defensible upper bound on correct duration, so the cutoff sits above every legitimate run and only catches the wedged ones; You bound the drain at shutdown (tenet XII): a graceful drain that waits forever is its own hang, so a deadline on the drain then force-exit is correct even here; You can pair it with cancellation (tenet VII) or idempotency (tenet X), so firing the deadline tears the work down or a retry does no harm. - Cost: On long work it murders correct runs whose only fault was needing time: the run at hour five that would have finished at hour six dies for nothing, and the cost lands on the slowest, often most valuable, jobs; It encodes a guess about duration as if it were a fact; the first run on bigger input quietly crosses the line and you debug a 'failure' that was the system working; A bare deadline that does not cancel and propagate just orphans the slow work and may fire a duplicate, so you have paid the cost of killing the run and still left it running; It hides the real question. What you wanted to know was whether the work is still advancing, and a clock cannot answer that.

How to decide. Start from who controls the wait and what silence means. A wall-clock deadline is the right bound when the wait crosses a boundary you do not control and a quiet line is indistinguishable from a dead one: there, the clock is the only signal you have, and an unbounded hang spreads. This fork is the other case. The work is yours, it is computing, and a progress signal is available to you, so the honest bound is on progress rather than on the clock. The tie-break lands here: minimise what a tired engineer must hold in their head, subject to a bounded blast radius for what they cannot control. The blast radius of a hung run is one worker, contained, and the input (a run that lasts longer than expected) is effectively self-controlled, so you do not need the blunt instrument that destroys correct work to contain it. You need the cheaper, truer bound: prove it is advancing, and allow it to be stopped. Reach for the deadline only on the genuine sub-waits inside the job, a network call or the shutdown drain, where silence really does mean death.

Reach for first. Emit a heartbeat tied to real progress (a monotonic count of work done) and have one watcher treat a missed beat, not elapsed time, as the failure. That alone bounds the wedge. Add a cancellation token the inner loop actually checks before you claim the run can be stopped.

Pitfalls. - Putting a wall-clock timeout on the whole job 'just to be safe'. It will fire on the first correct run that runs long, and you will spend an afternoon proving the system was right to be slow. - A heartbeat that beats on loop iteration rather than work completed: it stays green while the loop spins on a poisoned input doing nothing, so the liveness check passes and the work is dead. - A cancellation token that is created and passed but never checked inside the hot loop. Cancellation that does not tear down is a timeout that orphans the work and may fire a duplicate under retry. - No watcher. The work emits a perfect heartbeat into a void; nothing reads it, nothing acts on a missed beat, and the bound exists only on paper. - A cadence blind to the work's phases: a long setup or load phase before the first batch reads as a stall and the watcher kills a healthy run during warm-up. - Forgetting the drain at shutdown. 'No deadline' is right for the compute, but an unbounded graceful drain is a hang of its own; bound the drain, then force-exit.

Resource lifecycle

Scope-bound release (RAII/with/defer) vs manual release

You hit this when You have just acquired something that must be given back: a file handle, a lock, a database connection from a pool, a subscription, a timer, a child subprocess. The acquisition is one line and looks harmless. The release is the line that has to run on every way out of this scope, including the early return, the exception, and the panic you did not write a handler for

The call. Should the release be bound to the scope that acquired it (with, defer, try/finally, RAII), or written by hand at each exit?

Scope-bound release (with / defer / try-finally / RAII). The language ties the release to the lexical scope or object lifetime that owns the resource, so it runs automatically on every exit path: normal return, early return, exception, panic. The acquisition and the release are written next to each other, and the runtime guarantees the pairing. - Choose when: The language gives you a construct for it: Python with, Go defer, Java/JS try-finally, C++/Rust RAII, C# using, Swift defer; The resource is acquired and released within one scope or one clearly owned object, so a single owner is obvious; There is more than one exit path, or any path can throw or panic, which is almost always true once error handling is real; You want the next reader to see that the release is handled without tracing every branch to confirm it. - Cost: The construct is tied to scope or object lifetime, so a resource that must outlive the acquiring frame (handed to a caller, stored in a struct, parked on a queue) does not fit cleanly, and you contort the scope or fall back to manual anyway; defer in particular has footguns: a defer in a loop stacks until the function returns rather than releasing each iteration, and a deferred close whose error you ignore can swallow a failed flush (XII); RAII moves the cost into ownership: you now have to reason about moves, borrows, and who holds the strong reference, which is a real tax in C++ and Rust; Scope binding releases at scope exit, not a moment sooner; if you hold a scarce lock or connection across slow work inside the scope, the construct will faithfully hold it too long.

Manual release by hand. You call the release explicitly at each point the resource is finished with, with no language construct binding it to the scope. Correctness rests on you having written a release on every exit path, error paths included. - Choose when: No scoping construct fits: the resource's lifetime genuinely crosses the acquiring scope, so ownership is transferred to a caller, a long-lived registry, or another thread; You are in an environment with no such construct (older C, some embedded toolchains, a callback-based API that hands you the resource in one function and the close in another); A pool or arena owns the lifetime and individual call sites only borrow, so the release lives at the owner, not at the borrow site; The release is conditional in a way scope cannot express: you release only if you did not hand ownership onward. - Cost: Every exit path you forget is a leak, and the one you forget is the error path nobody exercised in testing, so it stays latent until handles or connections run out in production; It is an orphan in waiting: it looks harmless right up to the release that never comes, and the symptom (exhaustion) surfaces far from the missing line; The reader cannot confirm correctness locally; they must trace every branch, including the ones that throw, to be sure the release fires; Refactoring is hostile to it: a new early return added six months later silently bypasses the release, and nothing flags it.

How to decide. Reach for the scoping construct by default; manual release is the exception you justify, not the baseline. The deciding question is whether the resource's lifetime fits the scope that acquired it. If acquisition and release sit inside one frame or one owned object, scope-bound release is strictly less for a tired engineer to hold in their head: the release is pinned to the acquisition, every exit path is covered for free, and the error and panic paths (the ones that leak in practice) are handled without anyone remembering them. You step down to manual release only when the lifetime genuinely escapes the scope, ownership is handed to a caller, parked in a long-lived registry, or transferred across a thread, so that binding to this scope would either release too early or not at all. At that point the cost has moved: the hard part is no longer whether you released on every branch but who owns the resource now, and the right move is to give it exactly one owner and put the release there (VIII), rather than scatter manual closes across borrow sites. Match the blast radius to the resource. A leaked file handle in a short-lived script is a shrug, but a leaked lock, a leaked pool connection, or a leaked subscription in a long-running service is exhaustion or a wedged teardown that takes the process down. The scarcer the resource and the longer the process lives, the less tolerable a manual path you cannot verify by reading, and the more the scoping construct earns its keep. Remember that none of this is the garbage collector's job: the runtime reclaims memory and nothing else, so every non-memory resource is yours to release regardless of language.

Reach for first. The language's own scoping construct for the resource: with in Python, defer in Go, try-finally in Java/JS, using in C#, RAII in C++/Rust. It is the cheapest correct answer and covers the error and panic paths you would otherwise forget. Drop to manual release only once you can name why the lifetime does not fit the scope.

Pitfalls. - Assuming a garbage-collected language cleans up for you: the collector frees memory and never releases handles, sockets, timers, subscriptions, or locks, so you leak the resource that actually runs out first. - defer inside a loop: the releases stack until the function returns instead of firing each iteration, so a loop that opens N handles holds all N until the end. - Ignoring the error from a deferred or finally-block close, which can silently eat a failed flush and lose data you thought was written (XII). - Releasing in finally but acquiring inside the try, so a failure during acquisition runs a release against something that was never acquired. - Writing manual release where a construct existed, purely because the construct was unfamiliar, then leaking on the one early return added later. - Two owners both releasing, or both holding a strong reference so neither releases, which is the retain-cycle shape of the same mistake.

See also. Tenets (VIII), (XII). glossary: an orphan in waiting, retain cycles, GC reclaims memory only. phases: implementing.

You hit this when A consumer holds a copy of a producer's state (a cache, a replica, a UI, a downstream service) and that copy keeps going stale. You are deciding how the consumer learns about changes: by being told, by asking, or by repeatedly comparing what it has against the truth

The call. When a consumer must track a producer's state, do you push changes to it, have it poll on an interval, or have it reconcile against the truth, and which of those do you trust for correctness?

Subscribe / push. The producer emits an event on every change and the consumer reacts to it: a websocket, a message on a topic, a database trigger, a watch on a key, an in-process observer. The consumer learns of the change with the lowest latency the transport allows. - Choose when: You need the consumer current within milliseconds and a stale read is visible or harmful (a live price, a presence indicator, an invalidation); The transport gives you a delivery guarantee you can actually lean on, or the two sides share a process so no edge can be dropped; Changes are frequent enough that polling would either lag badly or hammer the producer; The event carries the change, so the consumer does not have to turn round and re-read the whole state. - Cost: Events are sharp but lossy: a dropped, reordered or duplicated message leaves the consumer silently wrong, and silence reads identically to nothing having happened; Every subscription is a registration with a teardown (tenet VIII); miss the cleanup and you fire callbacks into a dead consumer, or wedge teardown on the long-lived side; A reconnect after a gap needs a resync story, otherwise the consumer resumes confidently from a state it missed updates for; On its own it gives you no way to detect the failure that emits no event at all: a crash, a hang, a disconnect.

Poll on an interval. The consumer asks the producer for current state every N seconds. Staleness is bounded by the interval, and the machinery is just a loop and a timer with no registration to leak. - Choose when: Bounded staleness is good enough and you can name the bound the product tolerates; The signal you are waiting for is absence: a crash, a hang, a job that never finished, where there is no edge to subscribe to and a timer or heartbeat is the only thing that will ever fire; Change is rare or the producer cannot push, and the poll cost is negligible against the freshness you buy; You want the simplest thing that is correct and have no low-latency requirement. - Cost: Latency floor equal to the interval: tighten it for freshness and you pay in load, loosen it for load and you pay in staleness; At scale, many consumers polling a hot producer is its own load problem, and a thundering herd if their clocks align; If the poll only fetches a delta or a flag it has the same missed-edge weakness as a subscription; the safety comes only when it re-reads actual state; Wasteful in the common case where nothing changed, since most polls return the answer you already had.

Reconcile / level-trigger. A loop reads the current desired state and the current observed state, compares them, and drives observed toward desired, from whatever state it finds. It acts on the level, not on the transition, so a missed event is corrected on the next pass and re-running it is always safe (a level-triggered, convergent reconciler). - Choose when: Correctness across a boundary matters more than latency, and you accept that you can and will miss edges; The state is something you can fully observe and compare: desired vs actual, a set of resources, a target you can converge on; You want one mechanism that heals drift from any cause: dropped events, partial writes, a process that died mid-change, manual meddling; You are already building a control loop (a supervisor, an operator, a sync engine) and want it to be self-correcting by construction. - Cost: Latency is the sweep period: on its own a reconciler is as slow to notice a change as a poll on the same interval; The compare-and-converge step must be idempotent and safe to re-run from any state, which is real design work rather than a free property; A full reconcile over a large state can be expensive; you end up needing watermarks or incremental scans to keep the cost down; It tells you the system is wrong and fixes it, but it does not tell you fast: a user watching a live view will see the lag.

Subscribe for latency AND reconcile for correctness. Run both. Subscribe to events so the common case is fast, and keep a level-triggered convergent reconciler behind it that periodically re-reads actual state and heals whatever the events missed. The pairing buys the speed of events and the correctness of polling without forcing a choice between them. - Choose when: You need low latency in the common case and you cannot tolerate being silently wrong across a boundary where you might miss an edge; The failure mode includes both real change events and the absence of one (a crash, a silent disconnect): the subscription catches the first, the sweep catches the second; The state is observable enough to reconcile and changes often enough to justify pushing; This is most distributed sync worth getting right: replicas, caches, operators, anything tracking remote state over a lossy boundary. - Cost: Two mechanisms to build, test and operate, with the reconciler exercised rarely enough that its bugs hide until the day an edge is actually dropped; The fast path and the slow path must agree on the same truth, or the sweep keeps undoing what the event just did; More moving parts and more to hold in your head than either alone, justified only where a missed edge really would hurt; The reconcile interval still bounds your worst-case correctness lag; tuning it trades sweep cost against how long a missed edge can persist.

How to decide. Decide by where the truth lives and what its silence looks like. The governing question across the corpus is how much a tired engineer must hold in their head to stay correct, bounded by the blast radius of anything you do not control, and here the thing you do not control is the transport and the producer on the far side of a boundary. Inside one process, where no edge can vanish, a plain subscription with honest teardown is correct and you should not pay for more. The moment the change crosses a boundary you cannot make lossless, an event becomes a hint rather than a fact, because a dropped or reordered message is indistinguishable from nothing happening, so a pure event design is right only when the event is guaranteed to land somewhere you cannot miss it. Size the cure to the cost of being wrong: if a missed edge is cheap and self-correcting, a poll or a bare subscription is enough; if a missed edge wedges you forever (the seat sold twice, the replica permanently diverged), the property is real and the reconciler earns its keep. Latency and correctness are separate axes, so answer them separately. Use the event for the speed you need in the common case, put a level-triggered convergent reconciler behind it for the edge you will eventually miss, poll where the truth you are watching for is silence (since absence emits no event to subscribe to), and subscribe everywhere else.

Reach for first. Poll on an interval. If a bounded staleness the product can live with is acceptable, a timer that re-reads actual state is the simplest thing that is correct: no registration to leak, and no missed-edge weakness as long as it reads the real state rather than a delta. Reach for it first, then add a subscription only once you measure that the interval's latency is genuinely too slow, and add a reconciler only when a missed edge across a boundary would do lasting harm.

Pitfalls. - Trusting a subscription as the source of truth across a lossy boundary, so a single dropped or reordered message leaves the consumer confidently and silently wrong with nothing to heal it. - Registering a listener with no teardown, so it outlives the consumer and fires into a dead thing, or wedges teardown on the long-lived side (tenet VIII). - Polling for a delta or a changed flag instead of re-reading actual state, which inherits the exact missed-edge bug you switched away from events to avoid. - Writing a reconciler whose converge step is not idempotent, so re-running it from a partial state double-applies or thrashes instead of healing. - Reconnecting after a gap and resuming from the last seen state with no resync, treating the silence during the gap as if nothing changed. - Tuning the poll or sweep interval down for freshness until many consumers stampede a hot producer, especially when their timers align. - Pairing a fast event path and a slow sweep that disagree on the truth, so the reconciler keeps undoing what the event just did. - Using events for the case whose failure mode is absence (a crash, a hang), where no edge will ever fire and only a timer or heartbeat can notice.

Graceful drain vs hard kill on shutdown

You hit this when A SIGTERM lands (a rolling deploy, an autoscaler scaling in, an operator restart) while the process is still holding in-flight work: requests mid-handshake, messages consumed but not acked, buffered writes not flushed. What the process does in the next few seconds decides whether that work completes or vanishes

The call. When the process is told to stop while holding in-flight work, do you drain that work within a bounded window before exiting, kill immediately, or wait for the drain however long it takes?

Graceful drain with a bounded grace window. On the stop signal, close the listener so no new work arrives, let in-flight work finish (or hand it off) within a fixed deadline, flush buffers, ack or commit only what is durably written, then exit. If the deadline expires, force-exit on whatever is left. - Choose when: A unit of work is short relative to a few seconds of grace, so most of it genuinely completes inside the window; Dropping in-flight work has a real downstream cost: a 500 to a user, a half-processed payment, a stream offset committed for a record you never wrote; Restarts are routine and frequent (rolling deploys, autoscaling), so the per-shutdown loss compounds into a steady drip all day; You control the worker and can wire a signal handler, a listener stop, and a flush in the right order. - Cost: Real machinery on the exit path, which is the path that gets the least testing and is where the order of operations (stop listener, then drain, then flush, then ack) silently matters most; The grace window is a guess: too short and you still drop the long tail, too long and every deploy crawls and the orchestrator's own kill timeout may pre-empt you anyway; It only protects you against an orderly stop. A SIGKILL, an OOM, or a yanked power cord skips the drain entirely, so you still need a durability contract underneath.

Hard kill (exit immediately, drop in-flight work). Treat the stop signal as exit-now. In-flight requests are dropped, unacked messages are abandoned, buffers are discarded. Recovery, if any, is the caller's retry plus your own durable log. - Choose when: Every unit of work is already idempotent and durably recoverable: a WAL, at-least-once redelivery, and a caller that retries mean a dropped item simply replays; The work is genuinely fire-and-forget or read-only, with no downstream party counting on completion; Startup is so fast and crash-recovery so well-exercised that crash-only restart is the cheapest design to reason about; You want a single, trivial shutdown path and have paid for the durability that makes dropping safe. - Cost: A rolling deploy sheds whatever was in flight on every pod it cycles, so the cost is not one outage but a constant background loss that hides in tail latency and retry counts; Without an airtight durability contract this is where you mistake "process gone" for "work done": the absence of the process reads as completion when it was abandonment; If anything acks before it persists, the crash window between ack and write becomes silent data loss with no error to trace, surfacing only as a discrepancy on restart; Pushes the recovery burden onto callers and downstream systems, which may not retry, may retry without idempotency, or may have already moved on.

Unbounded drain (wait for in-flight work however long it takes). On the stop signal, stop accepting new work and then block until every in-flight item finishes, with no deadline. - Choose when: Almost never as a deliberate choice; it is usually graceful drain with the deadline left off; Arguably defensible only when every unit of work has its own hard internal timeout, so "however long it takes" is in fact bounded by those, and losing any item is unacceptable. - Cost: One wedged request (a hung dependency, a lock it will never get) holds the whole shutdown open forever, so a stuck item becomes your outage: the deploy never finishes, the node never drains; The orchestrator solves it for you in the worst way: it waits out its own grace period and then SIGKILLs you mid-flush, which is hard kill with extra latency and a corrupted-looking exit; An unbounded wait on shutdown is the same failure as an unbounded wait anywhere across a boundary you do not control: it does not fail, it hangs, and the hang spreads to whatever is waiting on the deploy.

How to decide. Decide by who pays when in-flight work disappears and how wide that loss spreads, not by which exit path is tidier to write. The signal is yours, so the shutdown is controlled input; the real question is the blast radius of dropping the work that is open when it arrives. Where a dropped item is a 500 to a caller, a committed offset for an unwritten record, or a payment left half-applied, the loss is real and recurs on every restart, so it is worth the machinery: drain within a bounded window, and order it so you ack only what is durable. Where every item is already idempotent and durably logged, draining buys you almost nothing over a clean kill, and the extra exit-path code is coupling you pay for a property you already have elsewhere; reach for the kill. Whichever you pick, the drain must be bounded, because the one thing you cannot afford is to convert an orderly stop into an unbounded hang: a deadline that force-exits is what keeps a single wedged request from turning your deploy into an outage. And because a SIGKILL or an OOM ignores the drain entirely, the grace window is an optimisation on top of a durability contract (a WAL, at-least-once redelivery, persist-then-ack), never a substitute for one. Spend the drain where completion is owed and the loss is visible; lean on durability everywhere, since that is what makes an abrupt exit survivable when the graceful path never runs.

Reach for first. First make an abrupt exit survivable, then the drain is an optimisation rather than a load-bearing necessity. Get the ordering right (persist, then ack or commit the offset) and put the durability contract underneath (a WAL, at-least-once redelivery, idempotent writes) so a dropped item simply replays. With that floor in place, add the cheapest correct drain your platform already gives you: trap the stop signal, stop the listener, let in-flight work finish inside the orchestrator's existing grace period, flush, exit. Only reach for a hand-tuned window or hand-off protocol once you can show the default window is dropping work that matters.

Pitfalls. - Acking or committing the stream offset before the record is durably written, so the gap between ack and persist eats data silently on restart and surfaces only as an unexplained discrepancy. - Leaving the drain unbounded, so one wedged request holds shutdown open until the orchestrator SIGKILLs you mid-flush, giving you the worst of both paths. - Reading the process being gone as the work being done: a missing process is not durability, and a dropped-request deploy is exactly this conflation. - Setting a grace window longer than the orchestrator's own kill timeout, so your careful drain is pre-empted by a hard kill you did not plan for. - Treating graceful drain as the durability story and shipping no WAL or redelivery, so the first SIGKILL, OOM, or power loss loses everything the drain was meant to protect. - Draining requests but forgetting the other obligations on the way out: buffered analytics, queued writes, open transactions, temp files left for the next run to mistake as complete.

See also. Tenets (XII), (VI), (X). glossary: mistook "process gone" for "work done", ack-before-persist silently eats data on restart, a WAL. phases: implementing, operating.

Reference strength on a long-lived watcher: weak vs strong-with-teardown

You hit this when A long-lived object (a cache, an observer registry, a parent screen or supervisor) needs to hold a reference to something shorter-lived (a subscriber, a child, a view). You are in a manual-memory or ARC language (Rust, C++, Swift), so the strength of that reference is yours to choose, and the choice sets the failure mode. Hold it one way and teardown wedges; hold it the other and the thing you are watching vanishes while a user is still on it

The call. Should the long-lived side hold the short-lived thing weakly and let it be collected when its owner drops it, or hold it strongly and remove that reference by hand in an explicit teardown on unsubscribe or unmount?

Weak reference from the long-lived side. The long-lived holder points at the target without keeping it alive. When the target's real owner releases it, the count falls to zero and it is collected; the watcher's reference goes nil or dangles to a tombstone, and a guard on each use skips the dead one. - Choose when: The watcher genuinely outlives the watched and is expected to: a cache or registry that will see thousands of short-lived entries come and go; The target has a clear owner elsewhere that controls its lifetime, and the long-lived side is only an observer, not a keeper; You would otherwise close a retain cycle, parent strong to child and child strong back to parent, where neither arm has a natural teardown; Losing the target silently is acceptable or even correct: a stale cache entry that should just be recomputed. - Cost: The target can be collected or evicted out from under code that is still using it: the listener that quietly stops firing, the cache entry that disappears mid-use. The failure is silence, not a crash; Every access has to handle the nil or upgrade-fails case. Forget one guard and you get a dangling read or a use-after-free, depending on the language; Lifetime now depends on an owner you do not control. A change to who holds the strong reference elsewhere can shorten the target's life without any edit to the watcher; Debugging 'it just stopped' is harder than debugging a leak, because nothing logs and nothing aborts.

Strong reference with an explicit teardown. The long-lived holder keeps the target alive and owns the obligation to drop that reference on a known event: unsubscribe, unmount, close, dispose. The reference is real for exactly as long as the registration is, and teardown removes it. - Choose when: The watcher must not lose the target while the registration is live: an in-flight operation whose callback must survive until it completes; There is a single, reliable teardown event you can hang the release on, and you will actually wire it (VIII); The short-lived side is expected to die first and the long-lived side is the keeper of record for the duration; You want the lifetime stated in the code at the registration site, not inferred from who else happens to hold a strong reference. - Cost: The teardown is now an obligation that runs on every exit path, the error and early-return paths included, or the target is pinned forever: a leak, and often a retain cycle where neither object can be freed; A pinned screen or subscriber keeps firing callbacks and holding its transitive graph long after the user left, which reads as a slow memory climb rather than a crash; The correctness of the whole thing rests on a teardown that is easy to skip and invisible in review, the same way a missing unsubscribe is; More machinery at the call site: register here, remember to release there, and keep the two in sync as the code changes.

How to decide. Decide by which side is expected to die first, and by what it costs when the watcher is wrong about the target still being there. The long-lived side does not control when the short-lived thing goes away; its owner does, or the user does. So ask what the watcher holds at the moment the target disappears. If the watcher can carry on without it, a stale cache entry to recompute or a gone subscriber to forget, hold weak and guard each use: the blast radius of a vanished target is bounded to one skipped or recomputed access, and you have spent nothing on teardown. If losing the target mid-flight corrupts something or drops work someone downstream is counting on, the target must stay alive for the duration, so hold strong and pay for the explicit teardown that releases it on the close event. The tie-break is the obligation, not the pointer: a strong reference is only correct if its teardown runs on every exit path, which is the same discipline as tearing down what you set up (VIII) and finishing your obligations before you exit (XII). If you cannot point to the single event that drops the strong reference, you do not have strong-with-teardown, you have a leak with extra steps, and weak is the safer default. Match the strength to whichever side dies first; spend the teardown machinery only where a vanished target costs more than the coupling the cleanup drags in.

Reach for first. Hold weak on the long-lived side and guard each use. It is the cheapest correct move when the watcher outlives the watched, which is the common case for caches, registries, and observers, and it cannot leak or wedge teardown. Reach for strong-with-teardown only once you can name the single event that releases the reference and the cost of the target vanishing mid-use.

Pitfalls. - Parent strong to child and child strong to parent: a retain cycle where neither is ever freed. Break one arm, usually the back-reference, by making it weak. - Choosing strong because the reference feels important. Strength controls lifetime, not priority; an important observer of a short-lived thing still wants to be weak. - Wiring the teardown on the happy path only. An early return or thrown error skips it and the target is pinned, so the release has to live in the language's scoping construct (defer, RAII, finally), not in a line you hope runs. - Holding weak across a boundary you cannot guard on every access, so a collected target becomes a dangling read instead of a clean nil. - Assuming a GC will sort it out. In ARC and manual-memory the cycle is yours; the collector frees memory and frees nothing about who keeps whom alive. - Treating the weak listener that 'sometimes stops working' as a flake. It is the target being collected under you, and it will not log anything.

See also. Tenets (VIII), (XII). glossary: reference strength, retain cycles, an orphan in waiting. phases: implementing.

Atomic temp-write-then-rename vs write in place

You hit this when You produce an output file, or any single artefact, that a reader or the next step will consume. The question is what a crash mid-write is allowed to leave behind for whoever opens the path next

The call. When a crash, a kill, or a full disk interrupts you halfway through writing the file, what does the next reader see: a complete result, nothing at all, or a fragment wearing the final name?

Temp-write-then-rename. Write the whole output to a temporary path beside the destination, then do one atomic rename to the final name once the write has fully gone through. A signal trap removes the temp file on an abrupt exit. The destination name appears only after every byte is there, so a reader gets either the complete result or nothing, never the third lying state. - Choose when: The output is read by another process, another step, or a future run, and a fragment that looks finished would be acted on as real; A crash, OOM kill, SIGTERM, or full disk part-way through the write is plausible, which on a long enough timescale it always is; Temp and destination sit on one filesystem, so the rename is a true atomic metadata operation rather than a copy; You want lock-free single ownership too: claim by atomic rename lets exactly one contender win the same file. - Cost: Two paths and a rename to keep straight, plus a signal trap and a startup sweep for orphan temp files a crash left lying around; The atomicity holds only within one filesystem; a rename across a mount boundary degrades to copy-then-delete and quietly loses the guarantee; fsync ordering still matters: rename before the data is durable can survive a power cut as an empty file, so the careful version flushes first; Slightly more machinery in a throwaway script than just opening the destination and writing.

Write in place. Open the destination directly and write the bytes as you go. The file carries its final name from the first byte. Fewer moving parts, and the one obvious path on disk is the one being written. - Choose when: Nothing else reads the file until your process has cleanly returned, and a crash means the whole run is retried from scratch anyway; The artefact is genuinely disposable: a log you tail, a scratch file, a cache entry that is revalidated before use; The write is a single small atomic-enough operation, or the consumer already tolerates partial input and checks for completeness itself. - Cost: A crash mid-write leaves a half-written file under the real name, and a half-written CSV never looks off, so the next step reads the truncated data as finished and the corruption spreads downstream unnoticed; The blast radius is unbounded once a downstream reader is in the picture: you find out from a data discrepancy nobody can explain, long after the crash; Concurrent producers race on the same path with no arbiter, so two overlapping runs can interleave bytes into one corrupt file; Recovery is manual: someone has to notice, work out which file is the fragment, and rerun.

How to decide. Decide by who reads the file and what a fragment costs them. If the only reader is your own process after a clean return, write in place: a crash throws the run away and you start over, so there is no fragment to mistake. The moment any other step, run, or process consumes the artefact, the controlling fact is that a partial file wearing the final name is indistinguishable from a finished one, and the reader has no way to tell. That is an input the reader does not control and cannot validate, and the blast radius is every step downstream of it, so the cure is worth its coupling. Temp-write-then-rename caps what the reader can ever see at two states, complete or absent, by spending one rename and a signal trap. The rename is only atomic within a single filesystem, so the rule is: same-filesystem temp path, fsync before rename if a power cut is in scope, and a startup sweep for orphans. That is the floor for anything that emits a file, down to a one-shot script; reach past it only when you can prove no one reads the fragment.

Reach for first. Temp-write-then-rename. It is the cheapest correct move and the floor for any file an outside reader will touch: write to a sibling temp path, fsync, rename, and trap signals to unlink the temp file on exit. Drop to write-in-place only once you can show nothing reads the artefact before your process cleanly returns.

Pitfalls. - Renaming before the data is durable: rename returns, then a power cut leaves an empty or short file under the final name. fsync the file before the rename when crash-consistency, not just crash-atomicity, is the property you need. - Temp file on a different filesystem from the destination, so the rename silently degrades to copy-then-delete and the window for a half-written destination reopens. - No cleanup path for orphan temp files, so a crashed run leaves litter that accumulates until the disk fills. Trap signals to unlink, and sweep stragglers on startup. - Trusting that the consumer checks for completeness when it does not; most readers open the path they expect and trust it. - Writing the temp file with a predictable name in a shared directory, letting a hostile or unlucky second writer clobber or pre-create it. Use an unguessable name or O_EXCL. - Assuming the rename gives you durability of the directory entry too; on some filesystems you must fsync the containing directory for the new name to survive a crash.

See also. Tenets (XII). glossary: temp-write-then-rename, a half-written CSV, claim by atomic rename. phases: implementing.

Observability & verification

Emit the proving signal vs log everything

You hit this when A partial failure has just bitten you and there was no stack trace to read: a latency cliff, an error rate creeping up, a fallback that served wrong-but-non-erroring results for a week before anyone noticed. You want to see what the running system is actually doing, and you are deciding what to emit so the next failure shows up before a user finds it

The call. When you instrument a component, do you emit a small set of signals that prove a named invariant held or expose a known failure mode, do you log everything and search it later, or do you instrument only the happy path?

Emit the proving signal. For each invariant you claim and each failure mode you fear, emit one signal that confirms or denies it: a trace id through every hop, rejection and retry counters, breaker state, source-versus-cache freshness, deadline budget remaining, records-in against records-out. The telemetry is tied to a specific property, and every signal has a decision that reads it. - Choose when: You can name the invariants the component is meant to hold, so you know which signals would prove or break each one; The failures that actually hurt are statistical and silent (rising rates, creeping queue depth, a quiet fallback) rather than clean crashes; The thing being watched sits on a request path, a queue, or a degraded mode where a wrong-but-quiet outcome is plausible; You want each signal gated on a real decision: an alert, a rollback, a capacity call. - Cost: Up-front thought: you have to articulate the invariant before you can instrument it, and a property you never named stays dark; A failure mode you did not foresee is one you have no signal for, so the first occurrence of a genuinely novel mode is still debugged half-blind; The set drifts from reality unless someone prunes it; a counter for a rule you have since deleted is noise wearing a useful name; Tying a signal to an invariant is more design effort per metric than emitting whatever the framework hands you for free.

Log everything. Emit verbose logs for every code path and store them, on the theory that if it all lands somewhere you can grep your way to any answer after the fact. Verification is deferred to query time rather than designed in. - Choose when: You genuinely cannot predict the question, for example a brand-new system in early exploration where you do not yet know its failure modes; Volume is low enough that retention and search stay cheap and the signal-to-noise ratio stays workable; A short, deliberately bounded window (a debugging session, a canary) where you turn it up, read it, and turn it back down; Regulatory or forensic capture is the actual requirement, and completeness rather than legibility is the point. - Cost: The one signal you need at 3am is buried under millions you do not, so the cure for blindness becomes a different blindness; It is itself an unbounded resource (tenet VII): real money to emit, ship and store, and a cost that grows with traffic exactly when you can least afford it; Every verbose path is a chance to write a secret, a token or PII into a log that is now widely readable and long-lived; High volume drives sampling, and naive sampling drops the rare event, which is precisely the one that mattered; Search-time verification means nobody is alerted, so the wrong outcome sits in the logs unread until a customer reports it.

Instrument the happy path only. Add the metrics and logs the success path naturally produces, things like requests served, rows written and p50 latency, and leave the error, retry, fallback and rejection paths uninstrumented because they are the rarely-taken branches. - Choose when: Almost never as a deliberate choice; it is what you get by default when you instrument what is easy rather than what fails; A throwaway prototype with no users, where being blind to failure costs nothing because nothing depends on it. - Cost: It goes quiet exactly when things break: the fallback, the breaker trip and the dropped retry are the unwatched branches, so the dashboard stays green through the incident; A silent degraded mode can serve wrong results for days with every success metric looking healthy, the fallback nobody noticed; p50 and throughput hide the tail, and the tail is where the partial failure lives; It feels observable, which is worse than feeling blind, because it buys false confidence in a review and then collapses in production.

How to decide. Start from what a tired engineer must be able to see to make the next correct decision, and let the controller of the input set the scope. For the inside of a component, where you own the invariants and you know which ones being false would hurt, the work is to name each one and emit the single signal that proves it: that is the proving-signal option, and the discipline of one-decision-per-signal is what keeps the dashboard legible at 3am. Volume is the tell that you have stopped reasoning and started hoarding. Telemetry is an unbounded resource like any other (tenet VII), and on anything an attacker or an unlucky caller can drive, log-everything turns their traffic into your bill and their payloads into your secret-leakage surface, so the blast radius of the cure can exceed the failure it was meant to catch. Reach for log-everything only inside a bounded window you open, read and close, or where the volume is genuinely small and the question genuinely unknown. Happy-path-only is rarely chosen and almost always inherited: the rule is to instrument the branch you are afraid of (the reject, the retry, the fallback, the breaker) before the branch you expect, because the unwatched branch is the one that fails in silence. And the signals can be true component by component while the whole stays dark: green parts are not a green system (law V), so the path a user walks has to be probed end to end as its own invariant, not assembled from per-node health.

Reach for first. Before any dashboard, write down the one question you will ask when this breaks at 3am ("is the fallback serving?", "is the retry budget draining?") and emit exactly the signal that answers it, gated on a decision. One proving signal you will actually read beats a terabyte of logs you never will.

Pitfalls. - Instrument theatre: dashboards and metrics nobody ever reads or gates a decision on, built to look diligent. Unread signal is not free; it clutters the views you do read and dilutes the ones that matter. - Logging the secret: turning verbosity up sweeps tokens, keys or PII into a widely-readable, long-retained store, and the leak outlives the debugging session. - Counting success without counting its absence: tracking requests served but not requests rejected or retried, so saturation is invisible until it is an outage. - Averages over distributions: a p50 or a mean hides the tail where the partial failure lives, so emit quantiles and rates, not just totals. - Naive sampling that drops the rare event: sampling to bound cost is right, but sample so the one anomaly you care about survives, not just the bulk of the boring traffic. - Adding up green health checks: per-component liveness can all pass while the end-to-end journey fails (gray failure), so the whole path needs its own probe.

See also. Tenets (XVIII), (VII), (XIX). the companion's fifth law. glossary: make it observable, not log everything, instrument theatre you never read. phases: implementing, operating.

Per-component health check vs end-to-end probe

You hit this when Your dashboards are green: every service answers its liveness check, every node reports up. Yet users are timing out, or getting answers that are quietly wrong, and nothing in your monitoring tells you where. Each part is an honest witness to the wrong question

The call. Do you verify health by polling each component on its own, by walking the whole route a user walks against a deadline, or by running both with tracing to localise what the probe catches?

Per-component health checks. Each component exposes a liveness or readiness endpoint, and a monitor polls it: is the process up, does it answer, is its own dependency reachable. Health is judged node by node. - Choose when: The orchestrator needs them: Kubernetes, a load balancer or a service mesh wants a liveness and readiness signal to restart pods and gate traffic; You need to localise a failure once you already know the system is broken, so per-node green and red tells you where to look; The failure modes you care about are the ones a node can see in itself: crashed process, full disk, lost database connection; They are the cheap, standard substrate and you want them everywhere as a baseline. - Cost: Structurally blind to gray failure: every check is green while the route across them is red, because the degradation is differential, bad enough to break the journey but too mild to trip any one check; A node that drops one packet in twenty passes its own liveness probe and is a slow poison to everything routed through it; You cannot add green signals up into a true picture of the whole; correctness that lives in the composition of behaviours is owned by no single node; Tail latency that is harmless per hop and fatal once eight hops stack is invisible to every local check.

Whole-path probe (end-to-end). A synthetic check that exercises the entire path a real request travels, edge to edge, against a deadline, instead of asking each component about itself. It answers "does the system do its job", traced so you can see where the budget went. - Choose when: The failure you fear is emergent: stacked tail latency, a wrong-but-successful pipeline, a route that fails while every part reports fine; You have a route whose end-to-end success is the thing a user actually depends on; You want one signal that matches the scale of the failure rather than a wall of green that lies; You can attach a trace id so a red probe points at the offending hop, not just at the whole. - Cost: The probe is itself load and itself risk: synthetic traffic is real traffic the system has to carry, and it needs bounding like anything a caller can grow without limit (VII); It tells you the route is broken, not always which node broke it, unless you pair it with tracing or a per-component signal; A probe nobody reads is just a silent fallback with a dashboard for a disguise; it earns its keep only if it is watched and acted on; It exercises one path; routes you never probe can rot unwitnessed, and writing realistic synthetic journeys is ongoing work; Choosing the deadline is a judgement call: too tight and it cries wolf, too loose and it misses the cliff it was meant to catch.

Both: liveness per node plus an end-to-end probe and tracing. Per-component checks for the orchestrator and for localisation, an end-to-end probe against a deadline to catch the composition-level failure, and a trace id running through every hop so a red probe resolves to a guilty span. - Choose when: The system is distributed and tightly coupled enough that gray failure is a real risk, not a theoretical one; The route matters enough to justify the probe budget and the tracing overhead; You need both the question "is each part alive" for restarts and the question "does the system do its job" for outages; You will actually run game days and act on what the probe and the trace tell you. - Cost: The most machinery to build and operate: probe scheduling, trace propagation through every service, alert tuning on two kinds of signal; Telemetry is an unbounded resource with a cost and a leak risk (XVIII); sampling, bounding and never logging the secret are now your problem; Two signals can disagree, and you must decide which governs an alert, or you trade green-everywhere blindness for noise; Tracing only helps if it is plumbed through every hop; one un-instrumented service is a blind segment on the very path you are trying to read.

How to decide. Decide by where the failure actually lives, because that sets the scale your verification has to match. A failure a node can see in itself (a crash, a full disk, a dead connection) is caught by a per-component check, and that is all you need. A failure that lives in the composition of behaviours (stacked tail latency, a route that times out while every hop reports fine, a pipeline that succeeds over a wrong answer because correctness was the stages agreeing and no stage owned the agreement) is invisible to every local check, by construction, and only a whole-path probe against a deadline can witness it. So the tie-break runs through blast radius and ownership: you spend the probe and the tracing only on routes whose end-to-end failure would actually hurt, and you keep liveness checks as the cheap baseline everywhere, since they are the orchestrator's input anyway. The trap the governing rule warns against is the inverse spend. A wall of green per-component dashboards is the maximum a tired engineer has to hold in their head and the minimum that tells them the truth, because it answers the wrong question honestly. An end-to-end probe holds the shape of the whole somewhere they can go and read it at 2am instead of reconstructing it from a sum of signals that cannot be summed. Buy that probe wherever gray failure is real, bound it like any other load, and make sure someone reads it; do not buy it for a route nobody depends on, and do not let liveness-green stand in for path-green on a route that matters.

Reach for first. One end-to-end probe on the single route whose failure would actually hurt: a synthetic request that walks the path edge to edge against a deadline, watched and alerted on like any other signal. It costs little, it catches the failure mode per-component checks are structurally blind to, and you can add per-node liveness underneath it later when you need to localise.

Pitfalls. - Reading green per-component dashboards as a green system; the sum of honest local signals cannot witness a failure that lives between the components. - An end-to-end probe nobody watches or alerts on: a silent fallback wearing a dashboard, giving false comfort while the route rots. - Running the probe with no deadline, so it cannot catch the stacked-tail-latency cliff it was built for; the budget (VI) is the whole point. - Treating synthetic probe traffic and game days as free; they are real load and a controlled outage, and an unbounded probe schedule is its own DoS (VII). - Probing the path but not propagating a trace id, so a red probe tells you the system is broken and nothing about where. - Probing only the happy path, so the degraded and error routes (the ones that actually fire in an incident) are never witnessed.

See also. Tenets (XVIII), (VI), (VII). the companion's fifth law. glossary: gray failure, end-to-end probe, differential observability. phases: integrating, operating, verify.

Prove it by fault injection/game day vs trust it works

You hit this when You have a fallback, a failover, a degraded mode, or a backup restore. The code exists, it passed review, and the happy path has run clean for months. The path that catches the failure has never once been made to catch a real one. You are deciding whether to provoke it on purpose or assume it works because it is written down

The call. Do you provoke the failure (inject the fault, run the game day, exercise the degraded path in daylight) or trust that the path works because the code is there?

Provoke it: inject the fault, run the game day. Deliberately put the failure into a real or production-like system: kill the node, add the latency, drop the packets, sever the dependency, then watch the fallback do its job with someone present. A game day is the rehearsed, whole-system version, run on a route whose failure would actually hurt. - Choose when: The path catches a failure that will genuinely happen and whose blast radius is wide: a datastore failover, a region evacuation, a cache stampede, a restore from backup; The failure is emergent and crosses components, so no unit test or green health check can see it; only sending a real fault end to end will; The cost of the path failing silently in the incident exceeds the cost of a bounded, scheduled outage you control; There is an owner who will act on what the drill exposes, so the finding becomes a fix and not a filed ticket. - Cost: The drill is itself load and risk: a game day is a controlled outage, and synthetic traffic is real traffic the system has to carry, so you must bound and pay for it (VII); It costs scheduling, blast-radius limits, a rollback plan, and people in the room, all time spent away from shipping; Run carelessly, it manufactures the very incident it was meant to prevent, so the staging fidelity and the abort switch have to be real; A drill nobody reads is theatre: an end-to-end check with no one acting on it is just a silent fallback wearing a dashboard for a disguise.

Trust it works because the code exists. Take the fallback on faith. It compiled, it was reviewed, the happy path is healthy, so you assume the degraded path will engage correctly the day the primary fails. No fault is ever injected and the branch is never forced. - Choose when: The trigger is genuinely rare and entirely under your control, and the blast radius when the path misfires is one widget or one request, not the system; The fallback is trivial and its branch is already covered by an ordinary test that forces it, so 'untested' is not actually true; The failure it guards against is local and visible: it would surface in normal monitoring long before it could compound; The cost of a drill clearly outweighs the cost of the path silently failing, and you have written that judgement down with an owner. - Cost: An untested degraded path is just a second bug waiting for the worst moment to fire, and that moment is when the primary has already failed; The fallback feels like insurance but is unverified code; the confidence it buys is false, and you only discover that mid-incident; The failures that hide best are exactly the ones this misses: each component reports green while the route across them is broken; You have pushed correctness back into human vigilance (the hope that it works) instead of into structure that re-runs the proof for you.

How to decide. Decide by who controls the failure and how far it spreads when the untested path turns out to be broken. What you are really verifying is not the code but the assumption that a tired engineer can lean on the fallback at 3am without holding its untested behaviour in their head, so the question is how much it costs you when that assumption is wrong. Where the trigger is rare and you control it, and the blast radius is one widget or one request, an untested path is a cheap bet: a forced-branch test or a staged injection is plenty, and a full game day is ceremony you will resent. Where an attacker or an unlucky caller can pull the trigger, or where the path is the system's last line (the failover, the restore, the region evacuation), the blast radius is the whole service, and the failure is exactly the kind that hides from green dashboards because it lives in the composition rather than in any one node. There you spend the coordination of a real drill, because the property is real and a fallback that fails in the incident costs more than the controlled outage and the coupling the drill drags in. Match the fidelity of the proof to the width of the blast radius, run only the drills you will actually act on, and bound the drill itself like any other load a caller can grow.

Reach for first. The cheapest correct move is rarely a full game day. Start by asking whether the path has ever fired at all, then shrink the proof to fit: a single injected fault scoped to one component in staging, a unit or integration test that forces the fallback branch, a manual failover in a pre-prod replica. Prove the narrowest version first, and reserve the whole-system drill in production for paths whose failure crosses components and whose blast radius is genuinely wide.

Pitfalls. - Counting a green dashboard as proof. Every node can be an honest witness to the wrong question while the path through them is dead. - Confusing 'the code exists' with 'the path is tested'. A forced-branch test or a single injection is cheap; skipping even that and calling the path verified is the trap. - Running the game day in an environment so unlike production that the fault it injects is not the fault that will actually occur. - Running drills with no abort switch and no blast-radius bound, so the rehearsal becomes the outage. - Injecting faults nobody will act on: a finding with no owner is a discovered bug you have chosen to leave armed. - Rehearsing the failover once at launch and never again, so the path rots as dependencies, configs and data shapes drift out from under it.

See also. Tenets (XIX), (XXIV). the companion's fifth law. glossary: fault injection, an untested degraded path. phases: verify, operating.

Test the contract vs the internals

You hit this when You are about to write tests for a unit, and the obvious move is to test what the code currently does: reach into its private helpers, assert on intermediate state, and chase the coverage number line by line. The unit also has a public promise to its callers, and that promise and the current implementation are not the same target

The call. When you test a unit, do you pin the observable contract at its boundary and decision points, or do you test the private implementation and drive line coverage through the internals?

Pin the contract at the boundary (and property-test the parser). Assert only on what the unit promises its callers: the result it returns, the failure modes it signals, the effects it commits, all observed through its public surface. At a boundary where a caller or attacker controls the input, push past hand-picked examples to property and fuzz tests over the parser, so the rule is checked across inputs you would never think to enumerate. - Choose when: The unit has callers who depend on its behaviour, and you want refactors behind that surface to stay cheap (XX); a contract test only fails when the promise actually changes; The input crosses a trust boundary (IV), so the parser is your most security-critical code and example-based tests will miss the adversarial corners that fuzzing finds; The invariant cannot be carried by a type (the dynamic-language floor for III), so an executable test is the only structural enforcer left; You want the test to keep enforcing the rule at call sites that do not exist yet. - Cost: A pure contract test can pass while an internal branch is silently dead or wrong, because it only sees the surface; you trade fine-grained fault localisation for refactor-survivability; Defining the real boundary takes design work up front, and if you pin the wrong promise a confidently green suite is worse than none; Property tests need you to state the property and a generator, which is genuinely harder than asserting f(2) == 4, and shrinking a failing case to a minimal repro takes time; Effects and async make the observable surface awkward to reach without test seams, and building those seams is real work.

Test private internals and chase line coverage. Assert on internal structure: call private helpers directly, check intermediate variables, mock collaborators to pin the exact call sequence, and add tests until the coverage number hits its target. The implementation as written becomes the specification the suite encodes. - Choose when: A single internal algorithm is genuinely intricate (a numeric kernel, a state machine) and you want a tight test on that piece while you stabilise it, with a note to lift it to the contract once it settles; You are retrofitting tests onto untested legacy code and need a characterisation net to freeze current behaviour before you dare touch it; this is a temporary scaffold, not the goal; A coverage floor is a hard external gate and you need a stopgap to pass it today. - Cost: Every test is coupled to the private shape, so renaming a helper or moving code reds the suite even though no behaviour changed: the tax that punishes exactly the refactoring the corpus wants kept cheap; Mocks pin the call sequence rather than the result, so the suite can stay green while the real behaviour rots, then goes red on harmless restructuring; High line coverage measures lines executed, not promises checked; it reads as safety while leaving the contract and the boundary corners untested; Asserting on internals freezes accidental behaviour into a contract you never meant to offer, so your bugs end up designing the boundary for you (Hyrum's Law).

How to decide. Decide by who controls the input and how far a wrong answer travels. The governing question is what a tired engineer must hold in their head to change this code safely: a suite roped to the internals forces them to keep the private shape and the test scaffolding in mind together, so the cheap refactor stops being cheap and they leave the code worse than they found it. Spend test effort where the property is real and violating it costs more than the coupling the test drags in. At a trust boundary, where a caller or attacker controls the bytes, the property is very real and the blast radius is the whole downstream that trusted the parser, so pin the contract hard there and property- or fuzz-test the parser, because that is the one chokepoint every bad input must pass through. For pure internal logic the caller never sees, the contract is the unit's own promise, so test that promise and let the implementation move freely beneath it. Reach inside only as a deliberate, temporary scaffold, a characterisation net over legacy or a tight clamp on an intricate kernel, and write down when it gets lifted to the boundary, because an internal test that outlives its reason is pure tax. If you cannot name the contract, that is the finding: design the boundary on purpose before you test it, or your bugs will design it for you.

Reach for first. Before writing any test, write down the unit's contract in one sentence: given these inputs, it promises this observable result and these failure modes. If you cannot, the test target does not exist yet and no amount of coverage will conjure it; if you can, you have just named exactly what to assert. Then test that, through the public surface, and stop.

Pitfalls. - Treating a coverage percentage as the goal: 100% line coverage with no assertion on the actual promise is a suite that runs the code without checking it. - Mocking every collaborator until the test asserts the call sequence rather than the result, so it breaks on refactor and survives real regressions. - Asserting on intermediate state out of convenience, which silently promotes accidental behaviour into a contract callers come to depend on (Hyrum's Law). - Using example-based tests at a hostile boundary and calling it done: the adversarial and edge inputs are exactly the ones you would never hand-pick, which is what fuzzing is for. - Leaving a characterisation or kernel test pinned to internals long after its reason expired, so it lingers as a permanent refactor tax.

See also. Tenets (III), (IV), (XX), (XXIV). glossary: test the contract, not the internals, a boundary you didn't design is one your bugs designed for you. phases: implementing, verify.

Backup (tested restore) vs failover/redundancy

You hit this when A disk dies, a region goes dark, a bad migration corrupts a table, or someone runs the wrong DELETE. You are deciding what stands between that event and a loss you cannot walk back, and the choice has to be made before the event rather than during it

The call. When you need to survive losing data or a component, do you reach for failover to a live redundant copy, a backup with a tested restore, or both for different threats?

Failover / redundancy. A second live copy of the component runs hot or warm and takes over when the primary fails, so the service keeps answering. Replicas, multi-AZ standbys, leader election, anycast: the copy stays current to within a replication lag, and what you buy is continuity of availability. - Choose when: The threat is a component dying (host, disk, zone) and the cost you are buying down is downtime, not data loss; The state changes fast enough that an hour-old copy is worthless, so you need one that is seconds behind; You can pay to run the standby continuously, and the failover path is exercised rather than assumed; An attacker or an unlucky caller cannot reach the replication channel and poison every copy at once. - Cost: Redundancy can be an illusion: replicas that share one DNS, config push, certificate, or power feed fail together, so three copies are worth one until you draw the graph and prove the independence; It faithfully replicates corruption and malicious writes; a bad DELETE or a poisoned row lands on every live copy inside the replication lag, so it defends against death but not against the wrong write; The cutover is a code path that runs only in anger: an untested promotion, a split brain, or writes acknowledged then lost on failover turns one failure into two; Continuous spend, and the standby is now state you must keep correct, monitor, and patch in lockstep with the primary.

Backup with a tested restore. A point-in-time copy is captured and stored away from the primary, and you have actually restored it and timed the result. It does not keep you up; it lets you get back the result you cannot regenerate after corruption, deletion, or total loss. - Choose when: What is at stake is the result you cannot regenerate: the only copy of source-of-truth data, not a derivable cache; The threat includes logical damage (a bad migration, ransomware, a fat-fingered delete) that a live copy would only have mirrored; You can tolerate the recovery window: the time to fetch, restore, and replay is within what the business can survive; You need a copy that is independent of the primary's failure modes and out of an attacker's reach (offline, immutable, or in a separate account). - Cost: No availability on its own: while you restore, you are down, and the recovery window is the outage; A backup nobody has restored is folklore; schema drift, a missing decryption key, an incomplete dump, or a restore that takes nine hours instead of one are all things you only learn by trying; Storage and lifecycle cost, plus the discipline to expire it: kept too long and too widely, the backup quietly becomes a permanent copy that is itself breach surface and legal liability; Recovery point is bounded by backup frequency; everything written since the last good snapshot is gone.

Both, for different threats. Failover for the component-death threat (keep serving), backups with a tested restore for the logical-damage and total-loss threats (get the irreplaceable thing back). The two are not redundant with each other; they answer different questions. - Choose when: You have data that is both highly available and irreplaceable, so neither threat is acceptable to leave uncovered; Component failure and logical corruption are both realistic, and the second would otherwise propagate to every replica; The system is important enough to justify two distinct mechanisms and the operational weight of owning both. - Cost: The largest standing cost and surface: two mechanisms to fund, monitor, exercise, and keep honest, each with its own failure modes; Twice the false confidence on offer: a green replica and an old backup can both lull you while neither has been proven against the failure you actually get; Easy to let one rot behind the other; the backup restore goes untested because the replica is healthy, right up to the day corruption reaches the replica.

How to decide. Decide by the threat and by who controls the write. Sort the failure into one of two shapes: a component you assumed independent dies (host, disk, zone), or a write you did not want lands and is now authoritative (corruption, ransomware, a delete by an attacker or an unlucky caller). Failover answers the first and is silent on the second, because it copies whatever the primary writes, including the write you are trying to survive; the moment the input is attacker- or caller-controlled, every live replica is inside the blast radius and the redundancy buys you nothing against it. A backup answers the second, but only if it is independent of the primary's failure modes and out of the writer's reach, and only if you have restored it, because an unrestored backup is a claim about the past you have never checked. Spend the machinery where the property is real: pay for a hot standby only where downtime is the loss you cannot accept, and pay for an out-of-band tested restore wherever the data is the result you cannot regenerate. Run both only when both threats are real and unacceptable, and budget for the thing that actually keeps either honest, which is the rehearsal: the value lives in the proven cutover and the timed restore, not in the existence of a second copy. The standing question for the tired engineer is not how many copies you hold; it is which threat each copy is verified to survive.

Reach for first. First name the threat and the thing at stake. If what you can lose is the availability of something you could rebuild, the cheapest correct answer is often a single managed replica from your provider (a multi-AZ standby, or a read replica you can promote) with the cutover actually rehearsed. If what you can lose is the result you cannot regenerate, the cheapest correct answer is a backup you have restored end to end at least once, with the restore time measured. Either way the first move is the test, not the copy: an untested standby and an unrestored backup are both hopes, and a hope is the most expensive thing in an incident.

Pitfalls. - Treating replication as backup: replicas mirror the bad write, so a logical-corruption or ransomware event reaches every live copy inside the replication lag. - Calling a copy a backup when it has never been restored; schema drift, a missing key, a partial dump, or a nine-hour restore surface only in the attempt. - Assuming an independence that is not there: replicas behind one DNS, config push, certificate, or availability zone share a common-mode failure and fall together. - Sizing the backup by recovery point but never measuring recovery time, then discovering the restore is far slower than the outage the business can survive. - Letting the backup restore go untested for years because the replica is healthy, so the one mechanism that covers corruption is the one you never exercise. - Keeping backups everywhere forever, so the safety net quietly turns into a permanent copy that is breach surface and a deletion-rights liability.

Pin inputs (lockfile, toolchain, seed, versioned data) vs leave unpinned

You hit this when A build, a test, or a data job depends on inputs that can drift: the dependency graph, the compiler or runtime, a random seed, the upstream dataset. None of these is declared, so the run's output is contingent on whatever happened to be on the machine the day it last passed, and nobody can say which versions made it green

The call. Should you pin the inputs so the run reproduces on demand, or leave them floating and trust that what works now keeps working?

Pin everything: lockfile, toolchain, recorded seed, versioned data. Commit a lockfile for the dependency graph, pin the toolchain (compiler, runtime, base image), record the random seed alongside the output, and version the upstream data as an immutable, addressable snapshot. The run becomes a function of declared inputs: same inputs, same output, on any machine and in any month. - Choose when: The output is one you will later debug, audit, roll back, or have to defend: a release artefact, a model's metrics, a regulated calculation; An attacker or an unlucky caller controls an input, so a silent version bump can change behaviour or pull a compromised dependency without anyone choosing it; More than one person or machine must get the same result: CI, a teammate, or a future you who no longer has today's caches; A failure must be reproducible to be fixed, so a run you cannot regenerate is one you cannot diagnose. - Cost: Pins are state, and state drifts: lockfiles go stale, pinned toolchains miss security patches, snapshots cost storage. You now own a refresh cadence you did not have before; A wall of pins gives false confidence; the build is deterministic but the pinned versions may be old and quietly vulnerable, so pinning without renewal trades one rot for another; Up-front friction: someone must stand up the lockfile, the seed-recording, and the data versioning before the first run that benefits; Pinned data snapshots can grow large and need their own lifecycle, so the storage and expiry bill is real, not free.

Leave it unpinned. Depend on whatever versions, seed, and data happen to be present at run time. The build worked last quarter and it works on the author's machine, so the inputs are left implicit and the run floats with the environment. - Choose when: A genuine throwaway: a local spike or experiment whose output nobody will trust, rebuild, or roll back; The input cannot meaningfully drift in the lifetime that matters, and nothing downstream depends on bit-for-bit sameness; The pinning machinery would cost more to stand up and maintain than the run is worth, and you can say so honestly rather than from inertia. - Cost: Correctness now lives on a machine instead of in the structure, so it stops holding the moment someone else runs it: the canonical 'works on my machine'; A green build is luck, not a property; 'it built last quarter' tells you nothing about whether it builds today, because the versions that made it pass are not guaranteed to come back; Non-determinism turns a failing run into a flake you chase for days, because you cannot summon the conditions that produced it; Drift is silent and arrives at the worst time: the upgrade lands not when you choose it but when a transitive dependency or a base image moves under you, often mid-incident.

How to decide. Decide by who controls the inputs and how far a quiet change in them can reach. If an input is attacker- or caller-controlled, a floating version is a blast radius you did not choose: a transitive dependency can change behaviour, or ship something compromised, without anyone deciding to upgrade. Pin it, and pin the toolchain that builds it, because the cost of a silent change there dwarfs the small coupling a lockfile adds. If the output is one you will later have to debug, audit, or roll back, reproducibility is the precondition for all three, and a recorded seed beside versioned data is the only way a tired engineer regenerates the exact run instead of carrying the day's machine state in their head. The case for leaving it unpinned is real but narrow: a throwaway spike, a local experiment whose output nobody will trust or rebuild, where the pinning machinery costs more than the run is worth. Everywhere the output crosses a machine, a person, or a month, pin it. The question is not whether to pay for reproducibility but whether you pay now, on purpose, or later, in a flake nobody can reproduce.

Reach for first. Commit the lockfile and pin the toolchain. That is the cheapest correct move: most ecosystems generate the lockfile for you, it costs one committed file, and it buys determinism across machines and months. Add seed-recording and data versioning only where the run has randomness or an upstream dataset to pin.

Pitfalls. - Pinning the direct dependencies but not the transitive graph, so the lockfile looks complete while the versions that actually run still float. - Pinning to a mutable tag (latest, a branch, a floating image tag) instead of a content hash or exact version, which is pinning in name only. - Recording the seed in a log that is rotated away, or not recording it alongside the output, so the number survives but the way to regenerate it does not. - Pinning everything and then never renewing, so determinism is bought at the price of running known-vulnerable, unpatched versions for years. - Versioning the data pipeline's code but not the dataset, so the same code over a changed upstream silently produces a different result that still looks reproducible. - Treating a passing CI run as proof of reproducibility when CI shares the author's caches, so the drift stays hidden until a clean machine exposes it.

See also. Tenets (XXIV). glossary: works on my machine, it built last quarter, a flake, a result you can't regenerate is an anecdote. phases: implementing, verify.

Lifecycle & rollout

Expand-contract + flag/canary/ramp vs instant cutover

You hit this when You have a change that touches a lot of traffic or a lot of data: a schema migration, a rewrite of a hot path, a new model, a pricing rule. It might be wrong in a way no staging environment will show you, and the cost of being wrong in production is high

The call. Do you stage the change behind an expand-contract sequence and a flag you can ramp and roll back, or do you cut over to the new shape in one move?

Reversible rollout (expand-contract + flag, canary, ramp). Ship the new shape alongside the old, keep every intermediate state runnable, and gate the switch behind a flag you ramp from a sliver of traffic to all of it. For data: expand the schema, backfill, dual-write, switch reads, and only then contract by dropping the old. For code: ship dark behind a flag, canary on 1% of traffic or one cell, watch the signals, then ramp to 100%. The rollback path is built and exercised before the first user sees the change. - Choose when: The blast radius is wide: the change touches most requests, a shared table, or every tenant, so being wrong everywhere at once is an outage; You cannot predict the change's behaviour from staging, because real traffic shape, real data skew, or a real model under live load is the only honest test; An undo exists and is cheap to exercise, so a bad ramp is a config toggle back to the old path rather than a fresh deploy; The change is to durable state, where a one-shot drop-and-rename is a cliff with no safe step backwards. - Cost: Real machinery and real time: dual-write code, a backfill, read-path branching, flag plumbing, and the discipline to drive a ramp rather than press one button; A window where both shapes are live, so you carry double the surface and must keep the two consistent until you contract; skip the contract and you pay for two schemas indefinitely; The flag is a seam, and an unowned seam rots into a zombie flag, a permanent dead branch nobody dares delete; The rollback path is itself code, and an untested rollback is a second bug waiting for the worst moment; the reversibility is only real once you have actually run the undo.

Instant cutover (a cliff). Deploy the new shape in one move and remove the old in the same step: the drop-and-rename in a single transaction, the hard switch of all reads, the model promoted to 100% at once. The only way back is to ship again. - Choose when: The blast radius is genuinely small or contained: an internal tool, one low-traffic endpoint, a change behind an existing bulkhead or a single tenant, where being wrong harms few; The change is trivially and provably correct, or fully reversible by a later forward deploy that is fast and safe to run; Staging the change would cost more than the harm of a bad cutover, because the dual-write and backfill machinery dwarfs the thing being changed; No durable state is at stake, so there is nothing a failed mid-step can corrupt and nothing to back out of. - Cost: No safe intermediate state: once it commits there is no gradual retreat, and a problem found mid-deploy forces you over the edge in one move; Recovery is a forward fix under incident pressure, the slowest and most error-prone moment to be writing code; Failure is correlated across the whole population: everyone gets the bad version at once, turning a contained incident into a system-wide one; You learn nothing on the way up, because there is no 1% signal to read before the other 99% are committed.

How to decide. Decide by the width of the blast radius and by who is exposed if the change is wrong. The reversible rollout is not free: it drags in dual-running, a backfill, flag plumbing, and a seam you must later sunset, and that coupling is wasted on a change whose failure harms almost nobody, so spend it only where the property is real. If the change rides most of the traffic, a shared table, or every tenant, then being wrong is being wrong everywhere at once, the staging machinery costs less than the outage it prevents, and the move is to keep every intermediate state runnable, build and exercise the undo first, and ramp from a sliver so a bad release loses 1% rather than 100%. The canary is the deploy-time twin of containment, bounding how much an unlucky change can break before you notice, the same way a bulkhead bounds a bad input; it is also the whole-path probe that catches a change whose component health stays green while the journey across it degrades. But where the radius is genuinely small or already contained, and recovery is a fast forward deploy with no durable state to corrupt, the cutover's cliff is short and the reversible path is just clutter you will later have to delete. The line is durable state and population size: the moment a wrong change would corrupt data or break everyone at once, pay for reversibility; below that, take the cheaper cut.

Reach for first. First ask whether the risky change is needed at all, or whether it can be made small enough to be obviously safe. Can it ship as a pure addition that nothing reads yet, so there is nothing to undo? Can it be scoped to one cell or one tenant so the blast radius is contained by construction? If the change is small, additive, and corrupts no durable state, a plain cutover is the cheapest correct answer and the flag is overhead. Only when the change is genuinely wide or touches durable state do you owe it the expand-contract sequence and a ramp.

Pitfalls. - Building the flag and the ramp but not the rollback path, so the undo is theoretical and the first time you reach for it under load it does not work. - Stopping after switch reads and never contracting, so the old shape lingers forever and you maintain two schemas indefinitely. - Leaving the flag in after the ramp hits 100%, so it becomes a zombie flag: a dead branch with no owner that springs back in a later incident. - Ramping by clock rather than by signal, jumping 1% to 100% on a timer without watching error rates, so the canary tells you nothing before the cliff. - Dual-writing without a consistency check, so the two shapes silently drift and the eventual cutover reads from a corrupted new copy. - Treating a one-shot data migration as reversible because the code deploy is, when the drop-and-rename has already destroyed the only thing that could roll you back.

Separate the decision from the effect (dry-run/apply, four-eyes) vs do it live

You hit this when A catastrophic verb (delete, migrate, charge, cut the release) is welded to the moment someone runs it: the same code path works out what to act on and then acts on it in one breath. You cannot exercise the choice without triggering the consequence, so the operation that most needs proof is the one you test least, and the only review available is reading the command before pressing enter

The call. Do you split the irreversible action into a reviewed plan and a thin apply (with four-eyes on the irreversible step), or do you decide and act in a single unreviewable keystroke?

Split it: plan, dry-run, then a thin apply (four-eyes on the irreversible step). The decision of what or whether to act becomes a pure function that returns a plan: a manifest of intended changes. The doing becomes a thin wrapper that only executes that plan. The plan can be printed (dry-run), reviewed by a second person, and applied as a separate step with the backout written beside it. - Choose when: The verb is irreversible or expensive to undo (delete, charge, send, migrate, overwrite) and a wrong target costs more than the seam costs to build; The decision carries real logic worth testing (which rows, which tenants, which files) and you want to hammer it with thousands of cases and zero side effects; The blast radius is wide, the input is partly someone else's, or the same script will be run at 2am by someone who knows less than the author; You need a second human on the irreversible step by process rather than by luck; the seam is what makes four-eyes, dry-run and undo possible at all. - Cost: The plan goes stale: it is computed against a snapshot, and the world can move before apply. You have to pin it to a state version and re-check at apply (compare-and-swap on the version), or the split reintroduces the very race it was meant to remove; Two steps and an artefact to carry, plus a review gate that adds latency and a second person's time to every irreversible run; On something cheap and reversible the intermediate plan is pure ceremony: indirection the reader has to step through for no safety bought; Four-eyes is overhead that teams route around if it is applied too broadly, and a gate kept for its own sake wears away the trust it was meant to encode.

Do it live (decide and act in one keystroke). One command works out what to do and does it at once. There is no plan to inspect, no separate apply, no gate. The reasoning is fused into the side-effecting code, so the only review is reading the invocation before you run it. - Choose when: The action is cheap and trivially reversible: a feature flag you can flip straight back, a cache you can warm again, a single idempotent write; The decision carries no logic worth previewing; there is nothing a dry-run would show you that the command itself doesn't; The blast radius is contained and the input is entirely yours, so a wrong run is annoying rather than unrecoverable; Speed of the inner loop matters and the seam would only slow down something that costs nothing to redo. - Cost: The decision is untestable without firing the effect, so the most dangerous code is the least proven; a delete() that decides as it deletes can only be exercised by deleting; No seam means the safeguards have nowhere to hook in: no dry-run, no four-eyes, no undo; One typo or one wrong glob erases the wrong directory, and the first time anyone sees the plan is in the postmortem; Review degrades to a human eyeballing a command line under time pressure, which is exactly the vigilance the structure was supposed to replace.

How to decide. Decide by reversibility and by who controls the target set. The seam earns its keep precisely where the verb is irreversible and the input that selects what gets hit is partly out of your hands: there, a wrong run is unrecoverable, so you pay for a reviewable plan, a dry-run and a second pair of eyes, because violating "the irreversible act cannot fire untested" costs far more than the indirection the split drags in. Where the action is cheap and undoable and the target is wholly yours, the plan is ceremony: the blast radius is contained, you can simply do it again, and the extra step only adds to what a tired engineer has to hold in their head. The honest test is whether you can prove the decision correct without performing it, whether the doing-part is obviously right once it carries no logic of its own, and whether the plan is still valid at the instant it executes. If you split, close the staleness window (pin to a version, re-check at apply); if you go live, be sure the worst case really is a quick redo and not a 2am restore. Reserve four-eyes for the genuinely irreversible, so the gate stays a real safeguard rather than friction people learn to route around.

Reach for first. First ask whether the verb needs to be irreversible at all. A soft delete, a tombstone, a reversible flag flip, or an idempotent write you can simply re-run removes the need for any seam: if undo is cheap, you have nothing to gate. Only once the action is genuinely destructive and the target is non-trivial do you reach for the plan/apply split, and only on that irreversible step do you add four-eyes.

Pitfalls. - Applying a stale plan blindly: the manifest was computed against an old snapshot and the world moved, so you reintroduce the TOCTOU race the split was meant to remove. Pin to a state version and re-check at apply. - A dry-run that takes a different code path from apply, so the preview lies. The plan and the effect must be the same decision, with only the final write withheld. - Putting logic back into the wrapper: the moment the apply step makes choices of its own, it is untestable again and the seam bought you nothing. - Four-eyes as a rubber stamp: a reviewer who approves a plan they cannot read, or the same person wearing both hats, gives you the latency of review with none of the safety. - Slapping plan/apply ceremony on cheap reversible actions, training the team to treat the gate as noise and skip it on the run that actually mattered.

See also. Tenets (XI), (XXV). glossary: separate the irreversible decision from its effect, dry-run, four-eyes, catastrophic verbs fused to the moment. phases: operating, implementing.

Rewrite vs incremental refactor (strangler)

You hit this when A system is painful to change: every edit drags in surprises, the original authors have gone, and the codebase fights you at each step. The instinct is to declare it a write-off and start again on a clean page. That instinct is usually the dangerous one, because the pain you can feel is loud while the embedded knowledge you would throw away is silent

The call. When an existing system is painful, do you replace it piece by piece behind a stable boundary, or stop and rebuild it from scratch?

Incremental refactor / strangler. You wrap the old system in a stable boundary (a facade, a router, a proxy) and move one capability at a time behind it onto new code. Traffic shifts capability by capability, the old implementation shrinks until nothing routes to it, and then you delete it. The system stays shippable at every step. - Choose when: The system is in production and someone depends on it staying up while you work; You can carve it into capabilities with a boundary you can actually draw and route through, so each slice ships and earns its keep on its own; The painful behaviour is mostly understood, or can be characterised by tests and shadow traffic before you touch it; You want every step to be reversible, so a bad slice is a routing toggle back to the old path rather than a rollback of the whole programme. - Cost: Two implementations run side by side for the whole migration, and that span is usually measured in quarters, not weeks. You pay double on operations and on the cognitive load of holding both in your head; The boundary itself is real work and real risk. A leaky facade lets the old model's assumptions bleed through, and you ship the rot you were trying to escape; Slow, undramatic progress is politically hard to defend. The programme can stall at eighty per cent with the worst slice left, and a stalled strangler is just permanent duplication; Some pain is genuinely architectural (a data model or a concurrency assumption baked through everything) and refuses to be sliced. Forcing it into slices buys complexity without buying the cure.

Big-bang rewrite. You freeze or sideline the old system and build a replacement from scratch, then cut over to it once it reaches parity. The new system carries none of the old code, and the cutover is a single irreversible event. - Choose when: The old system is small enough to rebuild and re-verify faster than you could safely refactor it, and you can prove that rather than hope it; A hard external forcing function (the platform is being decommissioned, the licence is expiring, the language is dead) removes the option of keeping it alive; No boundary can be drawn through the old system at all, so incremental replacement is not actually available to you; You can run old and new in parallel and compare outputs before the cutover, so the switch is evidence-led rather than a leap. - Cost: You throw away embedded knowledge. Every bug fix and edge case in the old code was a lesson paid for in production, and the rewrite re-learns each one the expensive way; The second-system effect inflates scope. The rebuild attracts every deferred wish and lands later and heavier than the thing it replaces; Until cutover you run two systems and ship to neither cleanly: the old one is frozen and rotting, the new one is not yet real; It is a bet you have forbidden yourself from losing gracefully. If parity slips there is no partial win to bank and no cheap way back.

How to decide. Settle it on whether you can name a boundary and whether the cutover is reversible, both judged against the load on the next engineer who has to make a correct change. The strangler is the default because it keeps every intermediate state runnable and turns each step into a decision you can undo: a bad slice routes back to the old path, and the blast radius of being wrong is one capability, not the whole system. The rewrite concentrates all of that risk into one irreversible event, so it is only correct when reversibility is already gone (an external deadline kills the old system regardless) or when no boundary can be drawn through the old code, which means the incremental option does not actually exist to choose. Be honest that the strangler is not free: you carry two systems and a facade for the duration, and that duplication is itself load. Spend it only where the boundary is real, where you can name the second capability that will route through it with an owner and an end date. If you cannot draw the boundary and you have no forcing function, the answer is almost always that you have not yet understood the system well enough to replace it by either route, and the cheap move is to learn it, not to torch it.

Reach for first. Neither, first. Most rewrite urges are a reaction to localised pain, not a verdict on the whole system. Characterise the system with tests and observability so the embedded knowledge becomes legible, delete the dead branches and unused config, and refactor the two or three hotspots that cause most of the pain. Subtraction and a few targeted cuts often dissolve the case for any large programme. You escalate to a strangler only once you can name a real boundary, and to a rewrite only once reversibility is already lost.

Pitfalls. - Treating a rewrite as a clean slate when it is really a knowledge transfer: the old code encodes years of edge cases, and nobody wrote down which weird branch was load-bearing. - Drawing the strangler boundary through a leaky facade, so the old model's assumptions thread into the new code and you migrate the rot along with the traffic. - Letting the strangler stall at the hard last slice; the duplication was meant to be temporary and quietly becomes the permanent architecture. - Starting the rewrite before you can characterise current behaviour, so you discover the requirements only when the new system gets them wrong in production. - Counting the rewrite as nearly done at feature parity while ignoring the long tail of operational behaviour (timeouts, retries, weird inputs) that the old system handled silently.

Deprecate with a window vs hard delete

You hit this when You want a feature, endpoint, flag, or field gone, and you cannot fully enumerate who still depends on it. Some callers sit inside your repo; others are clients, scheduled jobs, or integrations you do not control and cannot see from here

The call. Should you retire the thing through an announced, measured deprecation window, or delete it now and let the breakage surface?

Deprecate with a window. Mark the thing deprecated, instrument it to measure who actually still calls it, give the removal a dated and owned sunset, drive migration, and only then delete. The deprecated state is a temporary holding pattern with a committed end, not a permanent label. - Choose when: The consumers are outside your control, or you cannot enumerate them from the code alone: a public or cross-team endpoint, a wire field, a shared library symbol; Breaking a straggler is expensive or invisible until it fires: a quarterly batch job, a mobile build users have not updated, a billing path; You can actually measure usage, so the window ends on observed silence over a full usage cycle rather than on a guessed date; The removal can wait long enough for callers to drain off without you sitting on a permanent fork. - Cost: You carry both code paths and their tests until the window closes, which is exactly the temporary flag that tenet XXI warns becomes a permanent branch; With no committed date and owner the window decays into permanent debt: a deprecated label that changes nothing for anyone is just the old thing with an apology attached; You are tempted to keep extending the date for the last loud caller, so the cold window never actually ends; Measurement is real work: you have to instrument the path, watch it across the whole usage cycle, and resist trusting a quiet weekend as proof.

Hard delete now. Remove the thing and its references in one change. Anyone still depending on it breaks at deploy, loudly and immediately, and you deal with the fallout as it lands. - Choose when: You genuinely control every caller and can grep them all: an internal symbol, a flag wired in one folder, a field only your code reads; The blast radius is small, contained, and yours, so a broken caller is a fast revert and not someone else's incident; The thing is duty-bound data you are obliged to destroy, where keeping it alive for a window is itself the liability (tenet XX); Leaving it half-removed is worse than removing it, because a deprecated-forever state is one nobody will ever finish draining. - Cost: Hyrum's law: with enough consumers, every observable behaviour is depended on by someone, so a delete you believe is safe breaks a dependant you could not see; The breakage lands on whoever calls at the wrong moment, perhaps a job that runs only monthly, long after you have moved on; You lose the measurement that would have told you the path was actually cold, so you are betting your enumeration is complete; If anything derived from the thing (a cache, index, replica) outlives it, you are now answering queries with a ghost rather than failing cleanly.

How to decide. Decide by who controls the callers and how wide the breakage spreads, the same tie-break the rest of the corpus keeps returning to. If you control every caller and can grep them all, and a broken one is a contained, fast revert that lands on you, then a deprecation window is machinery defending a property that was never in danger; delete it now, take out the references in the same change, and spend nothing on a process you do not need. Once a caller is someone else's (a public endpoint, a wire field, a shared symbol, a mobile build in the wild) the blast radius is wide and not yours to bound, and Hyrum's law says your enumeration is incomplete by construction. That is precisely where you pay for the window, because the cost of breaking an unseen dependant exceeds the cost of carrying two paths for one usage cycle. The window only earns that coupling if it actually ends: a deprecated state with no dated, owned sunset buys none of the safety and all of the debt, so let observed silence over a full usage cycle close it rather than a guessed calendar date. And whichever way you go, data you are obligated to destroy overrides the convenience of keeping it warm; that data gets a true hard-delete path regardless.

Reach for first. Ask first whether anyone outside your own repo can reach the thing at all. If you can grep every caller and the breakage stays inside code you own, delete it and its references in one change; no window, no ceremony. The window is the heavier tool, reached for only once a caller is someone you cannot see or control.

Pitfalls. - A deprecation window with no committed removal date and no named owner: permanent debt wearing a warning label, where the deprecated path never dies. - Ending the window on a calendar date rather than on observed silence, so you either cut off a still-busy monthly job or wait an arbitrary stretch that proves nothing. - Deleting the source of truth but leaving a cache, index, or replica behind, which keeps answering queries with a ghost for a record that no longer exists. - Finishing the implementation removal but leaving dead references behind (config keys, dashboard panels, docs, flag names), so the next reader is told the thing still exists. - Treating removal as done once it ships. A revert, a copied migration, or a new feature can quietly resurrect the thing, so encode the absence as a test or alert that trips if it comes back.

See also. Tenets (XX), (XXV). glossary: the old thing with an apology attached, Watch for the resurrection. phases: retiring.

Remove the state vs guard it

You hit this when A failure mode has surfaced: a handler trips over stale state, a config option gets set wrong, a retry double-charges. You are reaching for the check that stops it. Before you write that check, notice there is a second move available: take away the state that made the failure possible at all

The call. When a failure mode exists, do you delete the state that allows it, or add code that handles it?

Remove the state. Restructure so the failure mode cannot be represented. A stateless handler holds nothing to go stale; a deleted config key cannot be misconfigured; an idempotent write cannot be corrupted by a retry. You kill the whole class of bug rather than catching one of its instances. - Choose when: The state is accidental, not essential: a cache you could recompute, a flag whose rollout is finished, a denormalised copy with one real owner elsewhere; You can make the operation idempotent or the handler stateless without dragging in coupling worse than the bug; An attacker or an unlucky caller controls the input that triggers the failure: a path that cannot exist is a blast radius of zero, and no guard can be forgotten on a path that is gone; The guard you would otherwise write is itself fiddly enough that it becomes a second thing to keep correct. - Cost: Restructuring is real work and can be invasive: making a write idempotent may need a dedup key, a unique constraint, or a rethink of the call site, and that touches more than the one branch; Removal can collide with reversibility and with one source of truth: the state you want gone may be someone's deliberate cache or read replica, and ripping it out trades a correctness bug for a latency or coupling problem; Some state is essential complexity that belongs to the problem; trying to delete it just hides it somewhere less honest; The win is invisible in a diff that mostly subtracts, so it is easy to undervalue and hard to get reviewed.

Guard it. Leave the state in place and add the check or branch that handles the bad case: validate the input, test for staleness before use, branch on the flag, swallow or reject the duplicate. The failure mode still exists; you have built a wall in front of it. - Choose when: The state is essential: you genuinely need it, and no restructuring removes the case without removing the feature; The failure is rare and self-controlled, the cost of the guard is one obvious legible branch, and removal would cost far more than the guard; You are mid-incident and need the smallest safe change now; deletion is the follow-up, not the hotfix; The guard makes the failure visible and impossible to swallow rather than papering over it, so the next reader can still see the case can fire. - Cost: The dead branch you leave behind is the one that springs back on you in an incident: a flag flips, a config changes, an edge case you forgot returns, and the inert code wakes up at the worst moment; Every guard is more code, and software is the only material where having more of it makes the rest heavier; the check is now a permanent line the next reader must hold in their head; Guards accrete: one check invites the next, and the state that should have gone grows a hedge of branches that each need their own correctness argument; A guard that quietly absorbs the case (an empty catch, a silent default) turns a wrong state into one that has learned to hide, which is worse than the original bug.

How to decide. Settle it by asking whether the state is essential or accidental, then by who controls the input. If the state is accidental, you carry it for no reason, and the most reliable way to handle the failure is to arrange that it cannot happen: remove the case before you handle it, because the state you don't have can't be wrong. The pull is strongest when an attacker or an unlucky caller controls the trigger, because a guard can be forgotten, bypassed, or quietly disabled, whereas a path that does not exist has a blast radius of zero and nothing to keep correct. Spend the restructuring cost where the property is real and where removal does not drag in coupling worse than the bug it cures: if the state turns out to be a deliberate cache, a read replica, or essential complexity that belongs to the problem, then deletion is just relocating the difficulty somewhere less honest, and a single legible guard that keeps the failure visible is the cheaper correct answer. The tie-break is the whole corpus's: minimise what a tired engineer must hold in their head to make a correct change, and prefer fewer states to more guards, because every guard you add is a permanent line the next reader has to reason about and a dead branch waiting to spring back.

Reach for first. Before either, ask whether the state needs to exist at all. The cheapest correct fix is usually deletion of something you already regret: the finished rollout flag, the unused field, the second copy with a real owner elsewhere. If the state is load-bearing, the next cheapest move is to make the operation idempotent or the handler stateless so the failure class evaporates without a branch. Only when the state is genuinely essential do you reach for a guard, and then write exactly one, kept visible.

Pitfalls. - Adding the guard and calling it done, leaving the removable state in place so the dead branch sits dormant until a flag or config change wakes it during an incident. - Writing a guard that swallows the case (empty catch, silent fallback) instead of one that keeps the failure visible, converting a loud bug into a hidden wrong state. - Deleting state that was a deliberate cache or read replica without an invalidation path, trading a handled correctness bug for an unhandled staleness or coupling one. - Treating all state as removable: trying to subtract essential complexity that belongs to the problem, which only pushes it somewhere harder to see. - Under-valuing a subtraction in review because the diff mostly removes lines, so the highest-value change in the changeset never lands.

Owner + sunset for an artefact vs leave it running

You hit this when You ship an operational artefact that exists to be temporary or contingent: a feature flag, a cache, a quota, a lease, a TLS certificate, a back-compat adapter, an in-flight migration. It works today. The open question is what becomes of it once you have moved on and the rollout it served is long finished

The call. Do you attach a named owner of record and a sunset or review date the day the artefact ships, or do you leave it running and deal with it if it ever becomes a problem.

Owner + sunset attached at ship time. Every persistent flag, cache, quota, lease, cert and migration gets a single named owner of record and a date written down the day it ships: a removal date for the temporary, a review date for the durable. The record lives where the artefact does (the flag config, the cert inventory, the migration ticket) and is enforced by something that nags: a linter, a dashboard, an expiry alert. - Choose when: The artefact is contingent by design and is meant to disappear: a rollout flag, a dual-write window, a back-compat adapter, an expand-migrate-contract step whose whole reason to exist has an end; Its silent failure or silent persistence has a wide blast radius: an expired cert takes down every caller, a stale cache serves wrong answers downstream, a forgotten quota throttles a launch; It carries authority an attacker or unlucky caller could trip: a lease that fences writes, a flag that re-routes traffic, a quota that gates spend; Nobody on the team can already answer "who deletes this and when" without going to look. - Cost: Real upfront friction on every artefact, and the cheap reversible ones feel it most. Ceremony applied without regard to cost is exactly what tenet XXV warns wears away the trust it was meant to encode, so the date and the owner have to bite or people route around them; A name and a date are only as good as the process behind them. An owner who has left, a date that slips with one click and no questions asked, a review that rubber-stamps: any of these turns the discipline into theatre that looks like safety without being it; Maintenance of the registry itself. The owner list, the expiry job and the dashboard are artefacts too, and they go stale the same way, so the map needs an owner and an expiry like anything else that can drift, which is the trap THE-SHAPE-OF-THE-WHOLE names for a dependency graph drawn once and left to rot (Law I).

Leave it running. Ship the artefact and move on. No owner recorded, no date set. It keeps working as long as nothing changes around it, and you revisit it only if and when it causes pain. - Choose when: The artefact is genuinely throwaway and you delete it in the same change: a flag flipped and removed in one pull request, a one-shot script that does not survive its run; It is fully contained, owned by you, with a blast radius that ends at code you alone touch, and no attacker or external caller can reach the trigger; The honest expected lifetime is hours, and you can point to the line that removes it. - Cost: Drift to a zombie. The flag outlives its rollout and becomes a permanent dead branch nobody dares delete, because no one is sure what still depends on it, and it sits there as a dormant branch that can spring back in an incident exactly as tenet XXI warns; The 03:00 "who owns this?". An unowned service or cert is the component nobody dares touch during the outage, because in the moment something has to change fast nobody knows who is authorised, what is safe, or what the side effects will be, so a repairable thing turns untouchable when you can least afford it; Two beliefs at once. An unowned cache with no invalidation path drifts from the rows it is meant to count, and at the worst moment you cannot tell which copy is real (XIV); The cost lands on someone else, later, with no context, and it grows with every other unowned artefact piling up beside it until the team is drowning in branches nobody will delete.

How to decide. Decide by what a tired engineer must hold in their head when this thing surfaces again, and by who can trip it. The cost of the discipline is fixed and small: a name and a date. The cost of leaving it running is paid later, by someone else, under load, and it scales with blast radius and with who controls the trigger. So spend the owner and the sunset wherever the artefact can fail silently or persist silently into a wide blast radius, or wherever it carries authority an attacker or an unlucky caller could trip: the cert every client trusts, the flag that re-routes traffic, the lease that fences writes, the quota that gates spend. There the property is real, and the unowned version costs far more than the line of config the cure drags in. Where the artefact is yours, cheap, contained and obviously about to be deleted, skip the ceremony and just delete it; and if you must defer the deletion, the deferral itself is now the artefact that needs the name and the date. The test is the one tenet XXV poses: when this breaks and you are not here, does someone know they own it and know what to do, without "who owns this?" being the first question of the incident.

Reach for first. Cheapest correct answer: do not create the durable artefact at all. A flag deleted in the same pull request that fully rolls it out, a cache made unnecessary because the derivation is now cheap enough to recompute (XIV), a migration finished and contracted in one stretch of work: none of these needs an owner or a sunset, because there is nothing left to own. The artefact you remove this week is the one you never have to track. Only when the thing genuinely must outlive the change that birthed it do you owe it an owner and a date.

Pitfalls. - A date with no teeth. An expiry anyone can push out indefinitely, with no alert and no review, is decoration; the date has to be enforced by something that nags and escalates, or it is not a sunset. - Owner is a person, not a team, and the person leaves. "Owner of record" has to survive staff churn, so bind it to a rotation or a team alias with a real human accountable, or the field is stale the day they hand in their badge. - Sunsetting the flag but not its dead branch. Removing the toggle while leaving the now-unreachable code path behind just relocates the zombie; the contract is to delete the branch too. - Ceremony on the cheap and reversible. Forcing an owner and a review date onto a flag you will delete next week is the friction tenet XXV says teams quietly route around, which corrodes the controls that guard the genuinely irreversible. - Treating the registry as truth without reconciliation. An owner list never checked against what is actually running rots into confident fiction, and a cert nobody tracked expires while the spreadsheet still says it is fine.

See also. Tenets (XXV), (XXI), (XIV). the companion's first law. glossary: zombie flags, one owner of record, the unowned service outage. phases: operating, integrating, retiring.

Soft-delete for recoverability vs hard-delete with enforced TTL

You hit this when You are removing data, and the removal has to mean two opposite things at once. Reversibility wants the row recoverable, because the commonest reason a record vanishes is that somebody fat-fingered a delete and needs it back within the hour. The duty to forget wants the row gone, because data you still hold is breach surface and a standing legal liability under deletion rights. A soft delete that flips a flag and keeps the bytes satisfies the first and quietly fails the second; a hard delete that overwrites in place satisfies the second and turns an honest mistake into a restore-from-backup incident. The fork is not which is safer in the abstract. It is which property is real for this particular data, and who controls the input that triggers the delete

The call. For this class of data, is the risk that you delete something you needed, or that you keep something you were obliged to destroy?

Soft delete with a recovery grace window. The delete marks the row (a tombstone, a deleted_at timestamp, a status flag) and hides it from normal reads. The bytes stay for a fixed window during which an operator or the user can undo the mistake, then a sweeper removes them. Reads filter out tombstoned rows by default; recovery is a flag flip, not an archaeology dig. - Choose when: The data is self-controlled and low-sensitivity: app state, drafts, config, anything where the worst outcome of keeping it a few more days is some wasted disk; An accidental delete is a realistic and frequent failure, and the cost of losing the row uncovered is higher than the cost of holding it a little longer; You can name and enforce the window: a TTL, a sweeper that actually runs, an owner who notices when it stops; Foreign keys and downstream consumers can tolerate a row that exists but is hidden, without leaking it through a join that forgot the filter. - Cost: Every read path now carries an invisible WHERE deleted_at IS NULL, and the one query that forgets it leaks deleted data; the filter is a rule you enforce by vigilance, exactly what the shape is supposed to remove; The grace window is retention by another name: for regulated data it is the duty to forget, deferred, and a sweeper that silently stops turns the window into keep-forever; Tombstones accumulate, indexes bloat, and uniqueness constraints fight the ghost of a deleted key; It reads as done when it is not: the data is still there, still breach surface, still discoverable, for as long as the window lasts.

Hard delete with an enforced TTL. The delete destroys the data, or schedules it for destruction on a retention clock that the storage layer enforces: a row-level TTL, a droppable time-partition, an object-lifecycle rule. The data you no longer hold cannot leak in a breach, cannot be subpoenaed, cannot be sold by a future product manager. The retention policy is encoded in the schema and the infrastructure, not remembered by a person. - Choose when: The data is duty-bound: personal data under deletion rights, secrets, anything where holding it past its purpose is itself the liability; The input is caller- or attacker-controlled, or the blast radius of a leak is wide: the less of it you hold, the smaller the incident; A wrong delete is rare or cheaply re-derivable, so reversibility buys little; You want retirement to be true retirement, not a renamed bucket nobody is allowed to empty. - Cost: A genuine mistake, a bad migration, a buggy batch job, is now unrecoverable past the TTL; you are trading recoverability for the guarantee; Hard delete must cascade honestly: a deleted user whose rows linger in a denormalised cache, a search index, a log line, or a backup is not deleted, and proving completeness is real work; Backups and replicas inherit the duty: a TTL on the primary that the cold copy ignores is a hole; Get the partition or lifecycle rule wrong and the clock destroys live data on schedule, quietly and on time.

Resolve per data class: soft window, then hard delete at the TTL. Classify the data, then compose the two. Sensitive or duty-bound classes get a short soft window for accident recovery followed by an enforced hard delete; benign classes can keep a longer window or none. The classification lives in the schema (a column tag, a per-table policy) so the retention behaviour is a property of the data, not a decision re-made at each call site. - Choose when: You hold a mix: some rows are pure app state, some are personal data, and one global policy would either over-retain the regulated class or under-protect the recoverable one; You can actually classify and keep the classification current as columns are added; The short window genuinely covers the realistic accident, so the hard delete at the end is rarely the thing that bites; You want one mechanism whose behaviour varies by data class rather than two parallel delete paths to keep in sync. - Cost: Classification is the load-bearing part and it rots: a new column lands unclassified and inherits the wrong default, usually keep; Two-stage delete is more moving parts: the soft sweep, the hard sweep, the clock that links them, each a place to fail silently; It is the most code and the most state of the three, justified only when the mix is real; A class boundary drawn wrong, sensitive data filed as benign, fails in the most expensive direction and you may not notice until the breach.

How to decide. Decide by who controls the input and how wide the blast radius is, not by which feels safer. Ask what a leak of this data would cost and who can trigger its creation. If the data is self-controlled, low-sensitivity, and you carry no duty to destroy it, the dominant failure is the accidental delete, so reversibility wins and a soft window with an enforced sweeper is right; the cost you accept is the read-path filter and some retained bytes. If the data is duty-bound or its leak has a wide blast radius, then keeping it is the liability and the duty to forget governs: hard-delete with a TTL encoded in the schema and the infrastructure, so the property holds without anyone remembering it, and pay the loss of recoverability as the price of not holding breach surface. Where you hold both kinds, do not pick one default for all of it; classify and compose, short window then hard delete for the regulated class, and treat the classification itself as the thing most likely to rot. The trap is letting reversibility quietly override the duty to forget because keeping data is the path of least resistance. Default-keep-forever is the wrong default. The grace window is retention you owe a reason for, and an archive is just a permanent copy nobody is allowed to empty unless the hard-delete clock is real.

Reach for first. A soft delete with a short, enforced TTL is the cheapest correct move for data you control and carry no duty to destroy: it recovers the common accident and the sweeper bounds the retention. Reach for it only once you have confirmed this data class carries no deletion duty. The moment it does, the cheapest correct move flips to a true hard-delete path with the retention clock encoded in the schema, because no amount of recoverability is worth holding data you were obliged to forget.

Pitfalls. - A read path that forgets the deleted_at filter and serves tombstoned rows: the soft delete leaked the very data it claimed to remove. - A soft-delete sweeper that silently stops running, so the grace window quietly becomes keep-forever and your retention policy is fiction. - Calling non-deletion an archive: regulated data renamed into a bucket nobody empties is still the data you were obliged to destroy, sitting on disk. - Hard delete on the primary while backups, replicas, search indexes, caches, and log lines keep an uncascaded copy: deleted everywhere except where it counts. - A TTL or droppable partition misconfigured so the clock destroys live data on schedule, on time, and unrecoverably. - Unclassified-by-default: a new column lands without a retention tag and inherits keep, so sensitive data accretes in the cracks of a per-class scheme. - Uniqueness constraints and foreign keys fighting tombstones, forcing hacks that leak the ghost of a deleted key.

See also. Tenets (XX), (XVI). glossary: reversibility vs the duty to forget, a duty to forget, an archive is a permanent copy. phases: retiring, implementing.

Security & trust

Narrow, short-lived credential vs broad/long-lived

You hit this when A component needs authority to do its job: a database role, an API token, an IAM policy, a service account. You are deciding how much it gets and for how long. The easy path is to hand it everything that might ever be useful, on a credential that never expires, because then it just works and nobody pages you about a missing grant. The question is what that convenience is worth on the day the credential leaks, or the day the code does the wrong thing with the power it happened to be holding

The call. Should this component get the narrowest scope and shortest lifetime that does the job, or a broad, long-lived credential that saves you the trouble of scoping it?

Least privilege, by construction. Grant the narrowest authority that lets the component do its job, and make the limit structural rather than a matter of trust: a read-only role, a token scoped to one bucket and one verb, a short TTL that expires on its own. A breach is then held back by authority that was never handed over, not by the code happening to behave. - Choose when: The credential touches anything an attacker or an unlucky caller can reach: a public endpoint, untrusted input, a third-party dependency in the supply chain; The broad version's blast radius is a catastrophe rather than an incident: a wildcard that can drop tables, read every tenant, or exfiltrate every bucket; The component's real needs are small and stable, so the narrow scope is a couple of grants rather than a moving target you re-edit every week; The platform can enforce it for you: per-resource IAM, scoped OAuth tokens, short-lived credentials issued on demand. - Cost: Up-front work to enumerate exactly what the component touches, and a re-grant every time its job genuinely grows; Short TTLs need a rotation or refresh path, which is real machinery: an issuer, a renewal loop, a fallback for when renewal fails; A scope drawn too tight fails closed at the worst moment, and a missing grant in the small hours reads like a bug rather than the safety net it is; More moving parts in the access model means more for a tired engineer to hold in their head when they change who can do what.

Broad, long-lived credential. Issue one credential with wide scope, often a wildcard on actions and resources, and no meaningful expiry, then reuse it everywhere. It is the path of least resistance: it works on the first try, never expires under you, and never pages anyone about a denied action. - Choose when: The component runs in a fully trusted, isolated context with no attacker- or caller-controlled path to it, and nothing else shares the credential; It is a genuine throwaway: a local spike or a sandbox you will tear down, where the blast radius is bounded by the environment itself; The narrow version cannot be expressed on the platform you are stuck with, and you have a compensating control (network isolation, a separate account) that bounds the damage another way; The authority is intrinsically broad and irreducible, so a narrow grant would be theatre that adds machinery without shrinking the real reach. - Cost: A single leaked key, injected dependency, or confused-deputy bug inherits authority over everything the policy can reach, and the breach becomes a catastrophe rather than a contained incident; Long life means a credential leaked today is still valid months from now, long after the leak is forgotten, with no expiry to close the window; The signature lies about intent: a function holding broad power tells the reader nothing about what it can actually affect, so over-reach hides in plain sight; It spreads. A convenient broad credential gets copied into the next service, and every future caller now wields power it never needed and you can no longer audit.

How to decide. Decide by who controls the path to the credential and how wide the damage spreads if it is misused. Trace the worst input that can reach this component and ask what it could touch with the authority in hand: where that path runs from anything an attacker or an unlucky caller controls, and the broad grant's reach is the difference between an incident and a catastrophe, the scoping is not optional, and you pay for it in construction rather than in trust. The narrowness has to come from the capability being absent, because trust must be re-earned on every change by every caller, whereas containment you buy once in the structure and it holds. Yet scope is not free, and the entry's job is to stop you spending it as ritual: a TTL and a refresh loop, a grant matrix another engineer must reason about, all add to what a tired person holds in their head while changing the system. So spend that complexity only where the property is real, where the authority genuinely reaches something costly and that reach is genuinely controllable, and where violating least privilege would cost more than the rotation machinery and the re-grant friction the cure drags in. Where the context is truly trusted and isolated, or the authority is irreducibly broad, the narrow version is theatre, and the honest move is a broad grant with the isolation written down. Everywhere a hostile or careless input can reach, draw it tight.

Reach for first. Before you design any grant, try to remove the need for one. The cheapest credential is the one that does not exist: read from a config the component already holds, take the two fields a function actually touches rather than an object that carries an authority with it, let the platform inject a short-lived identity at runtime so nothing long-lived is stored at all. Authority you never hand out cannot leak or be misused. Only once the need is real do you reach for scoping it.

Pitfalls. - Scoping the actions but leaving the resource a wildcard, or the reverse: a read-only role that can still read every tenant is broad where it counts. - Treating a short TTL as least privilege while the scope stays a wildcard. Lifetime and scope are two separate dials, and both have to be turned down. - Granting for the rare case: handing out write because one quarterly job needs it, so the everyday path carries authority it never uses. - Copying a working broad credential into the next service because it already works, quietly widening the reach and breaking the audit trail. - Scoping so tight it fails closed in production, so the on-call engineer, lacking the context, fixes it by widening to a wildcard. - Passing the whole authority-carrying object when a function needs two fields, so the call hands out sweeping power the signature never reveals.

Treat crossing input as hostile vs trust it (including the supply chain)

You hit this when Something is crossing into your program from outside: a request body, an uploaded file, a queue message, last week's data dump, a sibling service, or a package you just typed install against. You are deciding, at this exact crossing, whether to scrutinise what comes through or to take it at face value because it looks well-formed and the package has a version number and a friendly name

The call. At a given crossing into your program, do you treat the incoming value (and the dependency you import) as hostile until proven safe, or do you trust it?

Hostile until proven: validate, bound, and contain at the crossing, package included. You mark the crossing as a trust boundary and treat everything through it as untrusted until checked. Parse and validate shape, cap size and array lengths before allocating, re-derive identity from a verified token rather than from a field the caller asserts, and authorise the actor for this specific action separately from authenticating them. The dependency you import counts as input too: pin it with checksums, verify signatures, keep the surface small, and run the build with least privilege so a compromised package is contained by authority it was never granted. - Choose when: The other side is anyone you do not control: the public socket, a form field, a file upload, a queue, a partner service, or a value the request asserts about itself; Blast radius on a bad value is wide: it allocates memory, drives actuator maths, hits the database, escalates privilege, or runs with your secrets; You are importing or upgrading code that will run in-process with your full authority, where there is no privilege boundary around the import the way there is around a network call; The crossing is a real edge: a process boundary, a deserialisation point, or a privilege change, not an internal call between your own functions. - Cost: Validation, size caps, signature checks, and lockfile pinning are code you write, test, and maintain, and they are easy to get subtly wrong (a bound off by an order of magnitude protects nothing); Supply-chain rigour has standing overhead: pins go stale, signature tooling breaks builds, and a minimised dependency surface means writing things you could have imported; Applied past the real boundary it becomes the tenet III anti-pattern: re-validating at every internal hop is a latency tax, and zero-trust between your own in-process functions buys clutter for no real reduction in threat; Strictness rejects input that was merely odd rather than malicious, so a too-tight bound becomes its own availability bug.

Trust it because it parsed and the name looks familiar. You take the value on its face. It deserialised cleanly, the structure looks right, the dependency is popular and pinned to a tag, so you act on it directly: allocate to its size, believe the userId in its payload, import the package and run your build with whatever keys are lying around. The boundary is treated as a checkout step or a convenience rather than an attack surface. - Choose when: The source is genuinely inside your control and inside the same privilege domain: a value you produced and already parsed once, carried inward as a typed object; The crossing is internal, where re-checking would be the parse-twice anti-pattern and the threat reduction is nil; Throwaway code on trusted-only data where there is no attacker and no caller you do not know, and the cost of a wrong value is a crash you will see immediately. - Cost: A well-formed value can still be an attack: the 2 GB JSON body parses fine and then kills your heap, because well-formedness bounds shape, not magnitude; Crashing on a hostile-but-valid value hands an attacker a denial of service for the price of one request; Trusting an asserted identity or authority is the classic privilege escalation; authentication tells you who, it says nothing about what they may do; On the supply chain the failure is silent and total: a typosquat or a poisoned transitive dep runs with your full authority, and code you shipped but did not write is still code you shipped.

How to decide. Decide by who controls what comes through and how far a bad value can reach, not by how friendly it looks. The strict treatment costs real work and, applied everywhere, degrades into clutter and a latency tax, so do not spend it inside your own process: parse and validate once at each genuine crossing (process edge, deserialisation point, privilege change), then carry the parsed type inward so internal callers trust the type and not the wire. Spend the scrutiny exactly where the property is real, that is, where the other side is outside your control and the blast radius is wide: bound the size before you allocate, re-verify the authority before you act on it, and treat the import as input because there is no privilege boundary around an in-process dependency the way there is around a socket. The tie-break is whether violating safety here would cost more than the coupling the checks drag in. A public socket or an imported package controlled by a stranger, able to exhaust memory, escalate privilege, or run with your secrets, clears that bar easily; a value you produced and already parsed does not. When you are unsure who controls the input, assume you do not, because the failure mode of trusting a hostile value is unbounded and the failure mode of over-checking a friendly one is merely tedious.

Reach for first. Find the real boundary and check once there. Before any heavy machinery, name the crossing: cap body size and array lengths before deserialising, re-derive identity from the verified token, and for the package side pin with checksums in the lockfile and scope the build's authority down. One deliberate check at the true edge, with the parsed type carried inward, beats both scattering validation through internal calls and waving the input through.

Pitfalls. - Treating well-formed as safe: validating shape but not magnitude, so the attacker-sized payload sails through and then exhausts the heap. - Trusting identity the request asserts about itself (a userId in the body) instead of re-deriving it from the verified token, which is straight privilege escalation. - Re-validating at every internal hop, turning the parse-once discipline into the parse-twice anti-pattern and a latency tax for no reduction in threat. - Treating the dependency as outside the boundary because it is popular or has a friendly name, when it runs in-process with your full privileges and your CI runs with your keys. - Pinning the package but giving the build broad * authority, so a compromised dep is contained by nothing.

See also. Tenets (IV), (XVI). glossary: trust boundary, containment is cheaper than trust, the typosquat boundary. phases: implementing, integrating, triage.

Re-derive identity and authority from a verified token vs trust the payload

You hit this when A request arrives and it tells you two things about itself: who it is and what it is allowed to do. Maybe it carries a userId field, a role: "admin" flag, an accountId in the path, an X-Tenant header. The handler needs a subject and a permission to proceed, and the easy thing is to read them straight off the request

The call. Do you re-derive the subject and authority from a cryptographically verified credential and check authorisation against the actual resource on the path, or do you trust the identity and role the request asserts about itself?

Re-derive from the verified token, then authorise the actual path. Verify the credential's signature and freshness first, take the subject only from inside the verified claims, then run a separate authorisation check that the subject may perform this specific action on the specific resource named in the request. - Choose when: The request crossed a trust boundary: a socket, a browser, a sibling service, a queue message. Anything an attacker or an unlucky caller controls; The action touches a resource scoped to a subject or tenant: their order, their file, their account; The subject and the role can diverge from what a caller would like them to be, which is to say always. - Cost: Two checks rather than one: a verification step and an authorisation step, with the discipline never to let the second read from anything but the first; An ownership or policy lookup on the path itself (does this subject own order 42?), which costs a query the payload-trusting version skips; Token plumbing: signing keys, rotation, clock skew, revocation. Real machinery you now own and have to keep working.

Trust the user id or role in the payload (the confused deputy). Read the subject and authority from fields the request supplies, userId, role, isAdmin, and act on them directly. The server becomes a deputy executing whatever authority the caller claimed. - Choose when: The value never crossed a trust boundary: it was set by your own code on the inward side of a verified edge, in the same address space, and no external caller can reach it; A throwaway prototype with no real data and no exposure, where you will pay the verification cost later or never ship it at all. - Cost: This is the classic privilege escalation. Change the userId to someone else's, flip role to admin, and the server obeys. The blast radius is every resource the deputy can reach; It looks identical to the safe version in a code review: a field is read and used. The danger is invisible at the call site, which is why it survives; Combined with a broad * grant on the service, a single forged field turns a contained incident into account-wide compromise.

How to decide. Settle it by asking who controls the field and how far a forged value reaches. Authentication and authorisation are separate checks: proving who is making the request tells you nothing about what they may do, so being logged in is never the same as being allowed. The deciding factor is the direction the value came from. A subject or role that arrived from across a trust boundary is hostile until proven otherwise, and a payload field is exactly such a claim, asserted by the very party who benefits from lying about it. So the cost of the second check is not optional machinery; it is the only thing standing between a forged field and the resource behind it. Re-derive the subject from inside the verified credential, never from the body, and authorise against the actual resource named on the path. The reason this is worth the query and the token plumbing is the blast radius: a payload you trust is authority a caller can mint at will, and if the service also holds a broad grant, one forged field reaches everything that grant can touch. Spend the two checks precisely where the input is attacker-controlled. The one place you may legitimately read a subject off a plain field is strictly inward of a verified edge, in your own process, where no external caller can set it; there, re-checking buys clutter and no real reduction in threat.

Reach for first. Take the subject from the verified token the framework already gives you (the parsed JWT claims, the session principal) and ignore any identity field in the body entirely. Most handlers need nothing more: identity from the credential, then a single ownership check against the resource on the path. Reach for bespoke token plumbing only once that is in place and proves insufficient.

Pitfalls. - Verifying the token and then still reading the subject from the body, so the authentication is theatre and the payload field is what actually drives the action. - Checking authentication but not authorisation, so any logged-in user reaches an admin action because the code confirmed who they are and never asked whether they may. - Authorising against the id in the request rather than the resource on the path, so user A passes a check by claiming to be user A while operating on user B's order. - Trusting a header or claim set by an upstream proxy without confirming the proxy is the only route in; a direct request forges the header. - Decoding a JWT without verifying its signature, which reads the claims an attacker wrote and calls it authentication.

See also. Tenets (IV), (XVI). glossary: authentication vs authorisation, a broad * IAM grant. phases: implementing.

Simplicity & structure

Inline/duplicate (locality) vs unify (DRY/one source)

You hit this when You are writing or reviewing a change and two stretches of code read almost the same: a validation guard, a mapping table, a small calculation, a shape that turns up in two handlers. The reflex says fold them into one helper. But the two sites may answer to different owners and change for different reasons, and unifying them now couples them for good

The call. When two pieces of code look alike, do you unify them behind one definition, or let them stay as separate copies?

Unify: one authoritative source, derive the rest. Name the shared fact once, give it a single owner, and have every other site read or derive from it. The classic single-source-of-truth move: one definition, no second writable copy. - Choose when: The two sites encode the same fact and must change together: a tax rule, a wire format, a permission check whose meaning is owned by one part of the system; Drift between the copies is a correctness bug rather than a cosmetic nit (the cache that disagrees with the rows, two columns holding the same number); One side genuinely controls the meaning and the other should follow it, so the dependency direction is obvious and one-way; A type can carry the invariant, so unifying also makes a whole class of mistake unrepresentable rather than merely deduplicated. - Cost: The reader of either call site now has to look somewhere else to verify the code: the puzzle gains a piece that lives off-screen, which is the tax tenet I is built to avoid; Coupling is forced. The day the two callers need to diverge you pay to tease them apart, and the abstraction often grows flags and branches to serve both masters before anyone admits the unification was wrong; Premature unification of things that were only coincidentally alike is the wrong abstraction, and backing out of a shared helper that ten callers lean on costs far more than the duplication ever would; A shared helper is a shared blast surface: a caller that controls input can reach behaviour every other caller depends on, so the unified path needs the validation and bounds all of them needed, not the loosest.

Duplicate: keep each copy local. Leave the two stretches as independent copies. Each call site is self-contained and can be read, verified and changed without leaving the screen, even though the bytes look similar today. - Choose when: The similarity is coincidental: same shape now, different reasons to change, owned by different parts of the system, and likely to drift apart anyway; The copy is small and self-evidently correct (a three-line guard, a short literal table) so a reader needs nothing beyond what is in front of them; The two sites belong to modules you want to keep independently deployable or independently reasoned-about, and a shared dependency would bind them; You are early and the real axis of variation is not yet clear, so committing to one abstraction would guess wrong. - Cost: If the copies do encode one fact, you now have two facts: one update misses the other and they disagree, which is precisely the drift tenet XIV warns about, and not if but when; A bug fixed in one copy silently survives in the other, and nothing tells the next reader the twin exists; Duplication scales badly: three copies is tolerable, eleven is a refactor nobody schedules, and the longer you wait the more entangled each copy gets with its local context; A name carried by both copies can quietly become a duplicate of an invariant: when the fact changes, every copy and every name that asserts it has to be found and changed by hand, and the ones you miss start lying.

How to decide. Decide by the reason to change, not by resemblance. Ask whether the two sites encode the same fact owned by one part of the system, so that a divergence between them would be a bug; if so, unify, give the fact one owner, derive the rest, and let a type carry the invariant where a type can. If instead they are only coincidentally alike and will change for different reasons under different owners, copy them and keep each one local. The governing tie-break settles the close calls: minimise what a tired engineer must hold in their head to make a correct change. A copy that stands on its own costs nothing to read but costs you drift if it was secretly one fact; a unified helper saves the drift but taxes every reader with an off-screen lookup and welds the callers together. Spend the coupling only where the shared property is real, where violating it (two beliefs at 3am, no way to tell which is true) would cost more than the off-screen lookup the abstraction drags in. Weight the unify side harder when input crosses a trust boundary: a shared validation or parsing path is the right place to make the illegal state unrepresentable once for everyone, whereas a shared helper that quietly widens what any one caller can reach has enlarged the blast radius for all of them.

Reach for first. Duplicate and wait. Write the copy, leave it local, and let the real axis of variation reveal itself. Unify on the third genuine occurrence (the Rule of Three), once you can see what is actually shared and what was only the shape of the day, and even then only if the copies are the same fact rather than the same bytes.

Pitfalls. - Deduplicating on syntactic resemblance: two blocks look identical today and get merged, then sprout a boolean flag the first time one caller needs to differ, and the helper slowly becomes two functions wearing one signature. - Treating all duplication as debt: a small self-evident copy is often cheaper and clearer than the indirection that removes it, and chasing zero duplication manufactures the wrong abstraction. - Calling a genuine single source of truth mere duplication and copying it anyway. Two writable copies of one fact is not locality; it is a drift bug with a head start. - Encoding the shared invariant in a name (chargeOnceIdempotent) instead of a type, so the name becomes a second copy of the fact that keeps asserting it after the behaviour has changed. - Unifying across a module or service boundary you wanted to keep independent, trading a few duplicated lines for a coupling that defeats independent reasoning and deployment.

See also. Tenets (I), (XIV). glossary: a puzzle scattered across the repo, a name is a duplicate of the invariant. phases: implementing, reviewing.

Abstract now vs wait (rule of three)

You hit this when You have written the same shape twice, or you can see a third caller coming, and the instinct is to hoist the common part into one place now. The two copies look close enough that a single helper feels obviously right, and leaving them apart feels like a corner cut

The call. When you see a repeated pattern, do you build the abstraction now or duplicate until a third case reveals the real shape?

Wait (rule of three). Duplicate the pattern, accept two or three copies, and extract the abstraction only once a third real caller shows you which parts actually vary and which are stable. - Choose when: You have only one or two callers and cannot yet name, with confidence, the axis along which they differ; The copies are small and local, so a future edit lands in a handful of places that grep will find; The variation is driven by callers you do not control (other teams, external formats, user-supplied config), where guessing the seam early is guessing on someone else's behalf; You can mark the duplication honestly (matching names, a comment, a shared test) so the next reader sees the copies are the same shape. - Cost: A genuine cross-cutting change must be applied in every copy, and a copy missed in the edit is a real bug that diverges silently; Readers must notice the copies are the same shape with no name pointing at it, and over time the copies drift until they are no longer the same shape at all; There is a window where the code looks like a corner cut to anyone who has not internalised why you are waiting, so the discipline has to be visible, not merely intended.

Abstract now. Hoist the shared part into a single helper, base class, generic, or configurable engine the moment you see the pattern repeat, so there is one place to change and no copies to drift. - Choose when: The shared part is a real invariant of the domain, not a coincidence of two early callers, and you can name the varying axis precisely; Getting the copies out of sync would be a correctness or security failure, not just an untidy edit, so a single source of truth is worth the coupling it drags in; You own all the inputs, so the seam you pick is not hostage to a format or a team you cannot change; The abstraction stays in a real language with types and tests, rather than growing into configuration that quietly becomes one. - Cost: Placed on two points, the seam is a guess; when the third case lands on an axis you did not foresee, every caller pays to route round the wrong abstraction; Each later caller must learn the abstraction's parameters and flags, which accrete to cover differences that were never essential, and the machinery makes the surrounding code heavier to read and change; An abstraction with three dependents is far costlier to undo than two copies are to merge, so a wrong early bet hardens into something nobody wants to touch; A configurable engine that grows conditionals, variables and interpolation reinvents a programming language badly, with no debugger, types or tests.

How to decide. Settle it by who controls the variation and how wide a wrong guess spreads. An abstraction is a bet on which axes will vary, placed with the least information you will ever have: two points define a line, and almost any line fits two points. The third case is what disproves the lines that were only ever coincidence, so the rule of three is less folklore than a sampling rule, waiting until you have enough cases to tell an essential axis of variation from an accidental one. Cost the two failures honestly. Premature duplication costs a later edit applied in two or three known places, found by grep, each change local and obvious. A premature abstraction costs every future caller, who must learn the wrong seam, route their case through parameters and flags that paper over differences which were never real, and fight the structure when their need turns out to sit on an axis you did not foresee; software is the one material where adding that machinery makes everything around it heavier to read and change. So spend the abstraction only where the shared property is real, stable, and owned by you, rather than dictated by the shape of two early callers. Where the inputs come from outside (different teams, an external format, a config surface a user fills in) the variation is not yours to control, and an early DSL hardens into a half-language nobody can debug. In genuine doubt, duplicate: subtraction is real progress, and merging two copies later is a smaller, safer change than tearing apart a wrong abstraction that three callers now depend on.

Reach for first. Duplicate, and name the copies well enough that the next reader can see they are the same shape. Two honest copies cost almost nothing to hold in your head and almost nothing to delete, and the abstraction can wait until a real third caller tells you which parts genuinely vary. Better still, if you can remove the repetition outright (collapse to one caller, or drive both from a data table instead of code), do that first; the cheapest abstraction is the one you never wrote.

Pitfalls. - Counting syntactic matches instead of semantic ones: two blocks that look identical today but answer to different reasons will be forced together, then torn apart the moment one reason changes. - Building a configurable engine to dodge writing code, so the logic moves into a config DSL that grows conditionals and loops until it is a half-designed language with no debugger or tests. - Treating the rule of three as a licence never to abstract: where out-of-sync copies are a correctness or security failure, a missing single source of truth is a liability dressed up as simplicity rather than YAGNI. - Extracting before the third caller, then bending each new case to fit the guessed seam with extra flags rather than admitting the abstraction was wrong. - Leaving silent duplication with nothing to signal the copies are the same shape, so they drift until a fix lands in one and not the others.

See also. Tenets (XXI). glossary: software is the only material that gets heavier, a config DSL reinvents a language badly. phases: implementing, planning.

Make illegal states unrepresentable (types) vs runtime guard

You hit this when A value can sit in a state that means nothing: three booleans (isLoading, hasError, data) admit eight combinations when only three are real, so "spinner showing over stale data" is a state the code can build and a reviewer has to remember is forbidden. You are deciding whether to forbid the nonsense in the representation, or to catch it at the moments that matter

The call. Do you reshape the data so the bad state cannot be constructed, or leave the loose shape and guard against the bad state at the point of use?

Structure it out (type / discriminated union). Replace the loose representation with one whose nonsense combinations have no spelling: a discriminated union (loading | error | loaded(data)), or a narrow type that carries its own validity, so the malformed value cannot be built and the compiler refuses code that ignores a case. - Choose when: The illegal combination is genuinely impossible in the domain, not merely rare, so collapsing it loses nothing real; Many call sites read or build the value, and you cannot trust all of them (present and future) to remember the rule; The contradiction produces silent corruption rather than a loud crash, so catching it once for everyone is worth a refactor; You control the representation: it is your own type, not a shape a third party hands you. - Cost: A refactor up front: every existing construction and read site must move to the new shape before it pays off; The cure can metastasise, with phantom types and ever-narrower markers growing into machinery harder to read than the bug they prevent; Some invariants do not fit a sum type cleanly (cross-field arithmetic, ordering, sums that must total a constant), and forcing them in produces a contortion; A type can only carry facts known where the value is built; an invariant that depends on the outside world (a row still exists, a token has not expired) cannot live in the type.

Runtime guard at the point of use. Leave the loose shape and assert the invariant where it matters: an assert, a thrown error, or an early return in the function that depends on the value being sane. - Choose when: The invariant cannot be expressed in the type system you have, or expressing it would cost more clarity than it buys; Only one or a small, stable set of call sites actually depend on the property, so you can guard them all and keep them guarded; The value's shape is dictated by something you do not own (a wire format, a library type), and re-typing it everywhere is not on the table; The property depends on runtime facts the compiler cannot see, so a check at the moment of use is the only place it can be true. - Cost: The proof is thrown away the instant it is made: the fortieth caller, and every caller that does not exist yet, has to remember to re-check, and one of them will forget; Coverage is a claim, not a guarantee; the guard holds at thirty-nine of forty sites, and the fortieth is where it bites; The failure surfaces late, at the use site, often far from where the bad value was built, which lengthens every debugging session; A tired engineer must hold the rule in their head on every change, because nothing in the representation reminds them.

How to decide. Settle it on who can build the bad value and how far the damage travels. Tenet (III) is about shape, and shape is worth spending structure on exactly when the malformed value would otherwise be constructed by code you cannot police: many call sites, future call sites, or input you do not control. There the type pays for itself once and for everyone, and it shrinks what a tired engineer has to carry, because the compiler now carries it. Reach for the runtime guard when the property cannot be spelled in the type (it depends on the world, or on arithmetic across fields), or when the dependents are few and stable enough that you can guard them all and keep them guarded, or when the shape is handed to you by something you do not own. The real discriminator is whether the illegal state is genuinely impossible in the domain, in which case you collapse it into the representation, or merely undesirable under conditions the compiler cannot see, in which case you check it where it matters and accept that the proof does not persist. Spend the coupling of a refactor only where the invariant is real and a missed check would corrupt rather than crash; if violating it just throws loudly at one chokepoint, a guard at that chokepoint is the cheaper correct answer.

Reach for first. Before any refactor or scattered guards, see whether you can delete the illegal state rather than defend against it: collapse the three booleans into one field that can only hold the real cases (an enum, or a single nullable result), so there is nothing to check because there is nothing to get wrong. The smallest move that makes the bad state unbuildable, often a one-line change to the representation, beats both a cathedral of types and a scattering of asserts.

Pitfalls. - Modelling the data as independent booleans first and bolting guards on afterwards, when a single field holding only the real cases would have removed the question entirely. - Pushing the type cure past its worth: phantom-type markers and nested generics that encode invariants nobody reading the code can follow, trading a runtime bug for a permanent comprehension tax. - Reaching for a type to enforce a property that depends on runtime state (the record still exists, the lock is still held); the compiler cannot see it, so the guard was always the only honest option. - Guarding the value at use and assuming that settles it, then watching a new call site read the loose shape directly and skip the check, because nothing in the representation forced the question. - Throwing the proof away: validating at the boundary, handing back the same loose type, and making every downstream function re-ask whether the value is sane.

See also. Tenets (III). glossary: make the illegal state unrepresentable, discriminated union, phantom types. phases: implementing.

Enforce in code/structure vs in process (runbook/four-eyes)

You hit this when A rule has to hold (the irreversible migration gets a second reviewer; the orphaned flag gets deleted; the on-call knows what to do when the queue backs up), but the code structure cannot carry it. There is no type to write, no constraint to add, no gate the build can fail on. So the rule has to live somewhere a tired human will actually meet it

The call. When a rule cannot be made a property the system enforces, do you push it as far into code and structure as it will go, fall back to deliberate process (a runbook, an owner, four-eyes), or leave it to process that is really only a hope someone remembers?

Code/structure first. Make the rule a property the system enforces: a type that makes the illegal state unbuildable, a constraint the database rejects, a CI gate that fails the build, a required review check. The wrong thing cannot be expressed, so nobody has to remember it. - Choose when: The rule can be phrased as a fact about a value, a schema or a state transition, so a machine can decide it without judgement; The input or the action is controlled by an attacker or an unlucky caller, and the blast radius of getting it wrong is wide; The rule fires often enough that relying on vigilance guarantees it will eventually be skipped; You would rather pay once, in design, than pay on every change a tired engineer makes. - Cost: Up-front design effort, and sometimes a heavier type or an extra layer that every later reader has to understand; A gate that is too strict gets routed around, and a check nobody can explain decays into cargo cult; Some correctness will not compress into code: who to page, what to do about a known failure, whether an irreversible step is wise this time; Reaching too far couples unrelated things together in the name of one invariant, which is its own load on the reader.

Deliberate process (runbook, owner, four-eyes). When code structure has genuinely run out, carry the rule in designed process: a named owner of record, a runbook of known failure modes and first responses, four-eyes review required for the irreversible act. Process treated as a continuation of pushing correctness into structure, and built with the same rigour as code. - Choose when: The rule needs human judgement that no type can hold: is this migration safe to run now, is this the failure we feared; The act is irreversible and rare, so a second pair of eyes by policy is worth the friction; Something must be known before the incident rather than discovered during it: who owns this, what to do first; Observability can tell the on-call what broke, but only a written response tells them what to do about it. - Cost: Process is overhead, and ceremony on the cheap or reversible is friction that teams will quietly route around; It depends on people following it, so it degrades unless it is kept current and kept minimal; A runbook drifts out of date silently; nothing fails the build when it lies; Bureaucracy for its own sake wears away the trust it was meant to encode, so every step has to earn its keep.

Process as the silent default. No enforcing structure and no designed process: the rule survives only because someone is expected to remember it. A retro action with no owner and no date, a convention held in one senior engineer's head, a step done because people happen to be careful. - Choose when: Effectively never on purpose; this is what you land on by omission, not by choice; Tolerable only for the genuinely cheap and reversible, where forgetting costs little and is easy to undo; Defensible briefly when it is yours, the blast radius is contained, and you have written down why you deferred. - Cost: It looks identical to a rule that was never made; the day it is forgotten arrives without warning; It concentrates correctness in whoever happens to remember, and leaves with them when they go; The first incident is the test, and the test is taken live, with the pager going off; It violates the one thing the corpus is for: it loads the rule onto the tired engineer's memory and bounds nothing.

How to decide. Decide by who controls the input and how wide the blast radius is, then spend only where the property is real. If the thing the rule guards is controlled by an attacker or an unlucky caller and getting it wrong reaches far, the rule must not depend on memory at all: push it into code and structure so the wrong thing cannot be expressed, and pay the design cost now. If the rule needs judgement that no type can hold, or guards an act that is irreversible and rare, code structure has honestly run out, and deliberate process carries the load because nothing cheaper can. Build that process with the same rigour you would give code: a named owner, a date, a tracked response, so it is real structure rather than ceremony. The silent default, a hope that someone remembers, is the one answer the whole corpus rules out, because it loads the rule onto exactly the tired 2am engineer the objective exists to protect and bounds nothing. The test for any process you keep is whether it carries load through someone on the hook by a known time. If it does not, you have not solved the problem; you have only written it down.

Reach for first. Try to make the rule unnecessary or unbuildable before you write a single line of process. Most rules that feel like they need a runbook are really a missing constraint: a type that makes the illegal state unrepresentable, a unique index, a required CI check, a flag with a built-in expiry. Process is the fallback for what is left once code structure has genuinely run out, not the first reach.

Pitfalls. - Reaching for a runbook when a constraint would do: a rule that could have been a unique index or a required check instead becomes a paragraph nobody rereads. - Calling a slide of good intentions a process: an owner-less, date-less retro action is process in name only and changes nothing. - Heavy ceremony on cheap, reversible acts, which trains the team to route around all of it, including the four-eyes that mattered. - Letting a runbook rot: it is trusted at 3am precisely because it is assumed current, so a stale one is worse than none. - Confusing a capability the code enables (the decision/effect seam makes four-eyes possible) with the practice itself (four-eyes only happens if a policy requires it). - Process with no clock and no one on the hook, which is indistinguishable from a rule that was never made.

See also. Tenets (XXV). glossary: process is structure when code structure runs out, process with no structure behind it. phases: operating, planning.

Name to reveal the load-bearing fact vs name to label

You hit this when You are mid-flow, about to commit a token a hundred people will read: a function, a variable, an issue title, a branch slug. The thing is clear in your head right now, so any name feels good enough. The reader who meets it next week, cold, with a pager going off, has only the name. You are choosing what that reader is told before they read a line of the body

The call. Should this name encode the one fact a reader gets wrong without it, or just gesture at what the thing is?

Name to reveal the load-bearing fact. Spend the name on the single fact whose absence makes a reader wrong: the unit (timeoutMillis), the ordering guarantee (sortedByName), the side effect, the risk the issue actually carries. The reader can skip the body because the name already carried the thing the body would have told them. You pick the one fact the type cannot hold and stop there. - Choose when: The thing has a caveat a reader will trip on: a unit, a precondition, an ordering rule, a side effect, the failure mode an issue threatens; It is the most-read token in its scope: a public function, a long-lived variable, an issue title, a branch a team shares; The type system cannot already carry the fact, so the name is the only place the reader meets it before the body. - Cost: It takes a minute to work out which fact is load-bearing, and you have to think rather than reach for the first noun; A name that encodes a fact is a second copy of that fact, so when the unit or the guarantee changes you must rename everywhere or the name starts lying; Pick the wrong fact and you have spent the name's budget on noise while the real caveat stays invisible.

Name to label. A short gesture at the category of the thing: handle, data, doIt, fix reconnect. Cheap to write, cheap to read, and honest about saying little. It tells the reader which bucket the thing is in and leaves the rest to the type signature and the surrounding context. - Choose when: The thing is genuinely generic and carries no caveat: a loop counter, a local with a two-line lifespan, a private throwaway; Locality already carries the fact: the call site, the type signature, or three lines above make the caveat obvious without help; Any longer name would just restate what the type or the immediate context already says plainly. - Cost: On anything with a caveat, the label withholds the one fact the reader needed and sends them into the body to reconstruct it; Labels accrete: a codebase of data and handle forces every reader to read every body, which is the cost the name was meant to spare them; The line between safe-generic and lazily-vague is easy to misjudge under time pressure, and the reader downstream pays for the misjudgement, not you.

Encode everything. Cram every detail into the identifier: getUserByIdWithRetryAndCacheFromPrimaryReplica, or an issue slug that tries to hold the whole investigation. It looks maximally informative and feels like diligence, since you have left nothing out. - Choose when: Almost never as a deliberate choice; it tends to arrive by accretion as facts get bolted on one at a time; Defensible only at a true API boundary where two genuinely load-bearing facts both have to live in the name and no type can hold either. - Cost: A name that reveals everything reveals nothing useful: the reader cannot find the one fact that matters inside the pile; Every encoded detail is another copy that can drift, so the name has more ways to go stale than a short one; It reads as length, not precision, and trains readers to skim names rather than trust them, which quietly undoes the point of naming to reveal.

How to decide. Ask what a cold reader gets wrong if this name says nothing, then decide by how widely that wrongness spreads. A shared issue title, a public function, a branch a team builds on are read by many and corrected by few: the person who meets the name does not control what it says, so the misread travels far before anyone fixes it. The wider that reach, the more the name should reveal, and the rename cost when the fact changes is the fair price of having spent the name well. A loop counter or a two-line local has a blast radius of one screen, and locality already carries the caveat, so a label is the cheap, correct move; a longer name there is just a copy waiting to drift. At every width the thing to refuse is the misleading name: a wrong name installs a false model the reader trusts and debugs against for an hour, which costs far more than the silence of a vague one. Refuse encode-everything from the other end, since it buries the load-bearing fact under facts that are not, and the reader cannot find what would have saved them. Reveal the one fact whose misread travels furthest; let locality carry the rest.

Reach for first. Before you type the name, finish the sentence: "a reader gets this wrong if they do not know ___." Whatever fills the blank is the load-bearing fact, and that is what the name encodes. If nothing honestly fills it, the thing is generic and a short label is correct. If two things fill it, the type should carry one of them, not the name.

Pitfalls. - Naming the implementation instead of the contract: getDataFromCacheOrDb tells the reader how you did it, not the one fact (a unit, an ordering, a side effect) they need to use it correctly. - Encoding a fact the type already carries, so the name is pure duplication that drifts the day the type changes and the name does not. - Letting a name assert an invariant the code stopped honouring: chargeOnceIdempotent after idempotency was dropped is disinformation, worse than a plain name. - Treating issue titles and branch slugs as throwaway: they are the most-read line of the ticket, so fix reconnect hides the risk that unbounded retry will storm a flaky agent. - Mistaking length for precision and shipping getUserByIdWithRetryAndCacheFromPrimaryReplica, where the reader cannot tell which of the five facts is the one that matters. - Defending a vague label as minimalism when the thing genuinely has a caveat: brevity stops being a virtue the moment it withholds the load-bearing fact.

Composition & wholes

Verify disjointness / cells vs assume independent

You hit this when Two services run on different hosts, in different languages, owned by different teams, and you are about to treat them as independent: redundant copies, separate failure domains, one able to cover for the other. The architecture diagram draws them in separate boxes, so they look separate. Whether they actually fail separately is a question nobody has asked

The call. Do you derive the dependency graph and confirm the pools, zones and secrets two components share are genuinely disjoint before you bank on their independence, or do you assume it from the fact that they are deployed apart?

Verify it: derive the graph, confirm disjointness, isolate into cells. Generate the dependency graph from traces, the service mesh and the infrastructure-as-code, then read off what the two components actually share: the same DNS resolver, config store, identity provider, certificate, availability zone, or a deploy that touches both. Where the joint failure would be catastrophic, make the shared thing genuinely separate, so each cell carries its own pools, zones and secrets and a failure stays inside the one cell where it happened. - Choose when: The joint failure is catastrophic: the two components are your redundancy, or your failover, or the live path and its fallback, so if they go down together there is nothing left to catch the request; The shared thing is exactly the kind that withdraws itself under stress, a control plane or DNS or identity provider whose outage also removes the tools you would use to recover; You can derive the graph rather than draw it by hand, because traces, mesh and IaC exist to generate it from, and it can be given an owner and an expiry so it does not rot into a confident lie; A cyclic dependency is plausible: a boot or recovery order that closes a loop, A needs B needs C needs A, and deadlocks the moment everything starts cold. - Cost: Deriving the graph is real work, and the graph is itself state. Drawn once and left to rot, it reassures you in precisely the incident it was meant to illuminate, so it needs a generator, an owner and an expiry, none of them free; Genuine disjointness is expensive at the bottom: a second control plane, separate identity, per-cell secrets and quotas, duplicated config, all of it to buy separation you may rarely cash in; Cell isolation caps the blast radius but also caps efficiency, since pooling across the whole fleet is cheaper than carving it into independent slices that cannot lend each other capacity; You cannot map everything, so the honest version maps the shared substrate whose joint failure is catastrophic, defers the rest, and writes down which ones it deferred. That is a standing discipline, not a one-off.

Assume it: treat deployed-apart as failing-apart. Take separate hosts, languages and teams as evidence of independence and move on. The redundancy is counted, the failover is on the diagram, and nobody checks whether the two sides lean on one DNS, one config push, one certificate expiry or one zone underneath. - Choose when: The components are genuinely on disjoint substrates and you already know it, because the separation was engineered and recorded rather than hoped for; The coupling is cheap and visible: a small system where the shared dependencies fit in one head and a glance settles the question without a derived graph; The joint-failure cost is low, so if both do go down together you lose a feature rather than the system, and the request has somewhere else to land; You are early, the substrate is still moving, and the cost of formalising a graph now exceeds what a wrong assumption could cost at this size. - Cost: Common-mode failure is the bill, and it arrives all at once: three redundant systems that lean on the same thing are worth one, and you learn the multiplier was a lie at the worst possible moment; The shared substrate is invisible by construction and never written down, so no single component's own view can show you the joint cause; only the graph you did not draw would have; A cyclic dependency can run for years unseen, because in daylight the caches are warm and the loop never closes, then it deadlocks on the one cold boot when you can least afford it; Nominal isolation you booked as real becomes a single point of failure dressed up as redundancy, quiet in steady state and loud in the incident.

How to decide. Decide by who controls the trigger and how wide the blast radius is when the shared thing fails. The governing question for the whole corpus is what a tired engineer must hold in their head to make a correct change, bounded by the blast radius of anything an attacker or an unlucky caller controls; here the trap is that the substrate is invisible, so the tired engineer holds a false belief, that these fail separately, which no single component's view will ever correct. So spend the verification where the joint failure is catastrophic and the shared thing is one you do not control: a control plane, DNS, identity, a zone, a config push that fans out to both sides at once. There, derive the graph, confirm the pools, zones and secrets are genuinely disjoint, and isolate into cells, because the cost of the coupling the cure drags in (a second control plane, per-cell secrets, lost pooling efficiency) is smaller than the cost of discovering at 3am that your redundancy was one machine wearing two hats. Where the joint-failure cost is low, or the substrate is small enough that disjointness fits in a glance and is already recorded, do not build the machine: a derived graph that nobody owns is just more stale state lying confidently. The line is not assume-everywhere against verify-everywhere. It is this: map the shared substrate whose joint failure would take down everything you are treating as independent, write down what you deferred, and never let a box on a diagram stand in for a checked fact.

Reach for first. Ask the one question for the one dependency that matters: what single thing, if it died this second, would take down everything you are treating as independent? If you cannot name it, that is your answer, go and derive the graph. If you can name it, and it is genuinely disjoint, and you have evidence, you are done without building anything. Verify the catastrophic edge first and defer the rest on the record.

Pitfalls. - Drawing the graph once by hand and leaving it to rot, so it reassures you in exactly the incident it was drawn to illuminate; derive it from traces, mesh and IaC, and give it an owner and an expiry. - Counting redundancy without checking shared fate: three replicas behind one control plane, one config store or one identity provider are worth one, and the diagram will not tell you. - Treating separate boxes on an architecture diagram as separate failure domains, when the bulkhead is real only if the pools, zones and secrets are actually disjoint and not merely drawn apart. - Missing the cycle because it is harmless in steady state: A needs B needs C needs A deadlocks only on a cold boot with empty caches, so it runs invisibly for years until the day it cannot. - Verifying the substrate but ignoring the recovery path, when the worst shared dependency is the one that withdraws the very tools (DNS, identity, the status page) you would use to undo its own outage.

Defend your part from the whole vs bound your part's effect on the whole

You hit this when You are wiring a new service, screen, or job into a system that is already running and already has bad days. You can harden the seam against what the rest of the system does to you, or you can constrain what your part does to the rest of the system. These are two different pieces of work, and skipping one leaves you exposed in the opposite direction from skipping the other

The call. When you connect a part to the whole, do you spend your effort defending the part from the system's failures (inward), bounding the part's effect on the system (outward), or both, on purpose.

Inward: protect your part from the whole's bad days. Treat every call across the boundary as a wait that can hang, an answer that can be wrong, and a dependency that can be down. You add deadlines, breakers, a fallback, and a degraded mode so a sick upstream costs you a feature rather than the whole page. - Choose when: You depend on something you do not control: a third party, another team's service, a shared store you cannot make faster; A timeout, a stale read, or a 503 from upstream would otherwise surface to your user as a hang or a blank screen; You can name, ahead of time, the smaller thing you would serve when the dependency is gone (a cached value, last-known-good, a reduced answer); The failure is theirs to cause but yours to absorb, because you sit between them and the user. - Cost: Every fallback is a second code path, and an untested degraded path is just a second bug waiting for the worst moment to fire; Breakers and deadlines need tuning to the actual latency distribution; set them by guess and you shed healthy traffic or trip too late to help; A fallback can mask the upstream's decline so well that you stop noticing it is dying until it has died completely; Defending yourself does nothing to stop you being the one who knocks the system over.

Outward: bound what your part inflicts on the whole. Constrain the load, retries, and coupling your part pushes outward, so that your bad day does not become the system's. Cap your retry budget with backoff and jitter, avoid becoming the shared dependency everyone synchronises on, and keep your demand on any shared pool bounded. - Choose when: Your part can generate load others must absorb: it fans out, retries, or polls a shared downstream; You would otherwise quietly become a common dependency, so your wobble lines up everyone's failure through one substrate; Your retries on a struggling upstream are the load that keeps it struggling, the textbook self-sustaining storm; You hold or contend for a shared resource (a pool, a hot key, a lock) that other tenants also need. - Cost: A retry budget and jitter mean some of your own requests fail that a more aggressive client would have got through; you trade your tail latency for the system's stability; Refusing to be the shared dependency often means duplicating or caching what you could have just called, which is real coupling and real staleness to manage; Capping your own fan-out can make your part slower or less complete on a good day to keep it survivable on a bad one; It protects everyone but you: a well-behaved part can still be flattened by a system that does not return the courtesy.

Both, deliberately, as separate work. Do inward and outward bounding as two distinct passes over the same seam, because they defend against opposite failures and neither implies the other. You make yourself resilient to upstream death, you make yourself harmless to downstream health, and you write down which property each piece of code is buying. - Choose when: The seam is load-bearing: it sits on the responsive path, or it is a place where a failure would spread; You are both a consumer of something shared and a producer of load onto something shared, which describes most services in the middle of a graph; The blast radius of getting it wrong is wide enough that one direction of bounding is not enough; You can afford to build and, crucially, rehearse both paths rather than ship them untested. - Cost: The most machinery and the most state to hold: two sets of paths, both of which must be exercised or they rot into liabilities; Easy to over-spend, hardening a seam whose worst failure is a single retry, paying the second-term cost where the property is not real; Doubles the tuning and observability burden: you now have a breaker, a budget, fallbacks, and quotas all to keep honest; If the two directions are tangled into one mass of resilience code, locality of reasoning suffers and the next engineer cannot tell what each guard is for.

How to decide. Decide by who controls the input and how far the damage travels, then spend only where the property is real. Inward bounding earns its keep wherever the thing on the other side of the seam is outside your control and its failure would otherwise reach your user; the deadline, breaker, and fallback are the price of keeping a sick upstream from becoming your outage. Outward bounding earns its keep wherever your part can put load or contention onto something shared, because that is where you can quietly widen the blast radius for everyone behind the same substrate; the retry budget, jitter, and refusal to become the common dependency are the price of not being the trigger. The two are not the same work, and one does not imply the other. A service can be perfectly defended and still be the retry storm that flattens the cluster; a perfectly polite client can still white-screen because it never built a fallback. So ask the question in both directions at every load-bearing seam, and at each one spend the coordination and machinery only where violating the property would cost more than the coupling the cure drags in. A leaf with one caller and a trivial downstream needs almost none of this. A service in the middle of the graph, consuming shared infrastructure and producing load onto more of it, needs both, built and rehearsed, with each guard labelled so the next tired engineer can see what it defends.

Reach for first. Before any breaker or budget, put a deadline on every wait across the boundary and a sane cap on your own retries. A bounded wait plus capped, jittered retries is the cheapest correct answer that bounds you in both directions at once: it stops an upstream hang from reaching your user, and it stops your retries from becoming the load that keeps the upstream down. Reach for fallbacks, bulkheads, and degraded modes only once you have measured that the bare deadline is not enough.

Pitfalls. - Treating resilience as one undifferentiated heap of code, so nobody can say which guard protects you from the whole and which protects the whole from you. - Building inward defences only, then being the part that takes the system down, defended right up to the moment you trigger the outage. - Building outward politeness only, then white-screening your own users the first time a dependency you were polite to falls over. - Shipping a fallback or degraded path and never running it, so it is a second bug that fires for the first time in the incident it was meant to survive. - Tuning a breaker or budget by guess instead of by the measured latency distribution, so it sheds healthy traffic or trips far too late. - Becoming the shared dependency without noticing, so your routine wobble lines up everyone else's failure through the one substrate you all touch.

Triage & investigation

Reproduce first vs act on the report or from memory

You hit this when An issue has landed on you: a bug report, an incident page, a fix someone wants verified, an outage that needs explaining. In front of you sits an account of what happened, written by a human who saw a symptom and guessed at its shape. You can work from that account, or from your own memory of a similar failure, and move now. Or you can spend first on pinning the thing to a sequence that makes it fire on demand, and rank, diagnose and verify against that instead. The account is cheaper to act on and almost always wrong in some load-bearing detail

The call. Do you reproduce the issue deterministically before you rank, diagnose or verify it, or do you act on the report or your memory and proceed?

Reproduce deterministically first. Before you size, diagnose or sign anything off, pin the failure to a fixed sequence: the inputs, the seed, the clock, the data fixture, the environment. Strip variables until the symptom appears and disappears when you say so. The repro becomes the yardstick everything downstream measures against, and it travels with the issue to the next phase. - Choose when: The blast radius is wide: data corruption, an attacker-reachable path, anything that compounds or outlives the fix; You are about to rank one issue above proven work, or hand a diagnosis on, or verify a fix; each of those needs a fixed thing to measure against; The account is the only evidence and it crosses a trust boundary, so its framing and severity are unvalidated input; A later phase will inherit this and will pay again if it arrives as prose nobody can replay. - Cost: Pinning the inputs can eat the whole time-box before you have touched the actual problem, and some failures are irreducibly statistical and never go fully deterministic; A genuinely rare, genuinely serious bug can resist reproduction, and ‘reproduce first’ can quietly become the bin it goes to die in; On a trivial, contained, obvious fix the ceremony of a repro costs more than the fix it guards; Effort spent summoning the failure is effort not spent on the queue behind it.

Act on the report or from memory. Trust the account in front of you, or your recollection of the same failure last quarter, and proceed: rank it, start diagnosing the cause it points at, or accept the fix as working. No deterministic repro is built first. - Choose when: There is a fire and the signals plainly show it: error rate climbing, a clear first-seen timestamp, real users hit now, so acting beats sitting there asking for repro steps; The fix is small, reversible and low blast radius, and formally pinning it costs more than landing it; The account is a good lead even when it is a poor fact: ‘it started Tuesday’ is wrong as a timestamp and gold as a hint about which deploy to bisect; You can run the cheap validation in parallel with raising the alarm rather than after it. - Cost: You are ranking a feeling: you cannot tell a fix from a coincidence, so a flake passes itself off as a result and you trust the dice that agreed with you; Memory drifts to whoever is most confident, and confidence is not correctness; you debug the bug you remember, not the one you have; A screenshot proves something happened once, not what triggered it or how wide it goes, so you size scope and severity by guesswork; When the symptom goes quiet you cannot prove your change did it, so the cause returns next quarter wearing a new face.

Mark unconfirmed and rank below proven work. You cannot reproduce it now and the cost of chasing it does not yet earn a slot, so you label the issue unconfirmed, record what you have, and rank it beneath anything you can actually demonstrate. It waits for a repro, more reports, or signal before it moves up. - Choose when: The reach is plausibly narrow and there is real, proven work the queue should clear first; You have no on-demand repro and no telemetry that bounds the scope, so any rank you assign now is invented; You would rather state the issue's true status honestly than promote a rumour with a screenshot above demonstrated work; The disposition is reversible: new evidence can lift it the moment it arrives. - Cost: ‘Unconfirmed’ can become a quiet downgrade that buries a serious-but-rare bug under cosmetic noise; If nobody ever revisits the label it rots in place, neither alive enough to fix nor dead enough to close; You are deferring the cost of understanding, not removing it; the issue keeps its full blast radius while it waits; Repeated marking-down trains reporters that real intermittent failures are not worth filing.

Bounded investigation to get the repro. The claim is unconfirmed but its blast radius is high, so instead of downgrading it you open a time-boxed dig whose one job is to make the failure deterministic. You name the question, put a clock on it, and decide in advance what you do when the box closes still open: escalate, ship a workaround, accept the risk. - Choose when: The unconfirmed claim is high blast radius (corruption, a security hole, silent data loss) where being wrong is expensive and irreversible; An intermittent failure looks reducible: a concurrency, an ordering, a clock, a cold cache you can narrow to; Acting on a guess here would be worse than spending bounded effort to turn the guess into a fact; You need the repro not just to fix but to verify any candidate fix actually moved the needle. - Cost: The clock can cut you off one step short of the answer, and a too-tight box rewards the plausible guess over the proven one; On a low-radius issue this is over-spend: you are buying determinism the problem did not warrant; If the box closes with the failure still statistical you must fall back to measuring a rate, which is more work than a binary repro; The dig owes a result, so it competes for the same attention as the fire it was meant to clarify.

How to decide. Decide by who controls the input and how wide the blast radius is, not by how loudly the issue was reported or how vivid your memory of last time feels. The account in front of you, and your own recollection, are both untrusted input: they tell a tired engineer a story, and acting on a story means holding a guessed cause and a guessed scope in your head while you work. The repro pushes that out of memory and into a fixed artefact, which is the whole point of the move, but it is not free and it is not always warranted. So size the spend to the radius of being wrong. A wide or irreversible blast radius (corruption, an attacker-reachable path, a payments figure) earns a deterministic repro before you rank, diagnose or verify, and if you cannot get one cheaply it earns a bounded investigation to find one, never a quiet downgrade. A narrow, reversible, plainly-visible fire earns action now on the signals, with the cheap validation running alongside the alarm. The line to hold: nothing gets ranked above proven work, handed on as a diagnosis, or signed off as verified on the strength of an account alone, because a fix you cannot reproduce failing is a fix you cannot prove working.

Reach for first. Spend two minutes trying to make it fire on demand from the account you already have. If it reproduces, you now hold the cheapest correct thing: a yardstick to rank, diagnose and verify against. If it does not, you have learned the issue is currently a rumour, so mark it unconfirmed and let the blast radius decide whether it earns a bounded dig or a slot below proven work.

Pitfalls. - Treating an intermittent reproduction as a smaller version of a solid one. It is a different, harder problem: every hypothesis you test against it inherits the randomness, so reproducing by seance gives you findings you cannot trust. - Ranking severity off the reporter's asserted ‘P0’ or your gut, when the telemetry you shipped can tell you the real rate, first-seen and affected count for free. - Calling a fix verified because the symptom did not recur this once. Without a repro that failed first, the green is luck, not proof, and it protects no one after the release ships. - Running a retro from memory, where the account drifts to whoever is most confident; without a repro, a log or a trace to point at, the loudest hypothesis wins and the team learns the wrong lesson. - Letting ‘reproduce first’ become a stall on a genuine fire, or letting ‘act now’ become the default that never pins anything; both ignore the blast radius that should have decided it. - Accepting noise dressed as a finding from a flaky repro and building a diagnosis on top of it, so the next person re-derives the whole dig as folklore.

Rank by blast radius and evidence vs by the reporter's volume

You hit this when Several issues compete for the same scarce attention and someone is escalating that theirs is a P0. You are ordering the queue, not yet fixing anything, and the order you set is what the next phase inherits and trusts

The call. When reports compete for the same slot, what sets their order: who and what each one can harm and how widely, or how loudly and how senior the reporter is?

Rank by blast radius and the signals you shipped. Order the queue by who and what each issue can harm and how far the damage spreads, read off the telemetry, traces, and error rates rather than off the report's framing. An attacker-reachable path or a silent data-corruptor outranks a cosmetic flaw whatever the thread length. Where you cannot measure the scope, you file the blind spot as its own issue rather than invent a number for the priority field. - Choose when: The issues differ in kind: one touches caller- or attacker-controlled input or persistent data, another only annoys; You have observability that can answer how many are hit, how often, and since when; The decision is hard to reverse cheaply later, so getting the order wrong means a real harm compounds while you look at a louder, smaller one; Business reach is real and can be folded into the radius honestly, as affected-and-how-wide, not smuggled in as decibels. - Cost: Reading traces and error rates is slower than reading a Slack message, and the dashboards may lag or the query may be awkward; It can feel dismissive to someone in genuine, loud pain whose issue is narrow; you owe them the radius reasoning, not silence; A scope you genuinely cannot measure forces an explicit 'unknown, gap filed', which is honest but unsatisfying under pressure; Blast radius can be stretched to rationalise any pet ordering once you stop keeping affected-and-how-wide concrete.

Rank by the reporter's asserted severity and volume. Take the priority the reporter typed and the heat of the thread as the ordering signal: the longest thread, the most senior name, the loudest 'this is critical' goes first. The queue reorders itself around whoever is shouting most. - Choose when: You have no observability at all and the human account is the only signal you hold, so volume is a crude proxy for reach; The organisation is small enough that the loud reporter genuinely is the affected population; You are deliberately optimising for a single relationship, a flagship account or demo, and have said so out loud; Speed of acknowledgement matters more than correctness of order for this short window, and you will re-rank once the signals arrive. - Cost: Volume measures attention, not harm: the silent corruptor that nobody noticed yet sits below the cosmetic flaw everyone can see; Severity is a claim from an untrusted source (IV); the field is inflated precisely because inflating it works; Seniority and decibels reorder the queue for reasons unrelated to who gets hurt, and the pattern teaches everyone to shout; An unreproduced report is a rumour with a screenshot; ranked on volume it can outrank proven, demonstrated work.

Rank by cheapness-to-fix, quick wins first. Order by effort: the one-line, obvious fixes go to the top so the queue count drops fast and visible progress accrues, regardless of what each issue threatens. - Choose when: The backlog is genuinely all low-radius and roughly equal in harm, so effort is a fair tiebreak; A quick win is also a high-radius fix, since cheap and important is the best slot in the queue, not a conflict; You are clearing froth ahead of a focused session and want the trivia out of the way before the real ranking; Morale or a stakeholder needs a visible burn-down and nothing dangerous is being deferred to get it. - Cost: Effort is uncorrelated with harm: the expensive, dangerous corruption bug sinks under a raft of cheap cosmetics; A falling ticket count looks like progress while the one issue that can actually hurt you ages untouched; It rewards issues for being shallow, which is the wrong thing to reward in a triage queue; The hard, high-radius problem only gets harder and wider the longer the quick wins keep jumping the line.

How to decide. Settle it on who controls the input and how wide the blast radius is, not on the volume in the channel. Walk each competing issue back to two questions: can a hostile or unlucky caller reach it, and how far does the damage spread before something stops it. The one that scores worst on those takes the slot, even when a louder, narrower issue is being escalated harder. Volume is a lead, not a rank: mine the loud thread for where to point your verification, then let the telemetry and the radius decide the order. Spend the measurement only where it can change the order, so a corruption or attacker-reachable claim earns the trace read while a contained cosmetic does not. And state the losing case as fairly as the winning one: when you genuinely have no signals and a small enough population, the reporter's volume is a crude proxy for reach and may be all you have, but the moment a signal exists it overrides the decibels.

Reach for first. Before ranking anything, re-derive each competing issue's real reach from the evidence: pull the error rate and first-seen for the alleged P0, confirm a row actually vanished rather than a stale cache, check the affected-tenant count. That cheap sanity check, minutes not a forensic audit, usually collapses the contest on its own, because the loudest report and the widest blast radius are rarely the same issue.

Pitfalls. - Accepting the priority field as fact: 'critical' is an unvalidated claim from an untrusted source (IV), and the field is inflated because inflating it gets results. - Letting seniority or thread length reorder the queue, which trains the whole organisation that shouting is how you get triaged. - Treating 'blast radius' as a slogan you can stretch to justify any pet ordering; keep it concrete as affected-and-how-wide or it stops meaning anything. - Ranking an unreproduced report above proven work because it arrived with a screenshot and a confident tone. - Inventing a scope number to fill the priority field when the signal is missing, instead of filing the observability gap as its own issue. - Ignoring genuine loud, narrow pain in the name of radius; a cosmetic flaw that blocks a flagship demo carries real business reach, so fold it in honestly rather than dismissing the person.

See also. glossary: a rumour with a screenshot, blast radius, choose by blast radius. phases: triage.

Dispose of an issue explicitly vs leave it in the backlog

You hit this when An issue is leaving the triage gate. Someone has looked at it, and now it must be given a state, or quietly left without one

The call. When an issue clears triage, do you record a disposition with its reason, or leave it open and unranked for later?

Explicit disposition with a recorded reason. You commit the issue to one of a small set of named states, fix-now, schedule, needs-a-spike or won't-fix, and write the reason next to it. The state says what happens next; the reason is what lets a future reader reverse the call when the evidence changes. The decision is not final, it is recorded, which is what makes it cheap to revisit. - Choose when: The issue has reached the gate at all; this is the default exit for every triaged item; You want a later reader to reopen the call from the recorded reason rather than re-argue it from nothing; The queue is shared, so an unstated state silts up everyone's next pass, not just yours; You can name the next move and who owns it, even if that move is "close for now". - Cost: Deciding under uncertainty has a real price: a forced disposition can fossilise a wrong call that nobody revisits; Writing the reason is work, and a thin reason ("low priority") is barely better than none; A wrong won't-fix on a slow-burning issue buries it under a closed status until the symptom returns louder; The named states are a small vocabulary; an issue that fits none of them tempts a lazy mislabel.

needs-a-spike, the unknown named. A specific kind of explicit disposition for when you genuinely lack the information to decide. You do not guess at fix-now or won't-fix; you record that the call is blocked on a named open question and hand off the investigation. This is a decision, not a deferral: the decision is that the next move is to learn the missing fact. - Choose when: You cannot honestly rank or size the issue because a load-bearing fact is unknown; Forcing fix-now or won't-fix here would be inventing certainty you do not have; The unknown is nameable, so the spike has a question to answer and a way to finish; You would otherwise be tempted to leave it open in limbo because no other state feels true. - Cost: A spike is real work with an owner and a time-box; it is not free deferral dressed as diligence; "needs-a-spike" with a vague question becomes the ambiguous "investigating" that never resolves; Naming the unknown well is harder than it looks, and a sloppy question yields a sloppy spike; It can become a polite parking bay if no owner or budget is attached.

won't-fix, closed with a reason, reopenable. You decide the issue does not earn work now and close it, with the reason recorded so it can be reopened on new evidence. The closure is a state, not a verdict: it clears the queue while leaving a trail that a future reader can act on cold. - Choose when: The blast radius is genuinely small and the cost of fixing exceeds the cost of living with it; You can state plainly why, so a reopener inherits the reasoning rather than your mood; Leaving it open would commit nobody and just age into a guilty maybe; The issue is real but out of scope, and an honest close beats an indefinite hold. - Cost: A closed issue is easy to forget, so if the world changes nobody is watching for the trigger to reopen; A defensive or thin reason makes the closure unreopenable in practice, which is worse than leaving it open; Closing a contained-but-recurring annoyance can quietly accumulate user pain you stop seeing; It reads as dismissal to a reporter if the reason does not engage with what they saw.

Silent we-will-get-to-it. You leave the issue open and unranked. No state, no reason, no owner of the next move. It sits in the backlog as an unspoken intention to deal with it eventually. - Choose when: Almost never, as a deliberate choice; it is the absence of a decision rather than a decision; Arguably tolerable only for a brief, bounded window mid-triage before you actually dispose of it. - Cost: It is a zombie ticket that rots: not alive enough to fix, not dead enough to forget, silting up every future triage pass; It is tenet XIII's silent swallow applied to a decision, so the non-decision hides as if it were one; Nobody owns the next move, so each triage re-reads the ticket and re-defers it at a recurring cost; The reason that would let someone act was never written, so reviving it means re-arguing from scratch.

How to decide. The tie-break is who controls what happens next and how much a future reader must reconstruct to act. An explicit disposition with a reason is the cheapest correct exit because it moves the cost off the next tired triager, who reads a state and a reason rather than a wall of unranked tickets to re-feel one by one. The honest exception is real uncertainty, and the answer there is still an explicit state, needs-a-spike with the unknown named, not silence. Cost the loser fairly: forcing a disposition under genuine ignorance can fossilise a wrong call, and a thin reason is barely better than none. But the silent open ticket is worse on the axes that matter here, because it commits nobody and records nothing while taxing every pass that follows. So decide explicitly, and scale the ceremony to the blast radius: a corruption or attacker-reachable issue earns a sharper reason and a watched reopen-trigger than a cosmetic one. Reach for needs-a-spike the moment you notice you are inventing certainty to fill the field.

Reach for first. Record one of the four states and a one-line reason that a stranger could reopen on. If you cannot pick a state honestly, that itself is the signal to write needs-a-spike with the open question named, not to leave it blank.

Pitfalls. - "Investigating" as a permanent state: an ambiguous holding pattern that is neither a spike with a question nor a real disposition, so it never resolves. - A reason that records your verdict but not your evidence ("low priority", "out of scope"), so a future reader cannot tell whether the world has since changed. - won't-fix used as a tidy-up to clear the board, with a reason too thin to reopen on, which buries a real issue under a closed status. - needs-a-spike with no owner, no time-box and no named question, so it is silent deferral wearing a decision's clothes. - Treating the disposition as final rather than reversible, so nobody revisits a fix-now that has gone stale or a won't-fix whose trigger has fired. - Trying to encode the whole investigation in the disposition reason; the state names the next move, it is not the place for the full diagnosis.

See also. glossary: a zombie ticket that rots in the queue, encode the whole investigation in a slug. phases: triage.

One named owner with a due date vs a shared queue or intention list

You hit this when A triaged issue, or a retrospective follow-up, has been decided and now needs someone accountable for its next move. The work exists. What it lacks is a person and a clock. You are choosing the shape of that accountability before the room empties and attention moves on

The call. When a piece of work leaves triage or a retro, who is on the hook for its next move, and by when?

One owner of record plus a date. A single named person, or a real rota with a name resolvable today, is accountable for the next move, with a due date attached. The owner is accountable for the move, not necessarily the one who does the keystrokes; the move can be reassignment. The date is a clock the work runs against, not a guess at effort. - Choose when: The issue has a blast radius worth tracking: it can corrupt data, it sits on a security path, or it leaves a service in a state nobody dares touch later; It is a retrospective follow-up meant to close a class of failure, where an unowned item is the one that silts up until the same incident reopens it; Recurrence is the failure you are guarding against, and you need to be able to ask in six months whether the work shipped or just the intention; The next move needs a decision a specific person is trusted to make, not a task anyone can pick off a pile. - Cost: Naming one owner risks a bottleneck if that person goes away and the item stalls behind them; the owner must be accountable for the move, including handing it on, not a single point of failure; A date you cannot honour trains everyone to ignore dates, so a fake clock is worse than an honest none; Twenty owned, dated items with no slack ships none of them; the ceremony only pays where the few that matter get real owners and the rest are explicitly dropped; Owner-and-date applied to a cheap, reversible, low-radius chore is friction people quietly route around, and routed-around process stops meaning anything where it counts.

A team alias or shared queue. The work lands on a team, a label, or a backlog anyone could pull from. Accountability is collective; the next move belongs to whoever picks it up. The queue is the structure, and the assumption is that capacity finds the work rather than the work finding a person. - Choose when: The work is genuinely fungible: any qualified hand does it equally well and picking it up costs nothing in lost context; A real pull discipline exists, with a person who owns the queue’s health and a service-level expectation on how long anything sits; Throughput across interchangeable items matters more than any single item’s fate, and the cost of one ageing in the pile is low and bounded; Load-balancing across a team is the point, and forcing a single name on day one would just bottleneck on that person. - Cost: Without an owner of the queue itself, ‘anyone could’ decays into ‘nobody did’, and the item ages unclaimed because everyone assumes someone else has it; High-radius work hidden in a shared pile is exactly the unowned service that nobody dares touch during the outage, when who-owns-this becomes the first and worst question; No clock runs by default, so a serious-but-unglamorous item slips behind every shiny new pull indefinitely; Collective accountability tends to dissolve under pressure: the queue absorbs the blame, no one person is answerable, and the gap stays open.

An intention list with no owner and no date. A line in the minutes or a slide bullet: ‘we should add a test’, ‘we’ll keep an eye on it’. No name, no clock. It records that the room agreed something was a good idea and commits no one to anything. It evaporates the moment the meeting ends. - Choose when: Almost never as a tracked output; legitimate only as raw capture you will triage into owned work before the session closes; A deliberately-dropped item, recorded once with the reason, so a future reader knows it was considered and declined rather than forgotten; An idea genuinely below the bar for any work, parked transparently as ‘not doing this, here is why’, which is itself an explicit decision. - Cost: A sentence with no one on the hook and no clock running looks identical to a follow-up that was never made; both produce nothing; It reads as ‘handled’ to everyone who was not in the room, so the gap is now invisible as well as open; The lesson stays in human memory, which decays at the speed of attention, and the next engineer relearns it in production; A retro whose only artefact is a list of these has turned the loop back into a line: ceremony that changes nothing.

How to decide. Decide by blast radius and by who is on the hook, not by how the work feels in the moment. The governing question is what a tired engineer must hold in their head to make sure this work actually moves, and how bad it is if it silently does not. For anything whose failure compounds, corruption, a security path, a service that goes untouchable in an incident, an unowned item is itself a fault, so name one owner of record and run a clock; the cost of the work is dwarfed by the cost of it ageing unseen. For genuinely fungible, low-radius work where a real pull discipline and a queue owner exist, a shared queue is the cheaper correct shape, and forcing a single name on day one just bottlenecks on that person, so do not buy ceremony you will not use. The intention list is almost never the right output: it is honest only as raw capture you will convert before the room empties, or as a deliberately-dropped item recorded with its reason. The tie-break is the one the manifesto keeps making: spend accountability where the property is real and the cost of getting it wrong exceeds the cost of the structure, and keep the cheap reversible path light so the heavy controls still mean something when they show up.

Reach for first. Name one owner of record and attach a date before anyone leaves the room. It is the cheapest move that survives the meeting, and you can always relax it to a shared queue once a real pull discipline and a queue owner exist. The reverse, recovering accountability for an item that already evaporated, costs you the next incident.

Pitfalls. - Assigning a team alias and calling it an owner: an alias diffuses into nobody, and ‘who actually has this?’ becomes the first question of the next incident. - A due date set to a number nobody believes, which trains the whole team to treat every date as decoration. - Confusing the owner of the move with the owner of the keystrokes: the owner is accountable for the next step, which can legitimately be handing it on, not for personally doing the work. - Letting a single named owner become a single point of failure, so the item stalls the moment they take leave because no one is accountable for reassigning it. - Loading twenty owned, dated items onto a team with no slack, which ships none of them; pick the few that close this class of failure and explicitly drop the rest. - Putting owner-and-date ceremony on a trivial reversible chore, so people route around the process and lose the habit where it actually matters. - Leaving a deliberately-dropped item as a silent omission rather than a recorded decision with its reason, so a future reader cannot tell ‘declined’ from ‘forgotten’. - Ending a retro on a slide of intentions and treating the write-up as the deliverable, when the deliverable is owned, dated, triage-ready work fed back to the front of the loop.

Time-box with a planned exit vs run it to ground

You hit this when You are working a triage gate or an open investigation, and the thing in front of you has no natural finish line. A "look into the flakiness" grows a fresh hypothesis every hour; a "decide if this is real" quietly slides into debugging the fix. Before you start, you have to choose how much this gets and what happens when that runs out

The call. How long does this investigation or triage call get before you stop, hand off, or escalate, and did you decide that up front or let it decide itself?

Time-box with a planned exit. Fix a budget before you open the dig, sized to the blast radius of being wrong, and decide now what you do the moment it expires: hand off with what you know, escalate, ship the workaround, or accept the risk. The clock is not a guess at how long the work will take; it is the move you planned in advance so a two-hour question cannot quietly eat a fortnight. The exit is written down, so a closed box leaves an artefact ("spent the budget, here is what we know and what we would try next") rather than a silent slide into next week. - Choose when: The issue is reversible or low blast radius, so a good-enough disposition now beats a perfect one later; You can name a real escalation or hand-off target for the case where the box closes still open; The cost of the dig running long outweighs the cost of stopping a step early; Several issues compete for the same queue and this one must not starve the rest. - Cost: A clock can cut you off one step before the answer, and a tight box rewards the plausible guess over the proven one; Choosing the budget and the exit is itself work, and a box nobody enforces is theatre; An honest "spent the budget, unresolved" can read as failure to someone who wanted a verdict; If the exit move is vague the box just relocates the open question to whoever inherits the hand-off.

Run it to ground. Keep going until the unknown is actually reduced: a proven root cause and a red test, or a spike answer you can demonstrate with numbers. No artificial clock stops you short of the truth. You pay for certainty because the thing in front of you compounds if you act on a guess and get it wrong, so a fix that merely quiets the symptom is worse than no fix at all. - Choose when: High blast radius: silent data corruption, an attacker-reachable path, anything where a wrong guess propagates downstream; A symptom-only fix would hide the cause and let it return later wearing a new face; The repro is intermittent and you cannot tell a real fix from a coincidence until you pin it; One unresolved load-bearing unknown blocks everything behind it, so partial progress is no progress. - Cost: Open-ended by construction: with no stated question and no clock it bills by the hour and grows hypotheses faster than it kills them; It monopolises one person and the queue silts up behind them while they dig; Sunk cost sets in; the deeper you are, the harder it is to admit the budget should have closed; Chasing full determinism on an irreducibly statistical failure can burn the whole week and still not converge.

Fix on the spot inside the gate. Let triage become the fix: the cause is obvious, the change is a line, so you land it now rather than reproduce, rank, write the reason, and hand off. Cheaper than the full pipeline when the fix is genuinely small, reversible, and low radius. The risk is that the "obvious" cause was never validated, and that the gate stops draining the moment it starts solving, so the one ticket you fixed costs you the dozen behind it. - Choose when: The fix is genuinely trivial, reversible, and low blast radius, and you can see why with no real debugging; Formally handing off a one-line change would cost more than landing it; You are not mid-incident and the queue behind you is not backing up. - Cost: The moment it needs real debugging or design it has left triage, and you are now doing the next phase's job inside the gate with none of its time or owner; An "obvious" cause you never reproduced is an unvalidated claim; you may be fixing a rumour with a screenshot; The gate stops draining, the backlog builds behind it, and the bottleneck you were the gate for becomes you; The fix and its reason often go unrecorded, so the next person re-derives the call from nothing.

How to decide. Decide by blast radius, then by who controls the clock. Where a wrong answer propagates silently, corrupts data, or rides an attacker-controlled path, run it to ground: the cost of acting on a guess dwarfs the cost of the extra hours, and the right response to an unconfirmed high-radius issue is a bounded dig to get the proof, not a quiet downgrade. Where the issue is reversible and contained, time-box it, because the queue is a scarce shared resource and an unbounded investigation starves every other issue waiting behind it. Either way the spend is bounded; what changes is the size of the box and how loud the exit is. A box with no written exit move is not a time-box, it is a slow run-to-ground that nobody admitted to. The worst outcome of all is the third path gone wrong: the gate quietly absorbing the work it was meant only to route, so the backlog rots while one issue gets solved. Size the box to the blast radius, name the exit before you start, and let the box close loudly.

Reach for first. Write the one question whose answer unblocks you, then put a clock on it sized to the blast radius and name the exit move out loud. That single act ("two hours; if it is still open we escalate to the data-platform owner") costs a minute, caps the downside, and turns a closed box into a hand-off instead of a disappearance. You can always extend a box you decided on; you cannot recover the fortnight an undeclared dig already spent.

Pitfalls. - The box with no exit: you set a clock but never decided what happens when it rings, so the deadline passes and the dig just continues, unbudgeted and unowned. - Determinism eats the box: you spend the entire budget chasing a deterministic repro on a failure that is irreducibly statistical, when the honest move was to quantify the rate ("fails 3% under this load") and make that the number your fix must move. - Sunk cost masquerading as rigour: three days in, you keep going because stopping would waste the three days, not because the next hour is the cheapest correct move. - The silent box-blow: you quietly run long and tell no one, so the people downstream plan around a verdict that is not coming. - Triage that became the fix: you started ranking and ended up debugging, the gate stopped draining, and the queue behind you turned into zombie tickets that rot. - Acting on noise dressed as a finding: you time-boxed so hard you shipped a conclusion drawn from a flaky repro, where the symptom going quiet was the dice, not the fix.

See also. Tenets (XXI), (XIX). glossary: choose by blast radius, a zombie ticket that rots in the queue, a rumour with a screenshot, noise dressed as a finding. phases: triage, investigate.

Drive a throwaway spike to de-risk vs commit on the assumption

You hit this when A plan rests on an approach nobody has proven: the queue survives a 10x burst, the vendor's API holds your write rate, the data has the shape the migration assumes. You can settle the one unknown cheaply now, or carry it as a bet into the real work and meet the answer at full scale

The call. There is exactly one load-bearing unknown in the plan. Do you spend a small, bounded probe to settle it before committing, or commit the real work on the assumption and find out under load?

Spike: build the smallest throwaway probe, then bin the code and keep the finding. Name the single unknown that would move the date or the approach, build the least code that answers it (no error handling, no edge cases, hard-coded everything), get the number, then delete the branch and keep a written finding with the evidence and the rejected options. The deliverable is knowledge, not the probe. - Choose when: The unknown is load-bearing: being wrong about it changes the approach, not just a detail; Acting on a guess would have a wide blast radius: an irreversible cutover, a vendor you would be locked into, a schema you cannot easily walk back; The question is sharp enough to answer in a bounded box (‘holds 2k writes/sec on our payload, by Thursday’), and you can throw the code away once it answers; The probe is genuinely cheaper than the rework you would owe if the assumption broke at full scale. - Cost: Real time spent before any shippable work starts, and the spike can eat its own box if the question was never made sharp; The finding is a snapshot: a probe under pinned, friendly conditions can still flatter you about behaviour under real contention; Discipline tax: someone has to actually delete the code and write the finding down, or you get the worst of both paths; On a small, reversible unknown this is ceremony; you bought proof of something that being wrong about would have cost an afternoon.

Commit on the assumption and discover the unknown at full scale. Skip the probe. Build the real work straight onto the unproven approach and let production, or the migration window, return the verdict. - Choose when: The unknown is small and reversible: if the guess is wrong the fix is a scalpel, a config toggle or a one-line swap, not a demolition; You already hold strong evidence the approach holds, a near-identical system in production or a vendor SLA you trust, so a spike would only re-confirm what is known; The cost of the probe rivals the cost of just doing the work and backing out, and the work is shippable in reversible increments anyway. - Cost: You meet the answer at the worst time and scale: the null-rate at 2am mid-cutover, the rate limit under launch traffic; Sunk real work pulls you toward forcing a failed approach to limp rather than retreating, because retreat now means binning shippable code, not a throwaway branch; The unknown stays in the planner's head as an unmarked bet; nobody downstream knows the plan rests on a guess until it fires; If the blast radius was actually wide, the bet you forbade yourself from losing gracefully is the one you now have to lose at full cost.

Harden the spike and keep it (let the throwaway slide into production). The probe runs, the demo works, and rather than deleting it you wire it into the real path, keeping its spike-grade shortcuts: no input parsing, no edge cases, no tests, hard-coded config. - Choose when: Almost never as a decision. It is the default that happens when nobody decides to delete, so the honest version is: stop calling it a spike and re-cut it properly, parsing inputs and owning lifecycles as if it were written from scratch. - Cost: Every corner the spike was right to cut is now load-bearing under traffic it was never hardened for: unparsed hostile input, an unowned module, no test pinning its behaviour; It feels like thrift, yet it is the most expensive path; you pay full production rigour later, against code shaped by questions you no longer remember asking; The finding gets lost inside the artefact: what should have been a citable decision record is now a running system nobody can audit or safely change; Deleting working code later costs more nerve than deleting a throwaway branch did, so the shortcut tends to calcify rather than get fixed.

How to decide. Sort by who controls the unknown and how wide the blast radius is if you act on a guess and it turns out wrong. If the approach rests on an input you do not control, a vendor's real throughput, a stranger's data shape, a burst you cannot cap, and being wrong means an irreversible cutover or a lock-in, the spike is the cheap correct move: a day spent proving the number is far cheaper than rebuilding the real work when the assumption breaks at scale. If the unknown is small and the failure is a scalpel, a flag flip or a one-line swap on work you are already shipping in reversible increments, the probe is ceremony and committing on the assumption is right, provided you mark the bet in the plan so it is not a guess nobody flagged. The trap is the third path: a spike that survives. The instant you decide the probe must live, it is no longer a spike, it is unparsed input and an unowned module wearing a demo's clothes, and it owes the same parsing, ownership and tests as anything that ships. So spend the probe where the property is real and the cost of being wrong exceeds the cost of proving it; everywhere else, commit and mark the assumption. Either way, the only thing that should outlive the spike is the written finding.

Reach for first. Write the one question whose answer unblocks the plan, and the clock you will spend on it. If the question is sharp and the blast radius of guessing wrong is wide, that one line is already the spike's brief: build the least code that answers it, then bin the code and keep the finding.

Pitfalls. - A spike that ‘explores the options’ with no stated question and no clock: a rabbit hole that bills by the hour until something caps it. - Proving the approach under pinned, friendly conditions and reading it as proof under load: the hand-run that was fast ‘the couple of times I tried it’ with no real contention, noise dressed as a finding. - Letting the demo that ‘works’ slide into main six weeks later with every spike-grade shortcut intact, so unparsed input is now load-bearing under traffic nobody hardened it for. - Deleting the branch but writing nothing down, so the answer becomes folklore the next person re-derives from scratch instead of a citable finding with its numbers and rejected options. - Committing on the assumption without marking it: the bet lives only in the planner's head, and the implementer inherits a failure mode nobody decided how to handle. - Spiking the knowns, not just the unknowns, sinking days into questions whose answer never moves the date or the approach.

See also. Tenets (XX), (XI). glossary: dry run, noise dressed as a finding, folklore re-derived from scratch. phases: investigate, planning.

Define "done" as an explicit contract vs leave it implicit

You hit this when Work has been scoped and will later be verified. You are about to decide what "done" means: write it once as a checkable contract everyone measures against, or carry it in your head and judge it by eye when the change lands

The call. When you scope this work, do you write "done" down as something a second person could check, or do you leave it to be judged at the end?

Write "done" once, up front, as acceptance criteria everyone verifies against. Before the work starts, you state what finished looks like as a small set of checkable conditions: the behaviours that must hold, the inputs that must be rejected, the failure path that must degrade a named way. The criteria pin the observable contract, not the implementation. The same list the planner wrote is the list the verifier walks and ticks off in the running system. - Choose when: The work crosses a boundary someone else trusts: a contract change, a public response shape, a behaviour downstream callers depend on; Whoever scopes the work is not whoever verifies it, so "done" has to survive the handoff without the author in the room; Getting it wrong is dear or hard to undo: data is touched, money moves, a hostile input reaches the path; The acceptance criterion can also be the regression check kept forever once the work ships. - Cost: You pay up front, before you fully understand the problem, and a criterion written too early can lock in the wrong shape of "done"; Naming every condition is slower than starting work, and on a trivial change the writing costs more than the judging would have; Criteria pinned to internals rather than the contract rot on the first refactor and turn into a tax that punishes the cheap change; Someone has to keep the criteria honest as the work shifts, or the list and the real behaviour drift apart and the verifier ticks a lie.

Leave "done" fuzzy and judge it at the end. You start the work with "done" held in your head, and decide whether it is finished by looking at the result when it lands. No written contract, no list to walk: the judgement happens once, by eye, at the end, against whatever you remember the goal was. - Choose when: The change is small, reversible, and low blast radius, and the same person scopes, makes, and checks it within one sitting; "Done" is genuinely self-evident: a typo, a label, a one-line fix whose correctness you can see at a glance; The problem is still too unformed to commit a criterion to, and writing one now would only fossilise a guess; Exploration is the point, and the output of the spike is itself what tells you where the bar should sit. - Cost: "Done" drifts: judged by eye at the end, it quietly becomes whatever you happened to build, not what the issue asked for; Nobody else can verify it, because the bar lives in your head and leaves with you when you move on; The check happens once, by the author, against their own diff: a hope wearing a green tick, never a before-and-after anyone watched; Disagreement about whether it is finished surfaces at the end, when it is dearest to relitigate, instead of up front when it was cheap.

Let the spike's output define the criterion. You run a time-boxed investigation precisely because you cannot yet state what done looks like. The spike's job is to produce that statement: it ends by writing the criterion the real work will then be verified against. "Done" stays deliberately open for the spike itself, bounded only by the time-box and the named open question, and the deliverable that closes it is the contract for the phase that follows. - Choose when: You genuinely cannot name the conditions yet, and writing them now would be inventing a number to fill a field; Triage has dispositioned this as needs-a-spike with the open question named, not as ready-to-build work; The unknown is bounded and worth paying to resolve before anyone commits to a definition of done; The spike is scoped to answer "what would done even mean here?", and that answer is its deliverable. - Cost: Two phases where the eager path had one: you pay for the investigation and then for the work it scopes; A spike with no exit condition of its own becomes the work, draining its time-box and never producing the criterion it owed; If the spike's finding is never written down as the criterion, you are back to judging by eye, having paid extra for the privilege; The pull to keep spiking past the point of an answer, because an open question feels safer than committing a bar.

How to decide. Decide by who has to check this and how far a wrong "done" reaches. If the same person scopes, builds, and verifies a small reversible change in one sitting, the contract lives safely in their head and writing it down is ceremony: judge it by eye and move on. The moment "done" has to cross to a second person, or the work touches a boundary others trust, or a controlled input reaches the path, or the result is dear to undo, the bar has to leave your head and become something a tired verifier can walk without you. That is the governing tie-break made specific: minimise what the next person must hold to confirm the change is right, and bound the blast radius of getting "done" wrong. Spend the writing where the property is real, the contract not the internals, so the criterion survives a refactor and earns its keep as the regression check. Where you cannot yet state the bar, do not fake one and do not leave it implicit: disposition it as a spike whose output is the criterion, and pay for that explicitly.

Reach for first. Write one line of acceptance criteria for the load-bearing behaviour, phrased as the reproduction or contract a second person could check, before you start the work. For a genuinely trivial, reversible, single-hand change, judging by eye is the proportionate move. For anything else, that one line is the cheapest thing that stops "done" drifting into whatever you happened to build.

Pitfalls. - Writing criteria pinned to the implementation ("calls this helper", "sets this flag") instead of the observable contract, so they break on the first rename and punish the refactor they should have protected. - Letting the criteria and the real work drift apart as the change shifts, so the verifier ticks a list that no longer describes the system. - Treating "works on my machine, looked right" as a met criterion: a judgement by the author against their own diff, never a check anyone else could regenerate. - Leaving "done" implicit on a cross-boundary change, then discovering at verification that the author and the verifier never agreed what finished meant. - Running a spike with no exit condition of its own, so it absorbs its time-box and never produces the criterion it was meant to write. - Forcing a criterion when you genuinely lack the information, fossilising a wrong bar that everyone then dutifully verifies against.

See also. Tenets (XV), (XXIV). glossary: test the contract, not the internals. phases: planning, triage, verify.

Bisect the cause space (falsifiable hypothesis) vs scan or scattershot

You hit this when A defect reproduces, and the cause could be any of a large set of suspects: commits, layers, config, data, timing. You have to choose how the next probe spends its time

The call. A defect has a wide space of possible causes. Do you halve that space with each falsifiable test, walk the candidates in order, or poke at whatever looks guilty?

Bisect: form a falsifiable hypothesis and halve the cause space. State a prediction sharp enough that one observation can kill it, then design the next probe to rule out half the remaining suspects rather than to confirm your hunch. Binary search over causes: a hundred suspects fall in seven steps. Each step ends with a fact, because you decided in advance what would prove you wrong. - Choose when: The cause space is large and you can partition it cleanly: a commit range to git bisect, layers to confirm at a boundary, a config you can toggle; The blast radius of acting on a wrong cause is wide enough that you need the cause proven, not the symptom quieted; You can name a single observation that would falsify your current theory; if you cannot, you have a hope, not a hypothesis; The reproduction is deterministic, so each probe's result is signal and not a coin flip. - Cost: Setting up a clean partition costs time up front: a reliable repro, a bisectable range, a boundary whose value you can read; On a tiny cause space the ceremony of stating and falsifying can be slower than just looking; Bisection needs a monotone signal of good vs bad; a flaky or non-monotone failure breaks the halving, and every step inherits the randomness; It asks you to design probes that disconfirm, which is less satisfying than chasing the suspect you already blame.

Scan linearly: walk every candidate in order. Enumerate the suspects and check them one by one, in a fixed order, until the cause surfaces. No hypothesis, just exhaustive coverage. - Choose when: The cause space is small, or you genuinely cannot partition it because the suspects share no axis to halve along; Each check is cheap and the list is short enough that order barely matters; An audit needs a guarantee of completeness: every candidate looked at and recorded; The failure is non-monotone, so bisection's good/bad signal would not hold anyway. - Cost: Linear in the number of suspects: fine for ten, ruinous for a thousand, where a bisect would close it in ten steps; Tempts you to stop at the first plausible-looking candidate and call it the cause without ruling out the rest; No early structure: you learn nothing about the shape of the space until you happen to hit the cause; Easy to misorder so the true cause sits last, paying full cost for what one split would have isolated.

Scattershot: poke at whatever looks suspicious. Change things that feel guilty and watch whether the symptom moves. The fix that might help, applied on the off-chance, dressed up as debugging. - Choose when: Almost never as a method; only as a deliberate, logged symptom-relief on a reversible, low-blast-radius issue while the real dig is queued; You are mid-incident and a quick reversible mitigation buys time, with the cause explicitly left for later; The space is so small and familiar that one informed poke is genuinely faster than any setup. - Cost: It is a guess wearing a lab coat: it tests nothing and rules nothing out, so the cause space is exactly as wide after as before; When the symptom goes quiet you cannot tell a fix from a coincidence, so the bug returns next quarter wearing a new face; Each poke that helps a little entrenches a wrong mental model, because you never found out why; It leaves no citable record, so the next person re-walks the same maze from scratch.

How to decide. Two variables settle it: the width of the cause space and the blast radius of being wrong about the cause. When the space is large and you can partition it cleanly, bisection is the only move that scales. It halves what a tired investigator has to hold in their head at each step, and it lands on a proven cause rather than a quieted symptom. When the space is small or has no axis to split along, a linear scan is honest and complete, and the bisect's setup would cost more than it saves; choose it there without apology, but still rule each candidate out instead of stopping at the first that looks guilty. Scattershot is not a third method, it is the absence of one. Reserve it for a logged, reversible mitigation when the blast radius is small and the clock is against you, and never let a quiet symptom stand in for a cause you proved. The spend rule follows from the second variable: where the cost of acting on a wrong cause exceeds the cost of proving the right one, pay for the bisect; where it does not, the cheap scan or the recorded poke is the fair trade. The test for any probe is whether it halves the unknown or merely flatters your hunch.

Reach for first. Before any probe, write the one sentence that would prove your current theory wrong. If you can write it, design the next step to split the cause space in half. If you cannot, you do not yet have a hypothesis: go back to the reproduction and the telemetry you already emit until a falsifiable claim falls out.

Pitfalls. - Bisecting against a flaky repro: the good/bad signal is noise dressed as a finding, and every halving step trusts a coin flip. - Designing probes that confirm rather than falsify, so you spend ten steps gathering evidence for a theory no single observation could ever kill. - Calling a scan complete after the first plausible candidate, leaving the rest of the space unchecked and the real cause possibly still in it. - Letting a scattershot mitigation that quieted the symptom close the ticket, so the cause is never proven and returns unobserved. - Following the most confident voice in the room rather than the artefact, so the loudest hypothesis gets the first probe regardless of what the evidence shows. - Bisecting along the wrong axis, commits when the cause is data or layers when the cause is timing, so the halving is clean and the answer is still wrong.

Mitigate the symptom now vs find the root cause first

You hit this when Something is on fire. You can stop the bleeding now, or hold the line and refuse to touch anything until you understand why it's bleeding. The patch and the diagnosis pull in opposite directions while users are hurting, and the move you reach for under that pressure is the one you'll regret or thank yourself for later

The call. When a live failure is hurting people, do you mitigate the symptom now and chase the cause afterwards, or refuse the patch until the root cause is proven?

Mitigate now and record the debt. Stop the harm with the smallest reversible move you can (a flag flip, a rollback, a rate cap, a drained node), then open a tracked investigation for the cause with an owner and a deadline. The mitigation buys time; the recorded debt makes sure the time gets spent. The symptom is quiet and the cause is still on the books, named as a thing that is not yet understood. - Choose when: The blast radius is live and widening: data is being corrupted, money is moving wrong, or an attacker-reachable path is open, and every minute of investigation is paid in fresh harm; The mitigation is genuinely reversible and small (a toggle, a rollback, a cap), so it cannot itself become the new incident; You can name an owner and a date for the cause, so the patch does not quietly become the permanent answer; Staying down or staying corrupt while you investigate is the more expensive of the two harms. - Cost: You pay twice: once to mitigate, once to fix, and the second payment is the one that always feels optional after the alarm stops; A mitigation that hides the symptom can also hide the signal you needed to find the cause, so investigate against the unmitigated repro, not the patched system; The recorded debt is only as real as the process behind it; without an owner and a deadline it becomes a swallowed error that learned to hide, just at the ticket level.

Prove the cause first. Resist the patch. Hold the system in its failing state long enough to get a deterministic repro and a falsifiable diagnosis, on the grounds that any fix applied before you understand the cause is a guess that might mask the real fault or move it somewhere worse. You ship nothing until you can demonstrate why it broke. - Choose when: The blast radius is contained or already stopped, so holding the failing state costs little and buys a real diagnosis; The symptom is the only evidence you have, and mitigating it would destroy the repro you need (the corrupt row, the wedged queue, the live trace); A wrong mitigation could compound the damage: masking a corruption bug so it spreads unseen is worse than the visible failure; The failure is intermittent and a premature patch would let you mistake a coincidence for a cure. - Cost: You are spending users' pain as your investigation budget; every minute you hold the line for a cleaner diagnosis is a minute someone is still hurt; Purity here can tip into paralysis: an open-ended root-cause hunt with no clock is a rabbit hole that bills by the hour while the fire burns; Some causes are irreducibly slow to find, and insisting on full proof before any relief can turn a ten-minute mitigation into a multi-hour outage for no proportionate gain.

Silent symptom fix. Paper over the symptom and move on. Clear the cache, bump the pod, retry the job until it sticks, and close the incident once the dashboard goes green, without recording that the cause was never found. The failure is invisible again and nobody owns the question of why it happened. - Choose when: Almost never as a deliberate choice; it is named here because it is what mitigation decays into when the debt goes unrecorded; Defensible only for a genuinely one-off, low-radius blip with no plausible recurrence, and even then the honest move is a one-line note, not silence. - Cost: The cause is still live, so the failure returns on its own schedule, miles from here and harder to trace, now wearing a new face nobody connects to this one; You have spent the incident's attention (the freshest context anyone will ever have on this bug) and kept none of it; the next person re-derives the whole thing from scratch; The green dashboard now lies: it reports a fix that is really a coincidence, and the next on-call inherits a system that looks healthier than it is.

How to decide. Decide by blast radius and who controls the clock. Ask first whether the failure is still doing harm: if it is widening, attacker-reachable, or corrupting data that others trust, the harm controls the schedule, not your taste for a clean diagnosis, so mitigate now with the smallest reversible move and chase the cause behind a tracked, owned ticket. If the harm is already contained or stopped, you control the clock, and the cheaper mistake is a premature patch that masks the real fault, so prove the cause first against the unmitigated repro. The line between the two live options is not patch vs diagnosis as a matter of principle; it is which is the lesser harm right now, the visible failure or the unproven fix. The third option, the silent fix, is never on that line: skipping the recorded debt does not save you the cost, it only moves the cost onto whoever hits this next, with none of the context you have at this moment. Whichever live option you take, mitigation and diagnosis stay separate acts, and the cause is never closed on a symptom going quiet.

Reach for first. Stop the harm with the smallest reversible move available (a flag, a rollback, a cap), and in the same breath open one ticket for the cause with a named owner and a date. That is the cheapest correct move: it ends the bleeding without betting on a guess, and it makes the debt a thing on the books rather than a thing in someone's memory. Then investigate against the failing repro, not the patched system.

Pitfalls. - Closing the incident when the dashboard goes green, treating the symptom going quiet as proof the cause is gone. A mitigated symptom and a fixed cause are different states, and only one of them keeps the bug from returning. - Investigating against the mitigated system, so the patch has hidden the very signal you needed and your repro no longer fires. Reproduce against the unmitigated failure. - Recording the debt with no owner and no date, which is a zombie ticket: not alive enough to fix, not dead enough to forget, silting up the next triage. - Letting the root-cause hunt run with no clock when the fire is out, so a contained issue eats a fortnight chasing a diagnosis nobody is paying for any more. - Reaching for prove-the-cause-first while data is actively corrupting, spending users' harm as investigation budget when a reversible mitigation was sitting right there. - Picking a mitigation that is not actually reversible (an irreversible migration, a destructive cleanup), so the patch becomes a second incident on top of the first.

See also. Tenets (XIII), (XX). glossary: choose by blast radius, a swallowed error that learned to hide. phases: investigate, operating.

Hand off a diagnosis and a failing test vs fuse the fix and ship

You hit this when You have run the bug to ground. The cause is finally in view, and the fix is sitting right there. The question is what leaves the investigation: a proven root cause with a red test that pins it, handed to the implementing phase, or the fix itself, fused into the same motion and shipped

The call. You have found the bug. Do you hand on a diagnosis plus a failing test and let the fix be a separate act, or do you fuse the fix and ship it now?

Diagnosis plus a failing test. Close the investigation with two artefacts and no repair. A root cause you can demonstrate, and a red test that reproduces the defect at the contract, not the internals. The fix becomes the next phase's job, working against a finish line you have already drawn. You also leave the ruled-out list, so no one re-walks the dead ends. - Choose when: The verb at the end is irreversible or wide: a charge, a delete, a migration, anything where a wrong fix costs more than the test does; The cause is subtle and the obvious fix only quiets the symptom; the test is what proves the repair is real and not a coincidence; Someone other than you will implement, or will review, and needs the cause stated rather than re-derived; The bug escaped once already, so the missing guard is the actual deliverable, not the patch. - Cost: Two handoffs and a context switch where one motion would have done; the implementer reloads what you already hold in your head; The fix lands slower, which bites hardest mid-incident when the bleeding is live; A red test written before the fix can over-pin: tie it to a private helper and it becomes a tax on the very refactor that fixes the bug; The diagnosis can rot in a queue while the symptom keeps firing, if the next phase never gets picked up.

Fuse the fix and ship it. Diagnose and repair in one continuous act, then ship. The proof of cause and the proof of fix collapse into a single green build. Fastest path from understanding to a system that no longer misbehaves. - Choose when: The change is reversible and its blast radius is small: a config flip, a clamp, a one-line guard you can roll back in seconds; You are the owner and the reviewer, and the cost of being wrong is a quick revert, not a corrupted ledger; Mid-incident, where stopping the bleeding now beats a tidy paper trail you can backfill once it's calm; The cause is plain and shallow enough that a separate red test would only restate what the diff already makes obvious. - Cost: You lose the finish line: a fix with no failing test is a diagnosis you've decided to forget the moment it passes; Momentum tempts the fix that works for a reason you never pinned, so the bug returns on its own schedule wearing a new face; The cause stays in your head, never written, so the next person who hits this re-runs the whole dig from zero; Reviewing a fused change is harder, because the reader cannot separate whether the cause is real from whether the patch is right.

Quiet the symptom without a test. Make the visible failure go away with a targeted poke, clear the cache, bump the retry, restart the worker, without proving the cause or leaving a guard. The dig stops the instant the symptom does. - Choose when: A reversible, low-blast-radius stopgap is genuinely needed to buy time, and you log it as a stopgap; The clock on the investigation has run out and you are explicitly accepting the risk, on the record; The true cause sits outside your reach this round and the symptom must be held off meanwhile. - Cost: The cause is unproven, so the failure comes back unobserved, and next time it costs the dig again; No test, no guard, no record: it is debt that does not announce itself as debt; Done silently it becomes folklore, the workaround everyone applies and no one can explain; It trains the team to treat symptoms, so the same class of bug accretes a graveyard of pokes.

How to decide. Decide by the blast radius of a wrong fix and by who carries the cause afterwards. The investigation's job is to shrink the unknown and push what you learned out of your head into something the system carries, so the default deliverable is a diagnosis and a red test: that is what gives the repair a finish line and stops the bug returning unobserved. Fusing the fix and shipping is the right call only when the verb at the end is cheap and reversible and you are the one who owns and reviews it, because then a wrong guess costs a revert, not a recovery, and the diff itself is proof enough. The moment the fix touches an irreversible or wide-blast act, or the cause is subtle enough that the patch could pass for the wrong reason, the red test stops being ceremony and becomes the only thing that separates a proven repair from a hopeful one. Quieting the symptom without a test is a stopgap, never a close: it is admissible only when reversible and logged as accepted risk, because an unproven fix is a diagnosis you have agreed to forget. Spend the extra handoff exactly where the cost of the bug coming back exceeds the cost of writing the test, and not before.

Reach for first. Write the failing test first, before you touch the fix. It costs little, it forces you to state the cause sharply enough that one observation could be wrong, and it converts whatever you do next, fuse or hand off, into a change with a finish line. If the verb at the end turns out to be cheap and reversible and yours, you can fuse the fix straight onto the red test and ship in one motion; the test was never wasted.

Pitfalls. - Shipping the fix with no failing test, so a future refactor silently reintroduces the bug and nothing catches it. - Pinning the red test to a private helper instead of the observable contract, turning the guard into a tax that punishes the fix's own refactor. - Fusing under incident pressure and never backfilling the test or the diagnosis once it's calm, so the paper trail you promised never arrives. - Handing off a fix dressed as a diagnosis: 'I changed a few things and it seems better', which proves neither cause nor repair. - Letting the symptom-quieting stopgap become permanent because it worked, leaving the real cause unowned and unrecorded. - Dropping the ruled-out list, so the next investigator re-eliminates the cache, the clock and the index you already cleared.

See also. Tenets (XI), (XXIV). glossary: separate the decision from the effect, test the contract, not the internals, a bug that escaped is a missing test. phases: investigate, review.

Ship & learn

Block the merge (must-fix) vs name the cost and let it ship (consider)

You hit this when You have read the diff and found something. The change works, the tests are green, and the author is waiting. The question is no longer whether the thing you found is real; it is whether it stops the merge or rides along as a note. Get this wrong one way and you ship a fault an unlucky caller will trip. Get it wrong the other way and you become the reviewer whose approval costs a day, so authors learn to route around you

The call. You found something in review. Do you hold the merge until it is fixed, or record it as a non-blocking note and let the change ship?

Must-fix: block on it. You withhold approval until the finding is resolved. You reserve this for correctness and blast-radius problems: an unbounded thing the caller controls, a swallowed failure, a privilege the change didn't need, a boundary that trusts the wire, a missing test for behaviour the diff changed. You post these first and sorted hardest-first, so the author fixes the catastrophe before the cosmetics, and you say plainly what wrong thing the current shape still lets someone express. - Choose when: The fault is something an attacker or an unlucky caller controls, and the blast radius when it fires is wide: corruption, an outage, a leak, a charge made twice; A failure can vanish on some path with no signal and no decision, so production shows a latency cliff or a rising error rate instead of a stack trace; The diff changes behaviour and nothing executable pins the new contract, so the rule lives only in the author's head and the next refactor quietly breaks it. - Cost: You spend the author's time and your own, and a block that turns out to be taste in must-fix clothing burns the credibility that makes the next real block land; Held merges pile up; the longer a correct-enough change waits, the more it drifts from main and the more the rebase costs; A block is a heavier instrument than a comment, so over-using it trains authors to pre-empt you with noise or to seek a softer reviewer.

Consider: name the cost and let it ship. You record the finding as a non-blocking note and approve. You reserve this for taste, nits, smaller risks, and structure you would prefer but can live without. You still say the cost out loud, who pays it and when, rather than dropping a bare opinion, so the author can weigh it and the note survives as a real signal instead of a shrug. - Choose when: The finding is taste, naming, or a local readability nit whose blast radius is the next reader's patience, not correctness; The risk is real but narrow and reversible: contained to one feature, behind a flag, cheap to undo in a year if it bites; The fix is a genuine improvement but the change is already correct, and holding the merge would cost more than the note left for a follow-up. - Cost: A note is prose, and prose enforces nothing; an edge check would have made the bad state unexpressible, where the comment just hopes someone reads it; Consider items rot: the follow-up is never picked up, the smaller risk you waved through is the one that pages someone, and the backlog fills with notes nobody owns; Calling a thing ‘consider’ when it was really must-fix is the expensive mistake in this fork, because you have signed off on the catastrophe in writing.

How to decide. Sort the finding by who controls the input and how wide the blast radius gets, not by how much it annoyed you to read. If the thing you found is reachable by an attacker or an unlucky caller and the damage when it fires is unbounded (corruption, an outage, a leak, a swallowed failure with no signal), it is must-fix, and you block however nicely the diff reads. If the input is yours, the actor is trusted, and the worst case is contained and reversible, it is consider: name the cost honestly and let it ship. The asymmetry settles it. A wrongly-blocked change costs a day and some goodwill, both recoverable; a wrongly-shipped catastrophe costs an incident and the trust that you read carefully. So when you genuinely cannot tell which side a finding falls on, treat it as must-fix until the author shows the blast radius is bounded, and never spend a block where the tiebreaker is taste.

Reach for first. Before you decide block or note, name the worst thing the current shape still lets someone express, and who controls the input that gets it there. If that worst thing is wide and caller-controlled, it is must-fix; if it is narrow and yours, it is consider. That one sentence is cheaper than the argument that follows a mis-sorted finding.

Pitfalls. - Padding the block list with nits so the real must-fix sits at position seven, where the author fixes the cosmetics first and runs out of patience before the catastrophe. - Approving with a wall of consider notes you know nobody will action, which is shipping the risk while keeping a paper trail that says you saw it. - Demoting a finding to consider because the author pushed back or the merge was urgent, rather than because the blast radius is actually bounded. - Blocking on a personal preference dressed as correctness; the reviewer who cries must-fix over taste gets ignored on the one that matters. - Leaving a finding as a comment when an edge check at the boundary would make the bad state impossible: a note is the weakest enforcement there is. - Treating a flaky, non-deterministic observation as a solid finding, so the block rests on a coincidence the repro cannot reproduce.

See also. Tenets (XIX), (XIII). glossary: choose by blast radius, an edge check beats a comment, noise dressed as a finding. phases: reviewing.

Verify at production-like scale and environment vs on friendly dev data

You hit this when You are proving a change works before it goes wide. The diff compiles, the local run is green, and the next move is deciding what counts as proof. You can exercise the change at the size and in the environment the caller actually reaches, or on the local loop against dev fixtures that load in a blink and lie about nothing except the one thing that matters

The call. Where do you run the verification the team will trust as a fact: at production-like scale and config with your signals on, or on friendly dev data on your own machine?

Production-like scale and environment, watching the signals you shipped. You drive the cost-bearing path at a size the caller can really reach, in a place that shares production's config, dependencies and data shape: staging, a canary, a production-scale sample. You assert a bound, a query count or a latency budget rather than just a correct answer, and you read the metric, the log and the trace as you exercise it. - Choose when: The path costs more as the caller's input grows: a sort, a parser, an N+1, a fan-out, anything caller- or attacker-controlled in size; The change's blast radius reaches users, money or stored state, and getting it wrong is hard to walk back; The behaviour turns on config, a real dependency, or a data shape that dev fixtures do not carry; You shipped instrumentation for this change and the only honest check is to watch those signals move as you drive it. - Cost: Slow and awkward: standing up a prod-like environment and reading the dashboards costs far more than a local run, and it cannot fire on every push; A wall-clock number swings with whatever else the runner is doing, so a naive timing assertion is flaky unless you fall back to a cheaper proxy; Production-scale data and pinned environments are themselves state to own, refresh and expire, with their own storage and access cost; Done for a trivially-correct, low-reach change it is pure ceremony, and the queue for the shared canary becomes its own bottleneck.

Dev data on the local loop. You exercise the change on your own machine against small fixtures, with mocked or default dependencies and dev config. It is fast, repeatable and runs on every push; you confirm the answer is correct on data that fits in a glance. - Choose when: The change is reversible and low-reach, and its cost does not grow with the caller's input: a label, a copy fix, a pure refactor with no size sensitivity; You are checking correctness of logic, not the cost or the config, and dev data exercises the same code path the caller will; You need the tight, repeatable loop to iterate, with the heavier tier reserved for before the change goes wide; No part of the behaviour depends on production's config, real dependencies or data shape. - Cost: The smallness of the fixture hides the very thing that matters: the O(n squared) is instant on a hundred rows and melts on a million, and "dev data was fine" is exactly how it ships; Mocked dependencies and dev config mean a default silently stood in for the production value, so "works on my machine" is a claim about one machine and not the user's; An N+1 stays invisible until the table grows, and a 2 GB body the boundary accepts says nothing through a friendly fixture; It reads as proof and gets trusted as a fact, so the gap surfaces on the pager rather than in verification.

A proportionate proxy sized to the blast radius. You verify the cheap signal that scales with the risk rather than the full environment: feed the path the largest input the contract permits and assert the algorithmic bound or query count, run the job on a production-scale sample, or confirm the config-sensitive behaviour in a lighter environment that carries only the parts that decide it. - Choose when: The risk is real but a full canary is more than the change has earned: a meaningful but bounded blast radius; The cost is legible as a bound, a flat query count or a memory envelope, so you can assert it without a flaky wall-clock number; You can carry the one production-like property that matters (size, a real dependency, the live config) into an otherwise cheap run; You want the cliff caught early without paying for, or queuing on, the shared full-scale tier. - Cost: A proxy that is wrong about which property matters gives false confidence: the query count is flat but the per-row work still melts; Choosing the right proxy takes judgement the full run does not, and it is easy to talk yourself into a cheaper check than the risk warrants; It does not exercise the real config and dependency interactions a canary would, so some environment-shaped bugs still slip through; Maintained carelessly it drifts from production until it is friendly dev data wearing a production-scale label.

How to decide. Decide by who controls the size of the input and how far the change can reach if you are wrong. The dev-data run is the cheaper, faster, more repeatable check, and for a reversible, low-reach change whose cost does not grow with caller input it is the proportionate proof; reaching for a canary there is ceremony. The moment a path costs more as the caller's input grows, or the blast radius touches users, money or stored state, friendly dev data stops being proof: the smallness of the fixture is the very thing hiding the cliff (XVII). So size the verification to the blast radius. Where the input size is caller- or attacker-controlled, drive it at the size the contract actually permits and assert a bound (VII, XVII). Where production's config, dependencies or data shape decide the behaviour, confirm it where those are real, and read the signals you shipped while you do (XVIII). Spend the slow, awkward, full-scale run as a deliberate tier you own and trigger before the change goes wide, not on every push; and prefer a cheap proxy, a query count or an algorithmic bound, over a wall-clock number that swings with the runner. The test is simple: would the on-call engineer, knowing less than you do now, trust this green tick at 2am, or did you only ever prove it on the one input that was never going to cause trouble?

Reach for first. For a reversible, low-reach change whose cost does not grow with caller input, the dev-data run on the local loop is the correct proof; do not manufacture a canary for it. Where the risk is real, the cheapest move that scales with it is the proportionate proxy: feed the path the largest input the contract permits and assert a query count or an algorithmic bound, which catches the cliff without the cost and flakiness of a wall-clock number on a full environment.

Pitfalls. - Reading a green local run as proof for a path whose cost grows with caller input: the fixture is small precisely because it hides the cliff, and "dev data was fine" is exactly how it ships. - Asserting a raw wall-clock number on a shared runner, then chasing the flake it produces, instead of asserting a query count or an algorithmic bound that does not swing. - Standing up the full canary for a one-line label fix: ceremony that adds nothing to a reversible, low-reach change and clogs the shared tier for changes that need it. - Verifying on your laptop against a dev database and a mocked dependency, then shipping on the strength of "works on my machine" while a default silently stands in for the production value. - Confirming the parser accepts the example from the issue and never feeding it the largest body the contract permits, so the 2 GB JSON that parses fine and then kills the heap goes untested. - Driving the change at scale but with the instrumentation off, so you measure the cost without seeing whether the metric, log and trace you shipped actually move. - Letting the production-scale sample or pinned environment drift until it is friendly dev data in disguise, then trusting it as though it still mirrored production.

Leave a permanent regression test behind vs a one-off manual check

You hit this when You have just watched a fix work, or confirmed a behaviour holds, in a running system. The change is sound. The open question is what you leave behind: a guard that re-runs the check on its own for every later change, or the memory that you once looked and it was fine

The call. You have verified something is correct. Do you encode the check as a permanent self-rerunning guard, or confirm it once by hand and move on?

Permanent regression. Encode the verified behaviour as a self-rerunning guard: pin the inputs that triggered it, assert the invariant that was violated, and wire it into the run so it fires on its own for every later change, including the call sites nobody has written yet. - Choose when: A bug escaped to where someone felt it; the reproduction you built to pin the cause is the acceptance check, and an escaped bug is a missing test as much as a bad commit; The behaviour sits on a path an attacker or an unlucky caller controls, where a silent regression has a real blast radius (a boundary that bounds cost, a charge that fires once, a field a downstream team reads); The rule is one the codebase will keep brushing past: a refactor, a dependency bump, or a new call site could quietly undo it without anyone noticing; You can phrase the assertion against the invariant rather than the surface of one run, so it survives the refactors this manifesto wants to keep cheap. - Cost: A test is state you now own. It drifts, it needs a home, and it competes for the next reader's attention along with every other check in the suite; Pinned wrong, it couples to an incidental detail and breaks on unrelated changes, until someone deletes it in frustration and the lesson leaves with it; Building the pinned, seeded, contract-level reproduction is slower than looking once, and for a genuinely one-off behaviour that slowness buys nothing; A suite that only grows trains people to mute it; every guard you keep is one more thing the on-call engineer has to reason about at 2am.

One-off manual look. Verify the behaviour once, by hand, in the running system, satisfy yourself it holds, and leave no automated artefact behind. - Choose when: The behaviour is genuinely transient: a one-time migration, a data backfill, a manual operation that will never run again, where there is no future change for a guard to protect; Reproducing it for a machine is disproportionate to the stake, and you would be manufacturing a fixture for a path no caller can reach; You are still investigating and a confirmation now would only harden a shape that is about to move; pinning it would cost more than it protects; The blast radius is small and reversible, and the cost of the rare regression is lower than the cost of carrying and maintaining the guard. - Cost: The confirmation lives only in your terminal history; it protects this release and nothing after it, and the next change can quietly undo the behaviour with no red light anywhere; A pass you cannot regenerate is an anecdote, not a finding: nobody else can re-run your reasoning, and in six months neither can you; If the behaviour was load-bearing, you have left its hardest claim resting on a human remembering to look, which is the vigilance the manifesto refuses to rely on; The judgement about whether it was 'really' one-off is itself a thing a tired engineer has to get right, with no structure backing them if they get it wrong.

Keep one, expire the rest. Keep the one load-bearing guard that pins the escaped bug or the controlled-input invariant, and deliberately decline to encode the incidental detail around it. Treat reproduction infrastructure as state and prune it on the same discipline as any other state. - Choose when: You want the protection of a permanent regression without paying suite-bloat for every transitive fact a verification run happened to touch; The verified behaviour has one invariant worth keeping forever and a halo of incidental specifics that were never the point; A temporary mitigation or a snapshot needs a guard now but should carry an owner and a sunset, not become a permanent fixture by default; The suite is already noisy, and a new guard only earns its place if an old, stale one leaves to make room for it. - Cost: Deciding which detail is load-bearing and which is noise is a judgement you can get wrong, and a guard pruned too aggressively stops catching the class it was built for; It is more work up front than either keeping everything or keeping nothing: you assert the invariant and explicitly justify the omissions; Sunsets and expiries are themselves state to track; an unowned 'expire later' becomes the permanent workaround it was meant to avoid; Pruning an existing guard to make room can quietly remove cover that some later change was silently relying on.

How to decide. Decide by who controls the input and how wide a silent regression would spread. Where the behaviour sits on a path an attacker or an unlucky caller can reach, a boundary that bounds cost, an operation that touches money or state, or a contract a downstream team trusts, the blast radius of it quietly breaking is real, and the manifesto's premise that correctness lives in structure rather than vigilance settles it: leave the permanent regression, asserted on the invariant so it survives refactors. An escaped bug is already telling you a guard was missing; the reproduction you built to pin it is that guard, and confirming it by hand confirms only this release. Where the behaviour is genuinely transient, a one-time migration or a backfill that will never run again, with no future change for a guard to protect and a small reversible stake, a one-off look is the proportionate move and a permanent test is ceremony you will pay for in drift and noise. The tie-break is what a tired engineer must hold in their head later: a guard that re-runs without them removes a thing to remember, but only if it is pinned to the invariant and not the incidental detail, and only if the suite it joins is pruned hard enough that the signal still means something.

Reach for first. If the thing you just verified was a bug that escaped to where someone felt it, leave the permanent regression: promote the exact reproduction you built to pin the cause into an automated check, asserted on the invariant that broke. That is the cheapest correct move, because you have already done the expensive part, and the alternative leaves the next change free to undo the fix in silence.

Pitfalls. - Adding a green check that was never seen red: a test written to match your own patch passes whether or not it touches the bug, so watch it fail on the broken build before you trust it forever. - Pinning the test to an incidental detail of one run instead of the invariant, so it breaks on every unrelated refactor until someone deletes it and the lesson goes too. - Calling a behaviour 'one-off' to dodge the work, when it actually sits on a controlled-input path that the next change will quietly break. - Leaving the verification in your terminal history and treating 'it passed when I ran it' as a guarantee anyone else can reach. - Letting the suite only ever grow, so the guards that matter drown in checks nobody trusts, and the on-call engineer mutes the lot. - Turning every transitive snapshot and pinned environment from a verification run into a permanent museum piece, then never expiring any of it.

Advance a ramp on instrumented evidence vs on the clock or a hunch

You hit this when A canary or ramp is part way out. The change is live for some slice of traffic, the rollback path is still in place, and you are standing at the next step deciding whether to widen it

The call. When you promote a ramp from one percent to the next, what actually decides the step: the signals you shipped, the elapsed time, or your read of the dashboard?

Promote on signal. Cash in the metrics you shipped with the change, reading them before each step and letting them decide. Error rate, saturation, latency, and the canary's own numbers measured against its control. The step from one stage to the next is gated on a named budget being green, over a window long enough to mean something, with the rollback wired to trip if it goes red. - Choose when: The change is risky or wide enough that promoting past a regression would cost real users or real money; You shipped the signals that move with this change, and they have a window long enough to be trusted; A rollback is still cheap at the current step, so the gate is the thing standing between a contained blip and a widened one; The blast radius grows with each promotion, so each step deserves its own look. - Cost: Slower than the calendar on a release that was probably fine, because you wait for a clean read at every step; Only as good as the signals you shipped: a gate on the wrong metric, or one with too short a window, gates on noise and you overreact to it; Demands the instrumentation be in place before the ramp starts, which is work done up front when the change felt low-risk; Tempts you into instrument theatre, dashboards that exist to look diligent and that no promotion is ever actually gated on.

Promote on time. Advance on a fixed schedule. It has been an hour at ten percent, so push to fifty. The clock, not the data, sets the pace, and each step lands whether or not anyone has looked at what the last one did. - Choose when: The change is genuinely trivial and reversible, so a bad step is a toggle you flip in seconds and the cost of being wrong is near zero; You have no signal worth gating on and are honest that the timer is a placeholder, not evidence; The ramp is a formality on something already proven elsewhere, and the schedule just paces the rollout. - Cost: The calendar cannot see your error rate; an hour of green clock over a rising spike promotes you straight into a wider failure; Couples the decision to elapsed time, which correlates with nothing that matters about whether the step is safe; Feels like progress while telling you nothing, so it is the most comfortable way to widen a regression; Trains the team that promotion is a stopwatch exercise, and the habit outlives the one change it was harmless on.

Promote on hunch. Look at the dashboard, decide it looks fine, and push. The judgement is real but it lives in one head, leans on whatever caught the eye, and leaves no number behind that anyone else could check or that an alert could act on. - Choose when: You are the owner, you wrote the signals, and a quick eyeball genuinely does read them, with the gate still written down so it is not purely vibes; A truly exploratory ramp where no budget has settled yet and a human read is the best signal going; The step is small and reversible enough that a wrong call is cheap to take back. - Cost: "It looks fine" is a feeling with no number under it, so nobody can reproduce the call and no alert can make it for you; Depends on who is watching and how awake they are, which is exactly the operator vigilance the rollout is meant to replace; The spike you promote past is usually the one not on the part of the dashboard you happened to look at; Leaves no record, so when the next step goes wrong there is nothing to learn from about why this one was judged safe.

How to decide. Decide by who controls what widens and how far a wrong step reaches. A ramp promotion is an act whose blast radius grows with every percent, and the input driving it, real production traffic, is not yours to control. That is the signature of a step that earns a gate on evidence: each promotion exposes more users to a change you cannot fully predict, and the cost of widening a regression rises as you go. So gate on the signals that genuinely move with this change, over a window long enough to mean something, and let green decide the next step rather than the clock or the eye. The exception is narrow and worth stating plainly: where a step is trivial and instantly reversible, the gate is ceremony, and a timer or an eyeball is the honest, cheaper move, because the cost of getting it wrong is a toggle. Then spend the rigour where the property is real. The discipline is not to instrument everything, it is to ship the few signals a promotion will actually consume and then promote on those, so the decision a tired engineer makes at the next step is read off a number, not off how the dashboard happened to feel.

Reach for first. Before the ramp starts, name the one or two signals that would tell you this step is going wrong, and confirm you actually shipped them. If they exist and read clean, gate each promotion on them. If they do not, that is the work to do first, and until it is done, keep the ramp at a percent where being wrong is cheap rather than advancing blind on the clock.

Pitfalls. - Gating on a metric that does not move with this change, so the budget stays green through a regression it was never going to catch. - A window so short the signal is noise, so you halt a healthy ramp on a transient blip or chase a number that means nothing. - Building dashboards nobody promotes on, instrument theatre that clutters the views you do read and dilutes the signal that matters. - Calling a timer evidence: dressing "it has been an hour" up as a decision when nothing was actually read. - Promoting on a green eyeball while the spike sits on a panel you did not open, then having no record of why the step was judged safe. - Adding a new gate or alert without retiring a stale one, so you spend down the team's attention and end up muting the very signal you needed.

See also. Tenets (XVIII), (XX), (XXI). glossary: Cash in the metrics you shipped, instrument theatre, spend down the team's attention. phases: operating.

Automated rollback wired to a signal vs a human watching the dashboard

You hit this when A ramp is going out. You have decided to ship reversibly, and now you have to decide what triggers the retreat: a machine reacting to a metric, or a person reacting to a screen

The call. When this release starts going wrong, what pulls it back: an alert wired to the metric, or someone who happens to be watching?

Auto-halt wired to the metric that means trouble. The rollback condition is code. You pick the signal that means the change is failing (error rate above control, saturation past budget, the canary diverging) and wire the ramp to halt and revert itself the moment that signal trips, with no human in the path. The retreat fires at machine speed, the same way every time, whether or not anyone is awake. - Choose when: The failure is fast and the signal is clean: a clear metric moves before a person could read the graph and act, so the lag of a human in the loop is itself the damage; The signal already exists and tracks this change honestly, so the trigger is reading something real rather than a proxy you hope correlates; The revert is genuinely cheap and reversible (a flag flip, a ramp step back), so a false trip costs you a re-promotion, not an outage; The ramp runs out of hours: it will be live through the night or the weekend when no one is reliably on the dashboard. - Cost: A threshold tuned too tight trips on a transient blip and rolls back a release that would have been fine, and a flapping auto-rollback trains the team to distrust it and raise the limits until it never fires; It only catches the failure modes you named in advance; the novel break that no metric is watching sails straight through while everyone trusts the machine to handle it; You pay up front to build and rehearse the trigger, and an auto-revert path that has never actually run is itself a second bug waiting for the worst moment; An automatic revert mid-incident can erase the state on-call needed to understand what broke, so the rollback that saved you also costs you the diagnosis.

Human in the loop: someone watches and pulls the lever. A named person owns the ramp and watches the signals while it advances. The dashboard shows error rate, saturation, latency and the canary against control; the operator reads them and decides when to promote, hold, or revert. The judgement stays with a human who can weigh context a threshold cannot. - Choose when: The signal is ambiguous or noisy, so a yes/no threshold would either over-trip or miss it, and a person reading the shape of the data is the better detector; The failure mode is one you have not seen, so there is no metric to wire a trigger to and you are relying on a human to notice that something is simply off; The revert is expensive or partly irreversible, so you want a person to confirm the failure is real before paying the cost of pulling back; The ramp is short and supervised, advancing during hours when an owner is genuinely watching rather than nominally on call. - Cost: A person reacts in minutes, not milliseconds; for a fast failure the ramp is already at 50% by the time the operator has read the graph and typed the command; Vigilance does not survive the night. A dashboard nobody is looking at is no safety at all, and 'watch it' silently degrades into 'glance at it when I remember'; The safety lives in one operator's attention and recall, which is exactly the thing that fails at 3am: the planning you didn't do arriving with interest, paid by whoever happens to be paged; Watching is real, expensive human time spent staring at a release that is probably fine, and the same eyes cannot watch ten ramps at once.

Both: auto-halt on the clear signals, human for the ambiguous. You split the trigger by how legible the failure is. The unambiguous, fast, cheap-to-revert signals get wired to an automatic halt; the ambiguous ones page a human who decides. The machine handles the failures it can read without doubt, and the operator handles the ones that need judgement, each doing the part it is actually good at. - Choose when: The release is wide or shared enough that both a fast clear failure and a slow ambiguous one are plausible, so neither trigger alone covers the risk; You have at least one signal clean enough to automate and at least one failure mode that genuinely needs a human read, so the split maps onto real differences, not ceremony; The blast radius justifies the build: this touches the shared substrate or the irreversible, and getting the retreat wrong is expensive enough to pay for two paths. - Cost: Two trigger paths are more to build, tune and rehearse, and the seam between them is its own failure mode: the case that falls in the gap, caught by neither; The automatic halt and whatever the human does next are two control loops on the same lever; if they read the same signal on the same timescale they will fight, the revert flapping against the promote. One of them has to hold authority while the other defers; Drawing the line between 'clear enough to automate' and 'needs a human' is the hard judgement, and getting it wrong hands the machine a call it should not make or buries a human in pages it should have handled; On a small or easily reversible change this is over-engineering: you have built an incident-response apparatus for a release a single flag toggle would have covered.

How to decide. Decide by reaction speed against signal clarity, then let the blast radius set how much of it you actually build. Two questions settle the trigger. First, can the failure outrun a person: if the metric that means trouble moves faster than someone can read it and act, then a human in the loop is not a safeguard, it is a delay measured in the damage done while they catch up, and the trigger has to be machine speed. Second, is the signal clean enough to trust a threshold with the revert: a metric that genuinely tracks this change, paired with a revert cheap enough that a false trip just costs a re-promotion, points at automation, while a noisy signal or an expensive, partly irreversible pullback wants a human to confirm before you pay. Then the spend rule. The blast radius decides how much rig the retreat earns: a narrow, flag-gated change you can flip off in seconds does not justify an auto-rollback apparatus, whereas a wide release into the shared substrate, where the input on the ramp is other people's traffic, earns the clear signals automated and a human kept for the ambiguous ones. The honest default lands here: wire the fast, clean, cheap-to-revert signals to halt themselves, because vigilance does not survive the night and a safety that lives only in whether someone is watching is one nobody can reach at 3am; keep the human for the failures no metric can name.

Reach for first. Auto-halt on the one signal that unambiguously means this change is failing, error rate diverging from control, with the revert being a ramp step back. It is the cheapest correct move because it removes the dependency on someone watching for the failure you can already name, and it costs you only a re-promotion if it trips wrongly. Add a human for the ambiguous signals only once you can point at a real failure mode no threshold can catch.

Pitfalls. - Wiring the auto-halt to a metric you didn't actually ship, so the trigger reads a proxy you hope correlates and stays green through the real regression. - A threshold tuned so tight it trips on noise; the rollback flaps, the team stops trusting it, and the limits get raised until it never fires when it matters. - Calling 'a human watches the dashboard' a safety measure when no one is rostered to watch overnight, so the ramp runs unguarded through exactly the hours you can't react. - An auto-revert that has never been rehearsed, discovered to be broken at the moment it first has to run. - An auto-rollback that wipes the failing state before on-call can see it, trading the incident for a mystery. - The silent fallback: an automatic halt that fires, recovers, and is noticed by nobody for a week, so the underlying failure is never investigated.

Pre-write a runbook vs invest in observability and reason live

You hit this when Something you shipped will break at 3am for someone who does not own it. The question is not whether the page fires but what the person who answers it has to work with: a procedure you wrote in daylight, or a clear picture and the authority to think. You are deciding now, while it is cheap, how legible the response will be later, when it is not

The call. For the failure modes of this system, do you turn each one into a written procedure decided in advance, or do you spend the effort on signals and clear ownership so a capable human can reason it out live when it breaks?

Runbook: a procedure per known failure mode. For each failure you have seen or can clearly foresee, you write down what broke looks like and the first lever to pull, kept per service for whoever is on call. The decision is made once, in daylight, by someone with the context, and the 3am response becomes a checklist rather than an investigation. The signal points at the fault; the runbook says what to do about it. - Choose when: The failure mode is known and recurs, and the right response is the same each time, so writing it down once spends the thinking when you have context and saves it when you don't; The lever is non-obvious or destructive, the kind of step you would not want anyone reverse-engineering under pressure; Whoever gets paged does not own the system and has less in their head than you do now, so the procedure is the only context they can reach; The cost of a wrong move at 3am is high and the failure is frequent enough that the page is a matter of when, not if. - Cost: A runbook for a failure that never fires is dead documentation: it rots out of step with the system and misleads the one time someone trusts it; Writing and maintaining procedures is real work that competes with shipping, and the maintenance never ends while the system keeps moving; It only covers what you foresaw; the genuinely novel failure has no page, and a team drilled on checklists can freeze when the script runs out; A stale runbook is worse than none, because it carries the authority of having been written and the operator follows it off a cliff.

Observability + clear ownership: signals plus a named human who can decide. You spend the effort on making the system legible (tenet XVIII) and on naming an owner of record, then trust a capable human to reason from the signals when it breaks. The dashboard says what broke and where; the owner has the authority and the context to decide the fix, including for failures nobody wrote down. - Choose when: The failure space is open or the system is young, so you cannot enumerate the modes and any runbook would be guesswork dressed as procedure; The signals are good enough that what broke is visible at a glance, leaving the human to decide what to do rather than first work out what happened; The failures are novel or varied, where judgement beats a checklist and a script would only mislead; An owner with real context is reachable when it pages, so the authority to act live actually exists and is not a fiction. - Cost: A human reasoning live is only as fast as their context, and the on-call who does not own the system pays the planning you didn't do, arriving with interest; Observability you ship and never read is its own cost, and the signal that matters is the one you forgot to instrument; It leans on a specific person being awake, reachable and clear-headed, which is exactly what 3am erodes; ownership on paper is not the same as someone who picks up; For a known recurring failure this re-derives the same fix every time, burning the steepest-cost minutes on a decision you could have made once in daylight.

Improvise: whoever is paged works it out from scratch. Nothing is written down and nothing is named in advance. When it breaks, whoever holds the pager investigates the system cold and invents a response under pressure. The outcome depends entirely on who happened to be on call and how much they happen to know. - Choose when: The system is a throwaway or pre-traffic, where the blast radius of a bad night is trivial and writing anything down would outweigh the failure it guards against; The whole team owns it deeply and any of them can reason it out cold, so the runbook and the named owner would just restate what everyone already carries; It is genuinely too early to know the failure modes and even the signals are not worth wiring yet. - Cost: The 3am improvisation is the planning you didn't do, arriving with interest, paid at the moment stakes are highest and context lowest; The result is a lottery on who got paged, so the same failure resolves cleanly one night and becomes an outage the next; Nothing is learnt structurally: each incident starts from zero because no procedure or signal was left behind for the next person; The unowned system is the one nobody dares touch mid-incident, because no one knows who is authorised to decide or what is safe to change.

How to decide. Decide by who answers the page and how wide the blast radius is. For a known, recurring failure on a shared system whose pager lands on someone who does not own it, write the runbook: you are moving the decision from the moment of least context to the moment of most, and a checklist is what a tired stranger can actually execute. For an open or young failure space, or one where the modes are too varied to enumerate, spend the same effort on signals and a named owner instead, because a runbook for a failure you cannot foresee is guesswork that will mislead, and good observability plus a person with authority is what lets a human handle the case nobody wrote down. The two are not rivals so much as a split by foreseeability: write procedures for the failures you have actually seen or can clearly foresee, and invest in observability and clear ownership for the rest. Improvise only where the blast radius is trivial. The tie-break is the spend rule: commit the writing only where the failure is real and recurring enough that the cost of getting it wrong live exceeds the cost of the page you wrote in daylight. What you are minimising throughout is what the 3am operator must hold in their head: the runbook holds it for them when the path is known, and observability lets them rebuild it fast when it is not.

Reach for first. Name an owner of record and ship the signal that says what broke. That is the cheapest correct move and the prerequisite for both richer options: until what broke is visible and someone is on the hook to decide, a runbook has nothing to point at and a live responder has nothing to reason from. Add a written procedure the moment a failure mode recurs or its fix turns out to be non-obvious or destructive.

Pitfalls. - Writing runbooks for failures that have never fired, so the folder fills with pages that rot, drift from the system, and mislead the one operator who trusts them. - Treating the runbook as a substitute for thinking: an operator follows a stale procedure step by step into an outage because it carried the authority of having been written down. - Shipping observability you never read, the instrument theatre that looks like diligence while the one signal that mattered was never wired, so the dashboard is green as the system burns. - Naming an owner on paper who is not actually reachable or empowered at 3am, which is improvisation with a label on it. - Leaning on live reasoning for a failure that recurs nightly, re-deriving the same fix at the steepest-cost hour instead of writing it down once. - Letting a runbook with no expiry outlive the system it described, so it now documents a service that no longer behaves that way. - Drilling a team only on checklists until they freeze when the genuinely novel failure arrives and there is no page to follow.

Aim the retro at the system vs at the person

You hit this when The fire is out and you are deciding what the retrospective is allowed to conclude. The people in the room half-remember why it broke, and someone's name is attached to the keystroke that did it. What you let the meeting say about that person decides whether the next failure of this class shows up loud or hides

The call. When the incident is over, should the retro's conclusion point at the structure that allowed the failure, at the person who erred, or split the difference: no blame for the slip, a named owner for the missing guardrail?

Blameless by construction. Frame every finding as a question about the system: what did the structure let happen, and what shape made the wrong thing easy and the right thing hard. The deploy that took down production is read as one unguarded command, not one careless hand. The fix lands in the shape of the system, so the next person avoids the whole class knowing nothing about tonight. - Choose when: A competent person made a reasonable call given what they could see on the night; You want the team to keep reporting near-misses rather than learning to hide them; The failure mode is structural and recurring, so the durable fix is a guard, a deleted state, or a signal, not a stern word. - Cost: Says nothing on its own about who carries the missing guard now; left loose, real ownership can hide behind 'the structure failed'; Reads as soft to anyone expecting consequences, and needs a sharp structural finding to earn that trust; Asks more of the room than naming a culprit does: you have to reconstruct the path and name the gap, not just point.

Blame the person who erred. Find who did it, attach the consequence, and move on. The lesson is that this individual must be more careful, and the record closes on their name. - Choose when: The act was genuine misconduct, not a slip: a knowing bypass of a control that was present, clear, and enforced; An individual pattern of recklessness survives every structural fix you have already tried; Almost never for an honest error, because the cost below outweighs whatever the verdict buys you. - Cost: Teaches everyone to hide the next failure, and the signal a retro exists to surface goes dark; Fixes nothing in the system: the same trap stays armed for the next on-call who walks into it; Mistakes the hand for the cause, so you spend the meeting on a verdict and leave the structural gap standing; Cheap and satisfying on the night, which is exactly why it gets reached for when it should not be.

Blameless on the slip, accountable on the gap. No blame for the human error; a named owner and a date for the missing guardrail. The person is not the cause, and the unowned structural gap still gets an owner of record like every other follow-up. Blameless is not consequence-free for the system. - Choose when: The honest case, almost always: an error happened and a guardrail was missing, and both are true at once; The team confuses 'blameless' with 'nobody owns the fix', so the gap drifts unowned until it recurs; You need the reporting culture of blamelessness and the follow-through of accountability in the same meeting. - Cost: Two things to hold at once, and under time pressure the room collapses it into one or the other; The accountability half is theatre unless the owner and date are real and tracked, not a slide; Drawing the line between an honest slip and the rare knowing breach takes judgement, and that line is where the meeting can stall.

How to decide. Decide by who controlled the input and how wide the gap's blast radius is. The person at the keyboard did not control the conditions that made the wrong thing easy; the structure did, so the durable finding is the structure, and that is why blameless is the default rather than a kindness. But blameless settles only the question of fault, not the question of ownership. The missing guardrail is the input an unlucky future caller will trip over next, and the wider the blast radius of that gap, the more it needs a named owner and a date attached on the night, not a shrug that reads as fixed to everyone who was not there. So the honest answer is almost always the third: no blame for the slip, because punishing it drives the signal underground and starves the next retro, and a real owner for the gap, because an unowned structural hole recurs on schedule. Reserve blaming the person for the rare case that was misconduct rather than error, a knowing bypass of a control that was present and enforced. Spend the ceremony of consequence only where the act was genuinely a choice; everywhere else, spend the effort on the structure, because that is where getting it wrong costs you the most.

Reach for first. Ask the one question that fixes the frame for free: if the most careful engineer on the team had been at the keyboard, would the structure still have let this through? If yes, the finding is the structure, and you have your blameless framing before anyone reaches for a name. Then, in the same breath, give the gap an owner and a date so the blamelessness does not become a place for ownership to hide.

Pitfalls. - Letting 'the system failed' become the spot where a real ownership question quietly disappears, so the gap goes unowned and recurs. - Treating blameless as consequence-free for the system: no name on the slip and no name on the fix either, which is a shrug with better manners. - Blame-hunting under the banner of accountability, which teaches the team to hide the next near-miss and kills the reporting you depend on. - Closing on a culprit while the structural trap stays armed, so the same failure walks in behind a different name next quarter. - Heaping ceremony on the owner of every tiny gap until people route around the retro itself, and the heavy controls stop meaning anything where they matter.

See also. Tenets (XXV). glossary: blameless by construction, bureaucracy erodes trust, process is structure when code structure runs out. phases: retrospective.

Turn the lesson into a self-rerunning guard vs leave it in prose

You hit this when The retro is over and a real lesson came out of it. You have pinned a cause, agreed what should never recur, and now you are deciding what shape that knowledge takes once everyone leaves the room. The choice is whether the lesson re-runs on its own forever or lives as a paragraph someone has to find and recall at the worst possible moment

The call. A retrospective produced a lesson. Do you encode it as something that fails by itself when the failure mode returns, or do you write it up and trust people to carry it?

Encode it as a self-rerunning guard. Promote the lesson to a mechanism that trips on its own: a regression test built from the failing input, an alert on the metric that moved first, a pipeline check that rejects the bad shape at merge, a constraint that makes the state impossible. It fires in CI or in staging without anyone remembering the incident, and it outlives the team that learned it. - Choose when: The failure has a recognisable signature you can express as an input, an invariant, or a shape a check can reject; Recurrence is plausible and the cost of the next escape is real: money, data, or a customer-facing outage; The class of failure is broad enough that a tired engineer next year will plausibly walk into it knowing nothing of this incident; You can name the invariant that failed, not just the one run that exposed it. - Cost: A guard is state you now own: it can flake, age, and need maintenance like any other code; A test pinned to an incidental detail breaks on unrelated changes, and a flaky guard gets muted or deleted, taking the lesson with it; Writing the reproduction and the assertion costs more than a paragraph, sometimes more than the original fix; An alert with a poor signal-to-noise ratio trains the team to ignore it, so you spent signal and bought noise.

Leave it in the write-up. Record the lesson in the retro document or the runbook as prose, and rely on people reading it, remembering it, and applying it when the moment comes. The narrative explains what happened and what to watch for, and the defence is human attention. - Choose when: The lesson is genuinely judgement, not a mechanical rule: a heuristic about when to escalate, a trade-off with no single right answer; No check could express it without encoding a falsehood, and any guard you wrote would be theatre; It belongs in a runbook because the next responder needs a procedure to follow, not a gate that blocks them; The failure is a one-off tied to a context that is being deleted anyway, so encoding it guards a ghost. - Cost: Prose decays at the speed of attention: the next engineer never attended the retro and may never find the document; Nothing enforces it, so the lesson holds only until the person who remembers it leaves or is busy; You have left correctness in human vigilance, which is the exact thing the system is meant to take off people's shoulders; A write-up with no filed work has turned the loop back into a line; the lesson terminates in a document instead of changing the structure.

Assert on the invariant, not the surface. Encode it, but pin the guard to the rule that was actually violated rather than the incidental shape of the one run that exposed it. The check asserts the contract or the decision, so it survives a refactor and keeps protecting the thing it was built for instead of breaking on every unrelated change. - Choose when: You have decided to encode the lesson and now have to choose what the guard watches; The failing run had incidental detail (a specific id, a private helper, a log format) that is not the actual rule; You want the guard to survive the refactors this house style wants to keep cheap; The invariant is nameable at a boundary or a decision, separate from the implementation that happened to break it. - Cost: Finding the real invariant is harder than copying the failing call: it takes thought the day after an incident, when energy is low; Asserting too abstractly can let a near-miss of the same class slip through the gap between rule and surface; A boundary-level test sometimes needs more scaffolding than a quick assertion on the internals; If you misjudge the invariant the guard passes while the failure mode quietly returns, which is worse than no guard because it reads as safe.

How to decide. Decide by who controls the recurrence and how wide the next escape spreads. If the failure mode can return on input an attacker or an unlucky caller controls, or if its blast radius is money, data, or a wide outage, the lesson belongs in a guard that fails by itself; prose cannot bound a blast radius, because it does nothing until a human acts, and the human is exactly who was tired and wrong the first time. Spend the ceremony where the property is real: a reproduction promoted to a test that re-runs the rule forever earns its maintenance once the cost of the next escape exceeds the cost of owning the guard. Where the lesson is genuine judgement that no check could express without lying, encoding it is theatre, and a runbook entry is the honest move. When you do encode, assert on the invariant rather than the surface, so the guard survives the refactors this house wants to keep cheap and keeps protecting the thing it was built for. The tie-break lands here: minimise what the next engineer must hold in their head to avoid this class of failure, and prefer the structure that needs no memory at all, but do not pay for a guard whose property is not real or whose noise will get it muted.

Reach for first. The cheapest correct move is to promote the reproduction you already built to pin the cause into a regression test, asserting on the invariant that failed rather than the surface of that one run, and file it as a tracked issue with an owner and a date. You did the expensive part during the investigation; wiring it into the run so it fires on its own is the small remaining step that converts a recollection into a guardrail.

Pitfalls. - Filing an action item that reads 'be careful with null tenants': no machine enforces it, so it is a guess dressed as a fix. - Pinning the test to the exact id, log line, or private helper from the incident, so it breaks on the next unrelated refactor and gets deleted in frustration. - Adding an alert that also fires on every deploy, so the team mutes it and the next real instance scrolls past unseen. - Asserting so abstractly that a near-miss of the same class slips through the gap between the rule and the surface. - Polishing the narrative until the document feels like the deliverable, when the deliverable is the owned, triage-ready work that prevents recurrence. - Encoding a guard for a context that is being deleted anyway, so you now maintain a check that protects a ghost. - Leaving the guard unowned and undated, so it is theatre that evaporates the moment the meeting ends.

Feed findings back as owned issues vs publish a write-up

You hit this when The retro is winding down. The room agrees on what broke and roughly what should change. Now you decide what actually leaves the meeting: tracked work that re-enters the lifecycle, or a document that describes it

The call. When the retro ends, does the front of the lifecycle get new owned work, or does it get prose?

File owned, dated, triage-ready issues. Each finding leaves the room as a tracked issue with one named owner, a date, a reproduction or acceptance criterion, and a rough sense of blast radius. The regression test, the new alert, the deleted flag, the deprecation: each is scoped work sitting at the front of Triage, ready to be planned like any other change. The runbook is updated in the same breath. - Choose when: The findings name structural change someone has to do: a test to add, a flag to delete, a guard to remove, a runbook gap to close; This class of failure will recur unless something in the system changes, so the lesson has to outlive the people who were on the call; You want the next on-call, who never attended, to inherit the fix through the procedure and the guardrail rather than through memory. - Cost: Filing real issues is more work than writing a paragraph: each needs an owner who agrees to it, a date, and enough detail to act on cold; An owner and a date are a commitment, and a list of twenty with no slack ships none of them, so you have to prioritise hard and drop the rest in the room; The narrative of what happened, the single source the issues link back to, still has to be written somewhere or the issues lose their shared context.

Produce a report and consider the retro done. The output is a written account: timeline, cause, what we learned, recommendations. It reads well, it circulates, and closing it is closing the document. Any work it implies is left for someone to pick up later. - Choose when: The audience genuinely needs the story more than the fix: a post-incident review for stakeholders, a regulatory record, a teaching artefact for people outside the team; The findings are about understanding, not structural change, so there is no test to add or state to delete, only a thing now better understood; The actual remediation is already filed elsewhere and this document exists only to explain it. - Cost: A retro whose only artefact is a write-up has turned the loop back into a line: the lesson terminates in a document instead of flowing into the next cycle; Recommendations with no owner and no clock are theatre that reads, six months on, exactly like work that was never started; The polished document creates a false sense of closure, so the same class of failure reopens the same retro while everyone believed it was handled; Prose decays at the speed of attention, and the next engineer relearns the lesson in production because nothing in the system stops them.

Write the story once, but make the issues the deliverable. You write the narrative a single time as the authoritative account, and you file the owned, dated issues as the real output. The document exists to explain the issues and give them shared context; the issues exist to change the system. The report links to the work, not the other way round. - Choose when: The incident is worth understanding and worth fixing, which is most of them; Several issues share one cause and would each repeat the same background, so a single linked source keeps them from drifting out of sync; You want both the audit trail and the structural change, and you are willing to keep the by-product subordinate to the deliverable. - Cost: It is the most total effort: you write the story and file the work, and you have to hold the line that the document never substitutes for the issues; Two artefacts can drift, so if the issues evolve and the narrative does not, the doc starts to lie about what was actually done; Polishing the narrative is the seductive part, and time spent there is time not spent filing the work that prevents recurrence.

How to decide. Ask what each finding is for. If it names a structural change, a test, an alert, a deleted state, a runbook step, then its correct home is the front of the lifecycle as owned, dated work, because that is the only form a lesson takes that survives the people who were in the room and reaches the engineer who inherits this next year. That engineer is the one who controls nothing here: they never attended, and they meet the lesson only through whatever the system carries forward for them, so a guardrail reaches them and a paragraph does not. A document cannot fail in CI, cannot page anyone, and cannot delete a flag; it can only ask a human to remember, which is the exact failure the retro exists to remove. So the deliverable is the issues. Where the story has real value, as an audit trail, a stakeholder record, or shared context for issues that share a cause, write it once and link the work to it, but keep it the by-product. The blast radius is the next iteration: getting it wrong means the same class of failure recurs because nothing structural changed, and the cost of that recurrence dwarfs the cost of filing the issues properly. Spend the ceremony, an owner and a clock on each item, only on the few findings that close this class of failure, and drop the rest in the room rather than parking them in a backlog that rots.

Reach for first. Before anyone touches a document, file the one or two issues that close this class of failure, each with a named owner, a date, and the reproduction. That alone makes the retro a loop rather than a line. Write the narrative afterwards if it earns its keep.

Pitfalls. - Closing the retro when the document is done, so a polished write-up stands in for the work and the loop quietly stays a line. - Recommendations phrased with no one on the hook and no clock running, which read identically to follow-ups that were never made. - Filing twenty owned items with no slack, so the list ships none of them and the few that mattered drown with the rest. - Letting the narrative and the issues drift apart, until the document describes a fix that the issues no longer match. - Skipping the runbook update, so the next responder inherits the understanding only by having attended a meeting they were not in. - Treating an accepted risk as closed rather than as a dated decision record naming who accepted it and when to revisit.

Gate removal on a usage signal and guard the absence vs trust the belief nothing uses it

You hit this when You are retiring a feature, endpoint, flag, table or service, and the deletion is the one change you can't have another go at. The thing has callers you never met: a cron job in another team's repo, a dashboard query, a client three releases behind. The question on the table is not whether to remove it but how you know it is safe, and how it stays gone once you have pulled it

The call. How do you establish that nothing still depends on this, and how do you keep it removed?

Earn the signal, then guard the absence. Instrument the entry point with a counter or a log line, watch live traffic across a full usage cycle, grep the dependency graph, and remove only once the telemetry reads cold. Then encode the absence as a check: an alert on traffic to the dead route, a test that fails if the symbol or migration recreates it. - Choose when: The thing is reachable by callers outside your repo, or by clients you cannot force to upgrade; The removal is irreversible or wide: a table, a public endpoint, a shared service; A resurrection is plausible, by a revert, a copied migration, or a teammate re-adding the endpoint not knowing it was killed on purpose; You can name the real usage cycle, month-end, the quarterly batch, the seasonal spike, and afford to watch through it. - Cost: You carry the instrument and wait a real cycle before you can act, so the removal is slow to land; A rarely-used path can stay quiet for a quarter and then fire, so the window has to match the usage cycle, not the calendar you would prefer; The guard is itself state you now own; a removal-guard that outlives the risk it guarded is the cruft this phase exists to delete, so give it an owner and a sunset.

Trust the belief that nothing uses it. Reason from your mental model of the system, conclude nothing depends on it, and delete on that conviction with no signal and no guard behind it. - Choose when: The thing is fully inside a blast radius you control end to end, with no caller you cannot see; The cost of being wrong is a cheap revert, not an incident: a dead local branch, scratch state nobody reads; You genuinely own every consumer and can change them all in the same change. - Cost: "Nothing uses this" is a claim about a world you do not control, and the dependency you only believe is gone is the one a caller finds the moment the lights go out; The missed caller does not disappear; it surfaces as the 2am page, at the worst possible moment, with no telemetry to have warned you; Nothing defends the absence, so a revert or a copied migration brings the thing back and nobody notices until it breaks again.

Sampled search only: grep for callers and ship. Search the main repo for references, find none, and treat that as proof of safety. No instrument, no waiting, sometimes a guard, but the decision rests on the search. - Choose when: The codebase is a single repo with no external consumers, so the search really is the whole dependency graph; You need a fast answer and a wrong one is cheaply reversible; As the first cut, to be confirmed by telemetry before the irreversible drop, not as the gate itself. - Cost: A repo search is a biased sample that systematically omits other teams' services, dashboards, scheduled jobs and stale clients, exactly where the surprises live; Absence of a hit is not absence of a caller; you have proven nothing about the consumers outside the place you could grep; It feels like proof, which is its real danger: it gives the confidence of a signal without the coverage of one.

How to decide. Decide by who controls the input and how wide the blast radius is. If every caller sits inside a boundary you own and the worst case is a cheap revert, conviction or a repo search is proportionate, and the waiting and guarding would be ceremony you should not pay for. The moment a caller is outside your repo, or a client you cannot force to upgrade is in the picture, or the drop is irreversible, your belief is a claim about a world you do not control, and a grep is a sample that misses every other consumer. There you earn the signal: instrument, watch a real usage cycle, and let observed silence become evidence rather than guessing. Then bound the resurrection by encoding the absence as an alert or a test, so an accidental revert trips a wire instead of quietly undoing the removal. Spend the instrument-and-wait and the standing guard only where the property is real and the cost of getting it wrong, the page nobody warned, exceeds the cost of the work.

Reach for first. Add the counter or log line at the entry point before you do anything else. It is the cheapest correct move: a one-line instrument turns "I think nothing uses it" into a fact you can watch accrue, and it costs you nothing while you get on with planning the contraction. Grep first if you like, but let it narrow the search, not close it.

Pitfalls. - Sizing the observation window to the calendar you want rather than the real cycle: a week of zero proves a high-traffic surface is dead, but a year-end job stays silent for eleven months and then fires. - Treating a clean repo grep as the signal. The callers that break are the ones in the repos you could not search, so the search that finds nothing is the search that misses them. - Removing the code but leaving the guard you never sunset, or the guard you never built. A removal with no resurrection check quietly undoes itself; a guard with no owner becomes the next zombie. - Deleting on conviction because a revert is "cheap", forgetting that the revert is cheap only until a second caller has come to depend on the thing being back. - Shipping the guard as a test coupled to the old implementation rather than to the absence itself, so it breaks on unrelated refactors and gets deleted out of annoyance.

For the principles these decisions compress, see MANIFESTO.md and the companion THE-SHAPE-OF-THE-WHOLE.md; for how they play out by phase, the guides in howto/; for the phrases, the glossary.