How to ship and operate

A release is the point where a change stops being yours and becomes everyone's, and after that it runs by the clock and the load rather than by whether you happen to be watching. So the retreat has to be built into the shape of the rollout instead of living in what the operator remembers to do. Any safety you keep only in your head is one nobody can reach at 3am. This is MANIFESTO.md pointed at shipping and operating, the moment where the code tenets meet real traffic and the process layer (tenet XXV) is all that's left holding things up. The objective hasn't changed, but for this phase it reads like this: release the change so a mistake is cheap to undo, and keep it healthy once it's live, subject to bounding the blast radius of the irreversible and the shared parts of the system. When two of the moves below pull against each other, you don't settle it with a slogan. You settle it on who controls the input and how wide the blast radius is. None of it is free.

Ship reversibly, behind a flag, ramped and canaried

A release you can only undo by shipping again is a cliff; the one you can toggle off is a step you can take back.

The deploy is going to be wrong about something. The only thing in question is what being wrong costs you. So sequence the release for a cheap retreat (tenet XX): expand-contract so old and new run side by side, a flag so the change stays off until you say so, a canary that hits real traffic before anything else does, and then a ramp from 1% to 100% with the rollback path already in place before that first percent goes out. An instant cutover isn't bold. It's a release that has no losing move, and at some point you are going to need a losing move.

A feature: flag-gated and ramped 1% to 100%, so a bad release is a toggle, not a launch you can only undo by shipping a hotfix.
A schema change: expand, backfill, dual-write, switch reads, then drop, every intermediate state runnable, rather than the one-transaction cutover that is a cliff.
A binary rollout: canary on a slice of the fleet and watch vs flipping the whole fleet at once and discovering the regression with every user at the same time.

Tension: This trades against simplicity (tenet XXI). Every flag and every dual-write path is just clutter the moment the ramp is done, so only add the seam when the change is actually risky or wide, and the day you add it, give it a date to die. A flag on a one-line fix you can trivially reverse is ceremony, not safety.

Ask yourself: If this release is wrong at 100%, is the fix a toggle I can flip in seconds, or a build-and-deploy I have to survive while it burns?

Promote on evidence from the signals you shipped

Promote the ramp when the data you instrumented says it is safe, not when the hunch or the calendar says so.

A ramp you advance on a feeling is a ramp you advance blind, and the calendar doesn't know the first thing about your error rate. The change went out with the signals that make it legible (tenet XVIII): error rate, saturation, latency, and the canary's own metrics measured against its control. Read those before each promotion and let the data decide the next step, so "it's been an hour, push it to 50%" turns into "the canary's error rate matches control, promote". The signal you didn't ship is the regression you promote straight past.

A canary: promote when its metrics track the control group vs promoting on elapsed time, a clock that cannot see the spike you are about to widen.
A ramp step: gate 10% to 50% on a green error and saturation budget vs a hunch that "it looks fine", a feeling with no number behind it.
A rollback trigger: an alert wired to the metric that auto-halts the ramp, not a human noticing the dashboard went red an hour after it did.

Tension: This pulls against speed. Waiting for a clean signal at every step slows down a release that is probably fine, and a metric with too short a window is just noise you'll end up overreacting to. Gate on the signals that genuinely move with this change, give them a window long enough to mean something, and don't build instrument theatre you will never read.

Ask yourself: What signal would tell me this ramp is going wrong, did I ship it, and am I promoting because that signal is green or because I am impatient?

III

Separate the decision to release from the act, and force a second human on the irreversible

Make the rollout a reviewed, dry-run plan; put four eyes on the destructive step, with the backout written beside it.

The catastrophic verbs of operating, cut the release, run the migration, delete the data, get their danger from being welded to the moment someone decides to run them. A release that decides and acts in a single keystroke can't be reviewed without paying to run it. So split them (tenet XI): the rollout plan is the thing you review, dry-run it first, and the apply is the thin step that just executes a plan that's already approved. For the irreversible step, the destructive migration or the data delete, process enforces the four-eyes that the seam made possible (tenet XXV), with the backout procedure written down right next to the forward one.

A rollout: a reviewed plan dry-run against staging, then applied vs typing the deploy command live and finding out what it does in production.
A destructive migration: a named second approver signs before the DROP runs, not one engineer alone deciding the irreversible at 2am.
A data delete: the restore path written and tested beside the delete, rather than "we have backups somewhere" discovered to be untrue after the rows are gone.

Tension: The split lets the plan go stale between sign-off and apply (a TOCTOU window, tenet IX), because the fleet keeps moving while the plan sits in review. Pin the plan to the state it was reviewed against and re-check the canary and ramp position from Sections I and II at apply time, so a plan approved at 1% doesn't apply itself blind to a fleet that has moved on since. Skip the seam only where the act is cheap and reversible. A sign-off gate on a flag toggle you can flip instantly is friction, not control.

Ask yourself: Can this release be reviewed to death without running it, and for the step I cannot take back, has a second human signed and is the backout written down?

Degrade in tiers; contain the blast radius

The degraded path you never rehearsed is the one that fails when the dependency finally dies; wire the tiers to live signals and let on-call see which tier dropped.

A system that treats every dependency as load-bearing fails whole the first time any one of them so much as blinks, and the fallback you wrote but never actually ran fails right along with it. Decide the tiers ahead of time (tenet XIX): when the recommender dies the page still renders without it, and when a non-critical store is slow the request drops the nice-to-have and serves the core. Wire each tier to the live signals from Section II so it trips on a metric that's actually moving instead of a guess, and emit which tier degraded so on-call gets the answer straight from observability rather than having to reverse-engineer it. What separates an incident from an outage is whether the degraded path was built and rehearsed before anyone needed it.

A dependency: render the page without the recommender when it is down, a fallback decided in advance, not a 500 because one optional call timed out.
A noisy tenant: a per-tenant quota that throttles the one greedy customer vs a shared pool where their spike starves everyone else's requests.
A slow backend: serve cached or partial results behind a bulkhead vs letting one saturated dependency consume every worker thread and take the service down with it.

Tension: A degraded path you never test or rehearse is just a second bug waiting for the worst moment to fire (tenet XXIV). A fallback with no test and no game-day is code you are trusting blind, and it will misbehave at exactly the wrong time, right when the dependency it's meant to cover for goes down. Build and rehearse the tiers for the failures that actually happen, and write each fallback's behaviour down as a test. For the rest, accept the risk out loud rather than hiding it behind a fallback nobody has ever exercised.

Ask yourself: When the riskiest dependency fails right now, what is the smallest thing we lose, and can one tenant's bad day reach a second tenant?

Bound the load you create on others

Retry with backoff and jitter, break the circuit, and put a deadline on every cross-boundary wait, so a blip never becomes a retry storm.

A client that retries hard and waits forever takes a downstream blip and turns it into an outage it did to itself. The failing dependency comes back up to a thundering herd and falls over again. So bound the load you create (tenet VII), and then own the retry lifecycle (tenet XIX): retries with exponential backoff and jitter, so the fleet does not all retry at the same instant, and a circuit breaker that stops you hammering a dependency that is already down. Every cross-boundary call needs a deadline too, or a slow dependency just pins your threads with nothing to show for it (tenet VI). An unbounded retry is the queue that melts the fleet, only it is wearing a helpful name.

A failed call: retry with capped exponential backoff and jitter vs a tight retry loop that becomes a DDoS against a dependency that is merely slow.
A dependency that is down: a circuit breaker that fails fast and sheds load, rather than every request piling onto a backend that cannot answer.
A cross-boundary wait: a deadline that frees the thread when the budget is spent vs an unbounded wait that holds resources until the whole pool is exhausted.
Inbound overload: shed work whose deadline will expire before you can serve it and return an explicit "full" as a 503 (tenet V), not a deep buffer that just delays the rejection.

Tension: This trades against completeness. A deadline that is too tight fails requests that would have succeeded, and a circuit breaker can trip on a transient blip and shed load you could have served. Set the budgets from the latency you actually observe, not a guess, and tune the breaker so it opens on real failure, not noise.

Ask yourself: If the dependency I call gets slow or flaps, does my client back off and give up gracefully, or do I amplify the failure back at it until the fleet melts?

Finish in-flight work before the instance exits

On shutdown, stop taking new work, drain the in-flight within a bounded grace window, and ack only what is durable, so a rolling deploy drops zero requests.

An instance that exits the moment it is told to just drops whatever it was holding, and if a rolling deploy does that to one instance after another it ends up shedding requests all day. Treat shutdown as a resource you have to release cleanly (tenet XII): on the signal, stop accepting new work, let the in-flight requests finish inside a bounded grace window, flush buffers and ack only what has really been written down, then exit. The deploy that drops requests is the one that mistook "process gone" for "work done".

A rolling deploy: drain connections within the grace window, then exit, so a graceful restart is invisible to callers vs a hard kill that 500s every in-flight request.
A queue consumer: finish and ack the message in hand before exiting, rather than dropping it half-processed and relying on redelivery to paper over the gap.
A buffered writer: flush to durable storage and confirm before acking vs acking on enqueue and losing the buffer when the instance dies mid-flush.

Tension: This trades against shutdown speed. A grace window long enough to drain everything is a window an autoscaler or orchestrator may not grant, and a wedged request can hold the whole instance open. Bound the drain: finish what you can inside the window, then force-exit, because a deploy that hangs forever waiting to be polite is its own outage.

Ask yourself: When this instance gets the shutdown signal mid-request, does the caller ever see a dropped connection, and is anything acked that was not actually made durable?

VII

Every flag, cache, quota, and migration has an owner and an expiry

The zombie flag nobody removes and the unowned service nobody dares touch are the 3am problem; give each a named owner and a sunset.

Process is the structure you fall back on when the code structure runs out (tenet XXV). A flag with no expiry is the zombie branch that springs back a year later, and a cache with no owner is the stale copy whose invalidation path nobody can remember any more (tenet XIV). So give every operational artefact one owner of record, and then "who owns this?" is never the first question of the incident. Give it a sunset too, or the running system silts up with scaffolding nobody dares delete (tenet XX). The flag you keep forever is the one that fires when you have forgotten what it does.

A flag: an owner and a removal date attached the day it ships, removed when the ramp completes vs a permanent branch nobody remembers the purpose of.
A cache or quota: a named owner and a staleness budget not a derived copy with no invalidation path back to the truth and nobody accountable for it (tenet XIV).
A migration's leftovers: the dual-write path and old column scheduled for deletion, not dead scaffolding left running because no one owns tearing it down.

Tension: This is about accountability, not bureaucracy. An expiry on a flag that genuinely must stay is a false alarm you will learn to ignore, which trains you to ignore the real ones. Set sunsets that mean something, and for the long-lived flag make "permanent, owned by X, reviewed quarterly" the explicit decision rather than the silent default.

Ask yourself: For every flag, cache, quota, and migration still running, can I name its owner and its removal date, or is it scaffolding that has quietly become load-bearing?

VIII

When it breaks at 3am, the runbook and the owner decide, not improvisation

Observability says what broke, the runbook says what to do about the known failure mode, the named owner decides when it is novel.

When an incident gets handled by improvisation, the outcome just depends on who happened to be paged. Make the response legible before you need it. The signals you shipped (tenet XVIII) tell you what broke and where, and the runbook turns each known failure mode into a procedure that was decided in advance rather than invented fresh under pressure, and the named owner (tenet XXV) is who decides when the failure is one nobody wrote down. The 3am improvisation is the planning you didn't do, arriving with interest.

A known failure mode: the runbook says exactly which lever to pull vs an on-call engineer reverse-engineering the system under pressure at 3am.
A blind spot: the dashboard points at the saturated dependency, not a guess at what changed because nothing was instrumented to say.
A novel failure: the named owner decides and the team is paged in, rather than whoever is awake improvising a fix to a system they do not own.

Tension: This pulls against the cost of writing it all down. A runbook for a failure that never happens is dead documentation that rots and misleads, and you cannot write a runbook for the genuinely novel. Write procedures for the failure modes you have actually seen or can clearly foresee; for the rest, invest in the observability and the clear ownership that let a human reason it out live.

Ask yourself: When this pages someone at 3am who is not me, do the signals show what broke and does the runbook say what to do, or are they improvising on a system they have never owned?

The through-line

Every move here is the manifesto's one move wearing the release's clothes: push safety out of the operator's vigilance and into the shape of the rollout and the running system. A reversible ramp, promotion gated on evidence, a decision split off from the act, tiered degradation, bounded load, clean shutdown, an owner and an expiry for everything, and a runbook for the dark. The on-call engineer should be able to do the right thing while ignorant of 99% of what you knew when you shipped. Not because they remembered the trick, but because the rollout already carried it and the unsafe path was a toggle away from undone.

Build the release for whoever gets paged at 3am with less in their head than you have right now. The safe step should be the one that's obvious. The thing you can't undo should be hard to do by accident, and you should never leave them improvising something the rollout could have just done for them.