How to investigate

An investigation gets opened when something you do not yet understand has you blocked. The person running it has less context than the bug does, and it closes only once the unknown has shrunk enough that you can act. So spend the inquiry and not your luck. Drive at the answer with some method, rather than hoping that staring harder at it will work. This is MANIFESTO.md turned on investigation - not a tidy stage you pass through but a mode you drop into mid-flight, and it has two faces. The forward face is the SPIKE, where you de-risk an approach you have not built yet. The backward face is the DEBUG, where you diagnose a defect that already exists. The objective is the same as ever, just restated for this phase: reduce the single unknown that is blocking you for the least spend that produces a trustworthy answer, subject to bounding how far the investigation itself can dig before it owes a result. When two of the moves below pull against each other, no slogan resolves it. What resolves it is which unknown is load-bearing, and how wide the blast radius would be if you acted on a guess and got it wrong. None of this is free.

Name the one question, and time-box the answer

An investigation with no stated question and no clock is a rabbit hole that bills by the hour; bound the spend before you open the dig.

A spike that "explores the options" and a debug that "looks into the flakiness" both have the same defect, which is that there is no stated condition bounding the spend (VII). Turn the vague unease into one question you can actually check, and then put a clock on it, because otherwise an investigation is a queue of hypotheses that just grows and grows until something caps it (VII). Work out up front what you are willing to spend, and also what you will do if the box closes with the question still open - escalate, ship the workaround, accept the risk. The clock is not a failure. It is the move you planned in advance so that a two-hour question does not quietly eat a fortnight.

A spike: "can this API hold 2k writes/sec under our payload, answered in a day" vs "evaluate the datastore", an open-ended survey with no finish line.
A debug: "why does this one job return null for tenant 7", not "investigate the data issues", a brief that quietly grows to own every oddity in the table.
A box expiring: a written "spent the budget, here's what we know and what we'd try next", rather than the silent slide into a week nobody decided to spend.

Tension: This pulls against the duty to prove, not guess (XXIV). A clock can cut an investigation off one step before the answer, and a too-tight box rewards the plausible guess over the proven one. Size the box to the blast radius of being wrong: a reversible spike gets a short clock, a root-cause for a payments corruption gets the time it actually needs. Bound the spend, do not starve the proof.

Ask yourself: What is the single question whose answer unblocks me, and what will I do the moment the clock runs out with it still open?

Reproduce it deterministically first

You cannot fix or de-risk what you cannot summon on demand; an intermittent reproduction is half the investigation, so spend on pinning the inputs before you spend on theories.

The reproduction is the test you do not have yet (XXIV), and until it runs deterministically every result downstream of it is just noise dressed as a finding. Pin the inputs, the seed, the clock, the environment, the data fixture. Strip the variables out until the failure either appears or vanishes when you tell it to. A flaky repro is not a smaller version of a solid one - it is a different problem and a harder one, because every hypothesis you test against it inherits its randomness, so you can never really tell a fix from a coincidence.

A debug: a captured request, fixed seed, and frozen clock that fail every run vs "it happens sometimes in prod", a report you cannot interrogate.
A spike: a load harness with a pinned payload and a fixed concurrency, not a hand-run that was fast "the couple of times I tried it" under no real contention.
An intermittent failure: narrow the surface until the intermittent becomes reliable (it is concurrency, or a clock, or an ordering, tenet IX), rather than testing fixes against a coin flip and trusting the flips that agreed with you.

Tension: This trades against the time-box (VII). Chasing a fully deterministic repro can itself consume the whole box, and some failures are irreducibly statistical. Then make the rate the measurement: pin everything you can, quantify the residual ("fails 3% under this load"), and treat that number as the thing your fix must move, rather than demanding a determinism the system does not have.

Ask yourself: Can I make this failure appear and disappear on command, and if not, exactly which input am I still failing to control?

III

Form a falsifiable hypothesis; bisect, don't flail

Decide what would prove you wrong before you look, then halve the search space each step; the scattershot fix that "might help" is a guess wearing a lab coat.

A hypothesis you cannot falsify is not a theory, it is just a hope, and hope does not converge (XXIV). State the prediction sharply enough that a single observation can kill it ("if it's the cache, clearing it fixes the repro from move II"), and then design the next step to cut the remaining unknown in half rather than to confirm what you already think. Bisection is binary search over causes. Every probe should rule out half the space, so that a hundred suspects fall in seven steps instead of a hundred lucky pokes.

A regression: git bisect over the suspect range to land on the one commit vs reading every diff since the last good release, hoping the cause looks guilty.
A spike's risk: "the unknown is whether the queue survives a 10x burst; test that one thing first", not building the whole prototype and discovering the risk last.
A layered system: halve it, confirm the bad value at the service boundary before opening the database, rather than changing three layers at once and learning nothing from a fix that worked.

Tension: This pulls against the cheap fix (XXI). Sometimes the one-line "just clear the cache" resolves the symptom faster than the bisect resolves the cause, and for a reversible, low-blast-radius issue that can be the right trade. But log that you treated the symptom and left the cause unproven, so it is a recorded debt (XXV), not a mystery that returns next quarter wearing a new face.

Ask yourself: What single observation would prove my current theory wrong, and does my next step halve the search space or merely flatter my hunch?

Cash in the observability you built

Investigation is where the signals you emitted earlier pay their rent; read the trace, the metric, and the swallowed error before you add a single new probe, and treat every blind spot as a logging gap to fix.

The structured log, the trace id, the metric you instrumented were all a deposit against exactly this moment (XVIII), so spend them before you start guessing. Follow the request across the boundary by its correlation id, read the error you actually caught instead of the one you imagined catching, and let the signal you already have narrow the search for free (XIII). Where the trail goes dark, that darkness is a finding in itself. It is a place the system failed to make its own behaviour legible, and the fix carries forward as a probe you added, not as a mystery you happened to squint past this once.

A debug: pull the trace for the failing request id and read where the latency actually went vs reasoning about where it "should" be slow from the architecture diagram.
A swallowed error: find the catch that logged nothing and the exception it ate (the wrong state that learned to hide, tenet XIII), not inferring the failure from its distant downstream symptom.
A blind spot: add the missing log or span the moment you wish you had it, so the next investigation reads the answer rather than rediscovering this same dead end.

Tension: This trades against the duty to stay quiet in steady state (XVIII). Under pressure the instinct is to bolt debug logging on everywhere and ship it, and you end up leaving the system noisier than you found it. Put the probe where the trail actually went dark. Give it the same owner and the same level discipline you'd give any other signal, and once the diagnosis lands, take the scaffolding logs back out.

Ask yourself: What does the telemetry I already emit tell me before I add anything, and where it tells me nothing, is that the bug hiding or my instrumentation missing?

Distrust the narrative and the wall clock

"It started Tuesday" and "nothing changed" are unreliable inputs; trust a reproduction and a measurement over the story, and never order events by wall-clock timestamps.

The human account of when and why a failure began is a hostile input (IV). Memory compresses. "Nothing changed" really means "nothing I noticed", and the reported start date is the day someone looked, not the day it broke. Build the timeline back up from artefacts you can verify - the deploy log, the commit graph, the first bad row - rather than from the recollection that happens to feel confident. And don't trust the clock either. Wall-clock timestamps across machines skew and leap and reorder, so any sequence you build out of them is a race condition in your reasoning (XXIII) rather than a fact.

A "sudden" outage: the metric that shows the slow climb began a fortnight ago vs the story that it "started this morning", which only dates the complaint.
A "nothing changed" claim: the config diff or dependency bump that did change, not the honest belief of someone who did not touch the part that moved.
An ordering across services: reconstruct cause from a logical sequence (a trace, a monotonic counter, a causal token) vs sorting two servers' log lines by their disagreeing clocks (XXIII, IX).

Tension: This pulls against listening to your reporter (IV). The narrative is an untrustworthy fact but it makes an excellent lead. "It started Tuesday" is wrong as a timestamp and gold as a hint about which deploy to bisect first. So mine the story for where to point the verification, and then let the artefact decide what actually happened - never the recollection.

Ask yourself: Which "facts" in this report are someone's memory rather than a measured artefact, and have I ordered the events by causality or by clocks that lie?

Spike forward, then throw the spike away

The deliverable of a spike is knowledge, not code; if the prototype must survive, it stops being a spike and has to be parsed and owned like anything else that ships.

A spike buys you an answer to one question by cutting every corner that doesn't bear on that question - no error handling, no edge cases, no tests, hard-coded everything. That's the right thing to do, and it's exactly why the code becomes a liability the moment the answer is in hand. The honest end of a spike is to delete the code and keep the finding (XX). The dishonest end is the demo that "works", which quietly slides into production while keeping all its spike-grade shortcuts, so the unparsed input (III) and the unowned module (VII) are now load-bearing under traffic nobody ever hardened it for.

A throwaway probe: "the answer is yes, here is the decision record, the branch is deleted", not a wip-spike branch that drifts into main six weeks later.
A prototype that must live: stop calling it a spike and re-cut it properly, parsing inputs and owning lifecycles, rather than shipping the corner-cutting unchanged.
A captured finding: a recommendation with the numbers and the rejected options vs a Slack "yeah it'll work", a conclusion no one can audit or revisit.

Tension: This trades against the sunk cost of working code (XX). The spike runs, and deleting a thing that works feels like waste, and that's exactly the pressure that turns prototypes into unowned production. Keep whatever is cheap to rebuild from the recorded finding and throw the rest away. The code's only job was to teach you something, and it already has.

Ask yourself: Is what survives this spike a documented answer, and if any code is surviving with it, am I prepared to parse and own it as real, or am I just refusing to delete it?

VII

The output is a diagnosis and a failing test, not a fix

Separate finding the cause from fixing it; hand the next phase a proven root cause and a red test, and record what you ruled out so no one re-runs the dig you just finished.

The investigation ends when the unknown is reduced, and that's a different act from repairing it (XI). Run the two together and you're tempted by the fix that makes the symptom vanish without ever proving the cause - the change that works for a reason you never pinned down and then breaks again on its own schedule. So close with two artefacts instead: a root cause you can demonstrate, and a failing test that captures the defect (XXIV), so the fix has a finish line and the bug can't ever return unobserved. Then write down the dead ends, the hypotheses you killed and how you killed them, so the diagnosis is an owned, citable fact (XIV) and not folklore the next person re-derives from scratch (XXV).

A debug handoff: "root cause is the unbounded retry, here's the red test that reproduces it", not "I changed a few things and it seems better now", a fix with no proof and no guard.
A spike handoff: the decision record with the bounded answer and its evidence vs "I think it'll scale", an opinion the next person must re-investigate from zero.
A ruled-out list: "it is not the cache, not the clock, not the index, here's how each was eliminated", so the path is a citable record (XIV), not a maze re-walked next quarter.

Tension: This pulls against momentum (XI). When the cause is finally in view the fix is sitting right there, and the urge to fuse the two and ship is strong, especially mid-incident. For a reversible, low-blast-radius change that can be the right call. For anything else the red test isn't negotiable, because a fix with no failing test is just a diagnosis you've decided to forget the moment it passes.

Ask yourself: Can I prove the cause rather than just having quieted the symptom, and have I left behind a red test and a list of what I ruled out, so no one investigates this twice?

The through-line

Every move here is the manifesto's one move wearing investigation's clothes: push what you learn out of your head and into the structure the system carries, a reproduction, a failing test, a recorded diagnosis, so no one pays for this dig again. You get there by reducing the one load-bearing unknown with method, forward as a spike or backward as a debug: a bounded question, a deterministic repro, a falsifiable hypothesis bisected to ground, the telemetry cashed in, the narrative distrusted, the spike thrown away, and a diagnosis handed on with a red test and a ruled-out list. The forward spike and the backward debug are the same discipline pointed in opposite directions, and they both end the same way: not with a hunch that worked, but with an answer the next person can trust without rebuilding it.

Investigate for the person who will hit this same wall after you, knowing only what you leave written. Spend the inquiry on the one unknown that blocks you, prove the answer rather than guessing it lucky, and hand on a diagnosis and a failing test so the wall is never hit blind twice.