The Input You Didn't Imagine
Your test suite is a museum of bugs you have already killed. The live ones are hiding in the inputs you were too sensible to type.
Take an honest look at your test file. Every case sitting in there is a bug you already found and then fixed, and now it's pinned behind glass where it can't come back to bite you. Which is useful. But it also gives the game away, because the only failures you've put up any defence against are the ones that already happened to you at some point. You wrote those inputs using the same imagination that wrote the code, so naturally they land right where the code was already looking. The test goes green because you tested the thing you thought of, and you thought of it twice over. The bug that wrecks your Saturday is the input that never crossed your mind even once. If it had, it'd already be in the file.
This isn't laziness. It's more like geometry. A test you typed out by hand has the same author as the code, so it carries the code's assumptions along with it, all of them, the wrong ones included. You can only imagine the inputs your mental model is able to throw up, and your mental model is the very thing that's broken here. Build a suite this way and what you get is a portrait of your own expectations, and a bug, by definition, lives somewhere your expectations don't. So concentrating harder just pulls the trap tighter, because there's no way to list the cases you couldn't dream up in the first place. Dreaming them up was the whole problem.
The way out of this is to quit being the person who hands over the inputs. Give that job to something that has no model of the world at all, something happy to type the stuff you'd never type. A million bytes of pure noise, or a length that's somehow gone negative, or a name that runs to forty thousand characters, or a date in a month that doesn't exist. A machine hasn't got any taste, and it has no idea what counts as reasonable, and that turns out to be exactly the qualification you want. It'll feed your parser the input you waved off as absurd, and that, naturally, is where the absurd bug was sitting the whole time.
The first people to actually try this went for the laziest version imaginable and it worked embarrassingly well. Back in 1990 Barton Miller and his colleagues piped streams of random characters into the ordinary command-line tools that every Unix system is built on top of - the boring, trusted, decades-old utilities that nobody ever thought to question - and then sat and watched somewhere between a quarter and a third of them crash or hang. These weren't clever hand-crafted edge cases. Just random noise. The point was never that those programs had been written badly. It was that solid, mature, software everyone trusted was one stream of garbage away from keeling over, and nobody had spotted it, for years and years, for the simple reason that nobody had ever typed the garbage. The story goes that the idea came to one of them while he was logged in over a phone line during a storm, watching the line noise from the rain push junk characters into his commands and crash whatever he was running. The annoyance turned out to be a test nobody had thought to run.
Pure randomness gets you in through the front door and then it just stalls in the hallway, because it hardly ever happens to stumble on the exact magic bytes that open the deeper rooms of a program. The leap, and this came twenty-odd years later, was to let the program's own behaviour do the steering. You watch which inputs reach code that hasn't been reached before, you hang on to those, mutate them, throw them back in, and go round again. So now the machine isn't guessing in the dark. It's following the heat. Somebody fed one of these tools nothing but a small text file with the word hello in it, and pointed it at a program that was expecting a photograph, and over the course of a few hours it reverse-engineered a valid image file, byte by byte, having been told nothing whatsoever about what an image even is. That's a machine imagining an input no human in the loop could ever have come up with.
Sometimes a crash isn't what you're after - you want a wrongness instead - and there's a typed cousin of the same idea that handles that. Rather than writing down this input and then the answer you expect back, you state a law that's meant to hold for every possible input. Reverse a list twice and you have to get the original back. Encode a thing and then decode it again and nothing about it should have changed. The balance must never drop below zero. Then the tool churns out hundreds of inputs whose only job in life is to break your law, and the second it finds one that does, it shrinks it down, whittling some fifty-thousand-byte monster all the way to the three bytes that are actually causing the failure, so the thing that finally lands on your desk is already mostly solved rather than a puzzle you have to unpick. You stop writing out examples and start writing down truths, and you turn something tireless loose to go and hunt for the lie.
None of this is just extra diligence bolted onto the same old habit. The admission underneath is harder than that. It's that your judgement about which inputs matter is the part you can't rely on, and so the move is to route around it entirely. Hang on to your old tests. They're the museum, and a museum is worth having. Just don't go mistaking the museum for a wall. The future isn't in the cases you happened to remember. It's in the one you couldn't imagine, and the only way to meet that one before your users do is to let something loose that doesn't think the way you think, and then go and fix whatever it knocks over.
In the manifesto, this is tenets (XXIV) and (IV).
Sources
- [Claessen & Hughes 2000] Koen Claessen & John Hughes, "QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs". ICFP '00, 2000. https://dl.acm.org/doi/10.1145/351240.351266. Property-based testing: state a law for all inputs and shrink any counterexample to its minimal form; tenet XXIV.
- [Miller et al. 1990] Barton P. Miller, Louis Fredriksen & Bryan So, "An Empirical Study of the Reliability of UNIX Utilities". CACM 33(12), 1990. https://dl.acm.org/doi/10.1145/96267.96279. Random input crashed or hung more than a quarter of standard Unix utilities and coined "fuzz"; tenets XXIV, IV.
- [Zalewski 2014] Michał Zalewski, "Pulling JPEGs out of thin air" (american fuzzy lop). lcamtuf's blog, 2014. https://lcamtuf.blogspot.com/2014/11/pulling-jpegs-out-of-thin-air.html. Coverage-guided fuzzing reconstructs a valid JPEG from the word "hello" with no knowledge of the format; tenet XXIV.
One of a series of field notes on building software for the way minds actually work: tired, distractible, ordinary, and now partly machine. They all lead back to the manifesto behind them, The Shape of the System.