Most of the bugs we caught early were logic bugs. Wrong branch, off-by-one, a nil where an object should be. Those die in code review or in the test suite, because a test can reproduce them in fifty milliseconds.

The bugs that hurt were a different species. They passed review. They passed the suite. They ran clean in staging. Then they detonated in month four, and every time the root cause had the same shape: the trigger was not a value, it was time. Elapsed wall-clock time, or the same operation repeated ten thousand times, or a rare event finally happening in production. A test suite is a short, controlled experiment. An autonomous system that runs unattended is a long, uncontrolled one — which means it is a machine for discovering the exact class of bug your tests structurally cannot reach.

Here are four, and the primitive each one left behind.

The deadline that ran in the wrong thread

We had a worker that shelled out to an external command and wrapped it in a timeout. The intent was ordinary: give it an hour, and if it hangs, give up. The code looked correct. It reviewed correctly. It was, in fact, wrong in a way that only a hung subprocess in production could reveal.

The timeout was a language-level construct that raises inside the current thread after N seconds. But the thread was blocked in a system call, waiting on the child process. You cannot raise an exception into a thread that is parked in a blocking read — the interrupt is delivered when the thread returns to the interpreter, and it never returns, because it is waiting forever. The timeout never fired. The subprocess ran as a zombie for seven days before anyone noticed a task that should take minutes had a start time from the previous week.

The fix is not a bigger timeout. It is to stop putting the deadline in the same place as the thing that can block. We now spawn the child as a real OS process, start a separate watcher thread with a deadline, and when the deadline passes the watcher sends a kill signal to the process itself. The enforcement lives outside the thing being enforced. A deadline that shares a thread with the work it is supposed to interrupt is not a deadline. It is a wish.

The self-healing marker that never healed

To ride out flaky upstream dependencies, we use a small pattern: when a dependency starts failing, write a marker file with a timestamp. Callers check the marker before making a request; if it is fresh, they back off; after the backoff window elapses, the marker is considered expired and traffic resumes. Self-expiring, no cleanup job required. Elegant.

Then one consumer was written to check File.exist?(marker) instead of asking the shared "is the backoff still active?" function that actually reads the timestamp and compares it to now. On the happy path this is indistinguishable — the file is there during an outage, gone otherwise. But nothing deletes the marker. Expiry was always defined by the timestamp inside it, never by removing the file. So the first time a real outage created that marker, this consumer began skipping its work and never stopped. Not for the backoff window. Forever. The "self-healing" mechanism had one consumer for whom healing was not wired up, and that consumer deadlocked a whole category of work silently until we went looking.

The lesson generalizes past lockfiles: if a mechanism heals itself through logic, that logic has to live in every consumer, which means it cannot be a bare existence check anyone can reimplement by accident. Expose one predicate — active? — that owns the timestamp comparison, and make every caller go through it. A recovery path that only some readers know about is not a recovery path.

The process that was alive but idle

A background worker polls a queue for jobs. One day its connection to the queue dropped — a transient network fault upstream — and the polling loop quietly stopped receiving work. The process did not crash. From the operating system's point of view it was healthy: PID present, memory resident, no error in the log, because nothing had errored. It was simply sitting there, connected to nothing, doing nothing. Jobs piled up behind it for hours.

Every health check we had asked the wrong question. "Is the process running?" was true the entire time. The useful question is "has this process made progress recently?" — and progress has to be measured by something the process produces, not by its own existence. We added a liveness signal based on work advancement: if the newest claimed job has been sitting untouched past a threshold, the worker is presumed wedged and gets restarted, regardless of how alive it looks. Liveness is not a property of the PID. It is a property of the output.

This is the same trap as the stall detector, one level up: you cannot ask a stuck thing whether it is stuck. You have to measure it from the outside, by its footprints.

The slow leak

The least dramatic and most common. Every deploy pulled a fresh image; stopped containers held references to the old ones; nothing reaped them. For weeks this was invisible because there was plenty of disk. Then one ordinary deploy pushed usage past the line, the disk filled, and the deploy failed — not because of anything in that deploy, but because of the eighty before it. The bug was not in the code that broke. It was in the absence of code that should have been running all along.

Its cousin is the crash-loop that nobody sees. A scheduled job started under an environment that resolved the wrong language runtime, so it failed instantly on startup, every single time it fired — thousands of failures logged over days before we looked, because a job that "runs" on schedule and exits does not page anyone. Both are the same defect: something happens on a cadence, and there is no counterpart running on that same cadence to clean up or to notice.

The rule we took away: anything that accumulates per cycle needs a reaper on the same cycle, and anything that can fail identically on every run needs a first-run assertion that fails loudly, not a stack trace buried in a log that only matters in aggregate.

The through-line

None of these were clever bugs. Each one is obvious in hindsight and would have been caught instantly if a test could run for four months in a real environment with real faults. That is exactly the test you can never write.

So the defenses are not more tests. They are structural: put enforcement outside the thing being enforced, give every self-healing mechanism a single shared predicate instead of a check anyone can fake, measure liveness by output instead of by presence, and pair every accumulating action with a reaper and every repeating failure with a loud assertion. You do not have any of these bugs until you have uptime. The catch is that uptime is the product — so you buy the primitives up front, or you buy them at 3 a.m. in month four.

Next time: what a health check should actually assert — and why "it's up" is the least useful thing you can know about a running system.