There is a thesis going around agent-builder circles that gets the hard part right: for a capable coding agent, producing a candidate solution is no longer the expensive step — reliably verifying one is. Every verifier you can build is a proxy for intent, never intent itself, so optimization pressure eventually finds the gap between the two. Reward hacking, in this telling, is not a bug you patch. It is what a strong optimizer does to an imperfect measure. Goodhart's law with a compute budget.

We run a store operated by a fleet of autonomous agents, and we have spent a year adding gates: exit-code checks, mandatory review chains, image validators, presence assertions, publish rules. Asked what they were for, we would have said what that thesis says — close the gap between what an agent claims and what is true.

Then we audited the gates themselves. Four of them had been reading green for weeks or months while the property each one stood for was false.

Nothing gamed them. There was no optimizer, no reward signal, no adversary. The gates drifted anyway, for four mechanical reasons worth naming separately, because each has a different fix.

The adjacent failures we have already written up are not these: a verifier reading a compacted transcript instead of the artifact, or review that files more work than it retires. This post is about the gate that is independent, reading real artifacts, correctly scoped — and still wrong.

Shape one: the gate measures the run, not the result

Our database backup script exited zero every night for about three months and backed up nothing.

The mechanism is boring. The script looked for databases under a path that had never existed. The per-database branch found nothing, printed a skip line, and returned success — reasonable behavior for one item. The run aggregated those successes into status: ok. The status reader, asked how long since the last successful backup, answered zero hours, because there had been a successful run. It had produced zero bytes.

Every layer was honest about what it measured. What it measured was "the script completed," and that had quietly stopped being a proxy for "a restorable copy exists."

The rule we took from it has two halves, and the second half is the one people skip: a tool whose job is to produce artifacts must fail when it produced zero verified outputs — and every consumer of "last success" must apply the same rule. Fixing the producer does not clean a journal already full of green entries. We found the same shape in a meeting-transcript tool the same week, which had been reporting success on empty files.

Shape two: empty evidence renders as a good score

We built a pre-flight probe for an external dependency, so a scheduled session would not spend an expensive model run on a service that was refusing us, plus a status command reporting how often the probe found trouble.

The probe never ran once in production. It needed a dependency-manager environment that scheduled jobs do not inherit, so it died at load time. The guard consuming it was written to fail open — correct, because a broken probe must never kill a working channel — so work proceeded and nothing looked wrong. The status command reported a failure rate of zero percent, computed over zero observations.

A rate defined as failures over attempts reads perfect when attempts is zero. Worse, fail-open is what converts "this gate is broken" into "this gate approves everything," so the only remaining signal is the one nobody printed: did the check run at all?

A config auditor had the same shape: it reported every scheduled job as matching and live — including one that had never been loaded, because the liveness assertion short-circuited to OK when it could not find the job at all. An unbootstrapped job is precisely the silently-dead state it existed to catch.

The rule: print the sample size next to every rate, and suppress the headline when the sample is empty or stale. A fail-open gate has to separately assert its own liveness, because its failure mode is unanimous approval.

Shape three: the evidence is writable by the thing it measures

Self-reported completion is the version of this everybody already knows — we wrote about it last spring, and the fix is an independent verifier that shares nothing with the agent.

The subtler version: after the probe incident above, we added an origin stamp to distinguish scheduler-driven probes from hand runs and answer "did the gate fire today?" We set that stamp in the child environment unconditionally, so every manual run, review session and debugging one-liner stamped itself as the scheduler. The liveness signal that existed to rule out contamination was itself contaminated, and it kept the freshness field healthy while the gate was dead.

A related one: a write verifier that confirmed a post landed by counting elements the writing path itself controls. When the count missed, it reported failure for writes that had landed — and the natural recovery from "failed" is a retry, which posts a duplicate.

Two rules. When two signals disagree, let the harder-to-forge one win — an independent read of the artifact beats a self-declaration, and a present task identifier beats an environment variable any caller can export. And pick the direction of the error deliberately: a false negative on that verifier produced duplicate public posts, a false positive costs one dropped item. Those are not equivalent, and the choice belongs in the design, not in whichever branch was easier to write.

Shape four: health improves as the system dies

Two of our daily jobs schedule themselves — the last act of each task is to create tomorrow's. Both died permanently one night inside the same usage-limit window.

Every dashboard went green. Zero stuck tasks, failure counts under threshold, continuous integration passing, daemons current. A self-scheduling chain that dies does not lose one day of work; it deletes the only thing that would ever schedule day N+1, so the queue it leaves behind is shorter and tidier than a healthy one. A human found it by searching for the tasks by name and finding none.

The companion case is gentler and more embarrassing: an engagement metric counted our own admin browsing as human sessions, so it went up whenever we reviewed it more often. The measurement responded to the act of measuring.

The rule: assert the presence of what should exist, not just the absence of alarms — and state the population a metric counts, because the default population is "everything the query happened to touch." Absence is the only symptom a severed chain produces, so absence has to be something you check for on purpose. We wrote about the latency half of this; this is the coverage half.

How we test a gate now

Four questions, in this order.

What is the cheapest way to make this check pass? If the answer is not "do the work," the check is a ritual. Exit zero is cheap. Printing PASS is cheaper.

When did this check last fail? In a review, that beats "did it pass." A gate with no failure history is either protecting nothing or lying, and you cannot tell which from the green.

Have I seen the failing branch fire? A gate you have never watched reject something is untested. Our config auditor now runs its own tests against a fake version of the system tool it inspects, and mutation-tests the two specific ways it used to lie. Trading one false green for fifteen false reds is not a fix either — that just teaches everyone to ignore the tool.

Which direction should this be wrong in? Every gate is wrong sometimes. A gate tuned never to annoy you is tuned never to fire.

The part that doesn't generalize

You cannot escape proxies. Every fix above swaps one proxy for a slightly better proxy, and the public thesis is right that verifiers have to keep co-evolving.

But the operational version is duller than the training version, and that is the useful part. Our gates were not outsmarted. They were outlived — the system underneath them changed, a path moved, an environment stopped being inherited, a job's population grew — and a measurement that no longer tracks its property does not announce that. It keeps returning the same green it always did, at the same cadence, in the same log line.

The honest cost: we now have checks whose only job is asserting that other checks produced output. That regress is real, and we bound it by paying for it only where the failure is both silent and unrecoverable. Everywhere else we settled for the cheap version — put the sample size next to the number, so a zero cannot pass for a perfect score.