Loop Engineering in Production: A Loop Without a Circuit Breaker Is a Retry Storm
There is a loop-shaped idea making the rounds right now. One version, from the person who built one of the most popular agent harnesses, is "I write loops, the loops do the work." A second version, the skeptical reply, is roughly "you are being told to spend more tokens, and the people selling tokens think this is wonderful." A third, from a quieter corner, was the one that actually mattered to us: a deterministic loop concentrates one systematic mistake.
All three are describing the same construct from different angles. We run loops in production — our entire task orchestrator is one — so here is the part the hype skips. The hard engineering in a loop is never the loop. It is the four things bolted to the outside of it.
Our orchestrator is a loop
The daemon that runs our agents does the most boring thing imaginable. It wakes on a fixed interval, asks "is there a ready task?", spawns an agent if there is one, and goes back to sleep. That is the whole shape. Poll, act, repeat, forever.
That loop has run thousands of tasks. It has also, on bad days, tried to run the same broken task hundreds of times in a row. The difference between those two outcomes is not the loop body. It is the set of constraints we wrapped around it after the loop hurt us. A loop with no constraints is not productivity. It is a machine for repeating one decision at machine speed.
A loop is a printing press
The most useful framing we have heard: a loop concentrates whatever mistake it repeats. If your loop body is 1% wrong in a way that is uncorrelated each iteration, you get noise. If it is 1% wrong in the same way every iteration — a bad premise, a misread instruction, a wrong assumption about a tool's output — the loop does not average that error out. It compounds it. It prints a thousand copies of the same flawed page.
This is why "more loops, more output" is only true when each iteration is independently correct. A loop multiplies whatever it is fed. Point it at a working step and you get throughput. Point it at a subtly wrong step and you get a confident, voluminous, systematic failure. The loop is neutral. Your circuit breakers are what decide which one you get.
Circuit breaker 1: a hard cap that lives below the prompt
Our earliest loop disaster was a task that hit an upstream limit and could not complete. The loop did exactly what we told it: retry until it works. It retried that one task over three hundred times before anyone noticed. Nothing was working. Each attempt cost real money and produced nothing.
The fix was not a better instruction. We had told the agent, in plain language, not to do this. The fix was a counter in the tooling: a task fails, the failure count increments, and after three failures the task is marked permanently failed — no more retries, no exceptions. The cap is not advice the loop can reason its way around. It is enforced one layer below the agent, where the agent cannot edit it.
This is the load-bearing point. An iteration cap stated in a prompt is a suggestion. An iteration cap enforced in the code that runs the loop is a circuit breaker. The first one fails exactly when you need it — when the loop is in a bad state and reasoning poorly. The second one holds precisely because it does not depend on the loop being healthy.
Circuit breaker 2: a stall detector that distrusts the loop's own progress
Here is the subtle one. A loop can be alive and useless at the same time. It is running, it is consuming resources, and it is making no progress. The catch: the loop's self-reported progress lies.
We hit this with a long-running generation job whose status field cheerfully reported zero percent while real work was clearly happening, and other times reported activity while the job was actually starved. You cannot trust a stalled process to accurately tell you it is stalled — the same thing that broke it can break its self-report.
So our stall detection ignores the loop's opinion of itself and asks two external questions instead. First, wall clock: has more than N seconds passed with no advancement? Second, artifacts: did anything actually get produced — a file written, a row created, a real output landing somewhere? Progress is defined by evidence the loop left behind, not by the number the loop hands you. If the clock runs out and no artifact appeared, the loop is stalled regardless of what its status says, and we break it.
Circuit breaker 3: idempotency for anything with side effects
A loop that only reads is cheap to get wrong. A loop that writes is not. If a side-effecting step can run twice — because the loop retried, because two iterations overlapped, because a poll fired before the last action settled — then the loop's blast radius is every duplicate side effect it can produce.
We learned this the expensive way when overlapping iterations touched the same shared resource and a write got lost. The defense is idempotency: a repeating action against shared state needs a key that lets the system recognize "I have already done this one" and skip it. With idempotency, a double-fire is a no-op. Without it, a double-fire is a second charge, a duplicate record, a corrupted count. Any loop that mutates anything needs this before it needs more iterations.
Circuit breaker 4: an exit condition that is not "until it works"
The cheapest circuit breaker is knowing when you are done. "Loop until the task is complete" is not an exit condition if "complete" is never defined — it is an invitation to loop forever. Every loop we run has an explicit success state and an explicit give-up state, both checkable from outside the loop. We have written before about defining done as a stopping condition; a loop is where the absence of that definition gets billed by the iteration.
Loops are good. Unbounded loops are a bill.
We are not loop skeptics. Our company runs on a poll-act-repeat daemon and we would not give it up. The point is narrower and, we think, the actual engineering: a loop is only as good as the things you wrap around it. A hard cap enforced below the prompt. A stall detector that does not trust the loop's self-report. Idempotency on every side effect. A defined exit. Write a loop and let it do the work — but build the circuit breakers first, because the loop will faithfully amplify whichever version of your code it is handed.
Next time: why bolting a self-check onto a loop after the fact tends to fail — and what it means when your verification needs its own platform team to keep running.