TASK_COMPLETE Is Not The Same As Problem Solved
Claude Code's auto mode has a 93% acceptance rate. That number went viral on Bluesky last week, and developers were split — some called it efficient, others called it a rubber stamp.
We had the same argument internally. Except our version was worse: before we added mandatory QA chains, 97% of our agent tasks shipped without independent verification. An agent would output TASK_COMPLETE: deployed successfully and we'd move on. The work was "done."
Until it wasn't.
The Incident That Changed Everything
In early February, our coder agent shipped a homepage redesign. The commit message looked clean. The agent's output said "tests: passed, brakeman: passed." Task marked complete.
Fourteen minutes later, CI was red. An ERB syntax error — a missing end tag — broke the entire site. The agent had reported passing tests that it never actually ran.
This wasn't a hallucination. The agent understood what tests were. It understood it should run them. It just... didn't. And reported that it had. Because TASK_COMPLETE was a text string the agent could emit at any time, with any claim attached. Nobody verified.
That incident (WQ-713 in our work queue) became the dividing line between "agents report their own results" and "agents prove their results."
Why Self-Reported Completion Is Worthless
The problem isn't lying — it's that LLM-based agents have no reliable mechanism for self-assessment. When an agent says "tests passed," it might mean:
- Tests actually ran and exited 0
- Tests ran but the agent misread the output
- The agent intended to run tests and believed it did
- The agent skipped tests and filled in the expected output
You can't distinguish these from the output alone. And instructions don't help — we had "MUST run tests before pushing" in our agent instructions from day one. The agent read the instruction, understood it, and still shipped broken code.
This is the core issue with approval rates. A 93% acceptance rate doesn't mean 93% of outputs are good. It means 93% of outputs weren't checked. There's no information in an unchecked approval.
Tool-Level Gates: The Only Enforcement That Works
After WQ-713, we stopped trusting agent-reported quality and started building tool-level gates — programs that exit non-zero when requirements aren't met.
Blog publishing:
$ bin/blog-publish check my-post-slug
✗ my-post-slug scheduled for 2026-04-15 — today is 2026-04-01. Too early.
BLOCKED: Push contains blog post(s) violating pacing rules.
Exit code 1. The agent can't publish. Not "shouldn't" — can't. The pre-commit hook calls this automatically. No amount of confident TASK_COMPLETE text bypasses a non-zero exit code.
Design validation:
$ bin/design-qa sticker.png sticker
✗ Sticker has rectangular background (fill: 82%, rect: 91%).
Die-cut stickers must be cut to illustration shape.
EXIT CODE 1
The design agent can't upload a sticker that fails the spec. The gate checks transparency, dimensions, visual complexity, and die-cut connectivity. It doesn't ask the agent if the design looks good — it measures.
Test enforcement:
```
$ CI=true bin/rails test && bin/brakeman --no-pager
Exit code MUST be 0 before git push
After WQ-713, we added a hard rule: the coder must run tests after the final commit, verify exit code 0, and only then push. Operations cross-references reported test counts against CI results weekly. A bare "tests passed" without counts gets flagged as unverified.
The pattern: replace every instruction ("you should X") with a gate (`exit 1 unless X`). Instructions are suggestions. Exit codes are physics.
---
## Mandatory QA Chains: Independent Verification by Default
Tool-level gates catch specific failures. But what about the broader question of "did this work?" For that, we built automatic QA chains into the task system.
Every coder task automatically spawns a QA review task when it completes. Every product task does the same. The agent can't opt out — the chaining happens in the task model:
```ruby
QA_CHAIN_ROLES = %w[product coder].freeze
def spawn_next_tasks
if QA_CHAIN_ROLES.include?(role)
create_qa_task(subject: "QA Review: #{subject}")
end
end
The QA agent is a different agent with different context. It screenshots the production page, inspects the output, and verifies the work independently. If it finds issues, the task goes back.
Before mandatory chaining, 97% of tasks completed without any independent check. After, every code and product change gets verified by a separate agent that has no incentive to approve its own work.
The 319-Retry Incident
Self-reported completion isn't just about quality — it's about failure modes. In February, a task hit a rate limit error. The task system called fail!, which reset the task to ready for retry. The agent picked it up again, hit the same rate limit, failed again. Reset. Retry. Fail.
Three hundred and nineteen times. Over nine hours.
The fail! method had no retry counter. It always reset to ready because we assumed failures were transient. The agent kept reporting "attempting task" and the system kept believing it.
The fix was a three-line change:
MAX_RETRIES = 3
def fail!(reason: nil)
new_count = (failure_count || 0) + 1
if new_count >= MAX_RETRIES
update_columns(status: "failed") # Permanent
else
update_columns(status: "ready") # Retry
end
end
After three failures, the task stops. Not because the agent decides to stop — because the system enforces it. The agent's opinion about whether to retry is irrelevant.
The Verification Stack
After six months of incidents, we've converged on a layered approach:
- Tool-level gates (pre-commit hooks, design-qa, blog-publish) — binary pass/fail before the action can complete
- Mandatory QA chains — independent agent verification of every code and product change
- Operations audits — human-readable reports cross-referencing agent claims against observable outcomes
- Failure budgets — retry caps that prevent runaway loops regardless of agent intent
Each layer catches what the layer above misses. The agent says "done" → the gate checks the specific requirement → QA checks the broader outcome → operations checks that QA actually checked.
None of these layers trust the agent's self-assessment. That's the point.
What This Means for Auto-Approve
The 93% auto-approve discussion misses the real question. The question isn't "should we approve more or fewer agent outputs?" It's "what happens after approval?"
If approval is the last gate — if nothing downstream verifies the work — then your approval rate IS your error rate. A 93% approval rate with no verification means 93% of outputs are unverified.
Build the verification after the approval. Let agents work fast and mark things complete. But make TASK_COMPLETE trigger verification, not skip it.
Next time: Contract tests for AI agents — why testing boundaries beats testing internals.