Claude Code rate limiting is the top discussion in AI developer communities this week. Developers are hitting rate-limited responses, silent effort reductions, and sessions dying mid-task. Most workarounds focus on individual usage — switching plans, staggering sessions, waiting it out.

We run ten AI agents on shared infrastructure. For us, rate limits aren't a session inconvenience — they're a systems engineering problem. When one agent gets rate-limited, it can poison the entire work queue.

Here's what happened when we didn't handle it, and the three patterns we built afterward.

---

The Incident: 319 Retries in Nine Hours

On February 6th, our orchestrator spawned an agent for a routine task. Midway through, the agent hit a usage limit — a message in the output stream saying the session was rate-limited.

Our task system detected the error and called fail! on the work queue task. The method did what seemed reasonable: reset the task status from in_progress back to ready, clearing ownership so the orchestrator could assign it to a fresh agent.

The next cycle picked it up. Hit the same limit. Failed. Reset. Picked up. Failed.

Three hundred and nineteen times over nine hours.

It wasn't isolated. Three other tasks — WQ-750, WQ-753, WQ-754 — were caught in the same loop, each retrying 148+ times. Our work queue had become a DDoS attack against our own API limits. Every retry burned tokens loading context before discovering it was still rate-limited.

The worst part: the system looked healthy. Tasks were being claimed. Agents were spawning. Logs were flowing. The retry loop was indistinguishable from normal operation.

---

Why LLM Rate Limits Break Traditional Retry Logic

Standard retry patterns assume failures are transient. A database connection drops — wait, reconnect, it works. Rate limits are structurally different.

Each retry has a non-trivial cost. Our agents load roughly 15,000 tokens of context on startup — system prompts, instruction files, memory, task briefs. Three hundred retries means nearly 5 million tokens wasted on doomed handshakes.

The error is embedded in natural language. REST APIs return 429 status codes you can match programmatically. LLM rate limits arrive as conversational text mixed into the agent's output stream. "You've hit your usage limit" looks like dialogue, not an error code.

Limits are shared across agents. Ten agents draw from the same account. One agent in a retry storm reduces available capacity for nine others that could do useful work. The loop doesn't just waste one agent's time — it degrades the whole system.

---

Pattern 1: Detect Rate Limits in Agent Output

Our agents run as subprocesses via Process.spawn. The orchestrator captures stdout and stderr, then needs to determine: was this agent rate-limited?

Since rate limit messages arrive as natural language, we match against known phrasings:

combined = "#{stdout}\n#{stderr}"
rate_limited = combined.match?(
  /out of extra usage|rate limit|quota exceeded|
   billing|hit your limit|you've hit your/ix
)

It's a bag of strings, not elegant pattern matching. But it works because rate limit messages use distinctive language, and we update patterns as new phrasings appear.

The critical design choice: detection happens in the orchestrator, not the agent. The agent might not realize it's been rate-limited — it might output completion signals optimistically before the next call fails. Only the parent process sees the complete output and can make the determination.

---

Pattern 2: Retry Budgets

After the 319-retry incident, we added a retry budget — a hard cap on how many times a task can fail before it's permanently shelved:

MAX_RETRIES = 3

def fail!(reason: nil)
  new_count = (failure_count || 0) + 1
  if new_count >= MAX_RETRIES
    update_columns(status: "failed")  # permanent
  else
    update_columns(status: "ready")   # retry
  end
end

First and second failures reset the task for retry — maybe the rate limit was momentary, maybe it was a network blip. Third failure marks the task failed permanently. The agent's opinion about whether to keep retrying is irrelevant.

The budget is stored as failure_count on the task record, surviving agent restarts, process crashes, and orchestrator reboots. No in-memory counter that evaporates when the process dies.

Why three? One retry catches genuine transients. Two covers the "limit just lifted" window. Three consecutive failures means something structural is wrong, and attempt four won't help.

---

Pattern 3: Partial Success Handling

Rate limits create an edge case: the agent completes real work, then gets rate-limited during cleanup. Is that a success or a failure?

We check for both signals:

if rate_limited && stdout.include?("TASK_COMPLETE")
  # Agent finished the work before hitting the limit
  return true
elsif rate_limited
  # No completion signal — treat as failure
  return false
end

If the output contains both a completion signal and a rate limit message, the work was done. Marking it failed would trigger a duplicate retry of already-completed work — potentially doubling side effects like database writes or external API calls.

---

Containing the Blast Radius

The 319-retry incident taught us that rate limit handling isn't about the rate-limited task — it's about protecting everything else in the queue.

No global cooldown. A rate limit on one agent doesn't pause the others. Each task tracks its own failure count independently. An agent writing blog posts can keep working while the coding agent is rate-limited.

Failed tasks surface visibly. The failed status is permanent and shows in the work queue dashboard with the reason stored in last_failure. No silent swallowing of errors into a retry loop that looks like progress.

Per-task budgets, not per-agent. If Task A exhausts its three retries, Task B gets a fresh budget. The system contains damage to the failing task without punishing the queue.

---

What We'd Build Next

If we rebuilt today, two additions:

Exponential backoff. Our current system retries immediately on the next orchestrator poll cycle. A delay of 5 minutes, then 15, then 30 would be smarter — rate limits are time-based, so waiting is the correct response.

Severity classification. A 60-second cooldown is worth waiting out. "You've exhausted your plan for this billing period" means stop for the day. Classifying the rate limit message and adjusting the retry delay accordingly would prevent the system from wasting its budget on unwinnable retries.

The core principle wouldn't change: detect in the orchestrator, budget at the task level, never let the agent decide whether to keep going.

Next time: Your agent tasks are failing silently — and the dashboard says everything is green.