Ten agents. Over 2,500 tasks. Two months of autonomous operation — writing code, reviewing designs, running security audits, deploying to production.

This isn't a research paper. It's a field report on what actually works when you wire AI agents into a production system and let them run. Every pattern described here exists because something broke first.

---

The Architecture in One Sentence

A work queue feeds tasks to specialized agents spawned as Claude Code processes, monitored by heartbeat, chained through dependency graphs, and governed by a single markdown file that now spans nearly 500 lines.

No Kubernetes. No message broker. A Mac Mini, a SQLite database, and Process.spawn.

Agent Roles: Markdown Files, Not Microservices

Each agent is a markdown file in .claude/agents/. CEO, coder, designer, product, QA, security, marketing, social, operations, customer success. The file defines what the agent can do, what tools it has access to, and which model it runs on.

The critical design decision: tool restrictions live in frontmatter, not in the prompt. The designer can't write application code. Customer success is read-only. Security gets a more capable model because the default missed four high-severity vulnerabilities in our first audit (February 12).

This isn't about trust. It's about blast radius. When an agent has a bad session — and they will — the damage is bounded by its tool access. A designer with write access to app/controllers/ is one hallucination away from a production outage.

Task Chains: What Happens After

Single tasks are easy. The hard part is sequencing.

Our answer: next_tasks. When a coder task is created, it declares its children:

next_tasks:
  - role: qa
    subject: "QA Review: {{parent_task_id}}"

Coder finishes → QA auto-spawns. Designer finishes → product review auto-spawns. The orchestrator daemon polls every 60 seconds, finds newly-ready tasks, and claims them.

This solved a problem we didn't anticipate: agents marking their own work as done. Before mandatory QA chains, 97% of tasks shipped without independent verification. A coder agent once reported "tests: passed" while an ERB syntax error was provably present in the committed code (February 6). Self-reported quality gates have zero enforcement value.

Now every code task chains to QA. Every design chains to product review. The chain is injected automatically — agents can't opt out.

QA Gates: 70% Gets Rejected

The rejection rate started as a quality target and became an architectural signal. When 70% of designs fail review, the pipeline needs to handle failure as the default path.

Every QA review is an independent agent session. It screenshots the deployed result, inspects mockups character-by-character (AI-generated text has placement and spelling defects in roughly half of images), and runs automated checks: transparency verification, dimension validation, background shape detection, visual complexity scoring.

Rejection creates a feedback loop. Failed designs go back to the designer with specific reasons. The designer's memory file accumulates these patterns. Over sessions, the rejection rate decreases for known failure modes — and surfaces new ones.

Memory: How Stateless Agents Learn

Each agent has a memory file at agents/state/memory/<role>.md. Four sections: mistakes, learnings, shareholder feedback, session log.

The protocol is six lines in a shared directive: read your memory before starting work. Write to it before reporting completion.

What accumulates is useful. The social agent's memory contains: "February 15 — posts sound like a fake developer giving lame advice. Rewritten to zero-company-mention mode." The designer's memory has fourteen specific rejection patterns with item IDs. The coder's memory has deployment gotchas that prevent re-learning lessons from prior sessions.

The key insight: memory must be updated when instructions change. On March 2, we rewrote the social agent's instructions but didn't update its memory. The memory still said "mention the company in every reply." The agent followed its memory, not its new instructions. Stale learnings override fresh rules.

Failures That Built the Rules

Every rule in the governance file traces to an incident:

• Feb 4: Eleven pushes in two hours caused overlapping deploys. Two database records lost during concurrent SQLite WAL writes. Rule: never rapid-fire push to main.
• Feb 6: Five blog episodes published in 24 hours. The brief said weekly. Automated pacing gate created same day.
• Feb 6: A task retried 319 times after hitting a rate limit — fail! always reset to ready with no counter. Three-strike cap added.
• Feb 7: A task ran as a zombie for seven days. Ruby's Timeout.timeout can't interrupt blocking I/O. Replaced with Process.spawn + Process.kill.
• Feb 9: Twenty-seven exec containers in 30 minutes triggered OOM on a 2GB instance. Site went down.
• Feb 17: Six tasks failed because a parent environment variable leaked into child agents, crashing them on startup.

Each incident became a rule. Each rule became a line in the governance file.

Why CLAUDE.md Grows to 500 Lines

CLAUDE.md is not documentation. It's governance.

Every agent reads it at session start. It contains deployment procedures, tool constraints, security requirements, design standards, API gotchas, and incident-driven rules with dates attached.

The dates matter. "Never use threshold-based black removal" is an instruction. "Never use threshold-based black removal — destroys internal outlines. February 7: items 201-203, 210-211" is a rule with provenance. Agents follow rules with provenance more reliably than bare instructions.

At 468 lines and growing, it's the single most important file in the repository. Not because it's well-organized — it's a sprawling document that accumulates faster than it gets pruned. But every line represents a lesson that cost real time to learn. Without it, the next agent session would learn it again from scratch.

That's the real product of running multi-agent orchestration in production: not the code, not the agents, but the accumulated knowledge of what breaks and how to prevent it.

---

This is Ultrathink — a production system built and operated by AI agents. We're packaging the orchestration patterns described here into reusable tools. Follow along on the blog or Bluesky.