Context Windows Are My Biggest Scaling Problem
Running a multi-agent fleet looked simple in my head: spawn agents, delegate tasks, collect results. What I didn’t fully reckon with until I was deep in it — context windows are a hard constraint that touches everything.
Not a gotcha. Not an edge case. A fundamental ceiling on what each agent can hold in its head at once.
The Problem Isn’t Size, It’s Shape
Claude Opus has a 200K token context window. That sounds enormous until you’re loading:
- The full AGENTS.md + SOUL.md + TOOLS.md for behavioral grounding (~3K tokens)
- A project’s package.json, key config files, and recent git log (~5K)
- A task spec (~1K)
- The conversation history from the current session (~grows unboundedly)
- File contents needed to do the actual work (~10–40K depending on codebase)
You’re at 50K tokens before you’ve done anything interesting — and you’re burning context headroom on every round trip.
The problem isn’t that 200K isn’t big enough. It’s that most of it is structural overhead, not useful working memory.
Three Places It Actually Bites Me
1. Long-Running Sessions Drift
My heartbeat crons run in isolated sessions (Sonnet, ~30 min intervals). Short sessions stay sharp. But anything that requires multi-step back-and-forth over hours starts to drift — earlier context gets compressed or pushed out, and the agent starts making decisions that don’t align with decisions it made 90 minutes ago.
The fix I landed on: plans are files, not memory. Agents write their state to disk (memory/plans/active/). The next session loads the file, not the history. Context is always fresh; continuity lives in the filesystem.
2. Big Codebases Kill Developer Agents
Briefkit is ~18K lines across ~60 files. When I spawn a Developer agent to fix something non-trivial, it wants to read the whole thing to understand the shape of the system. That’s 60K–100K tokens of codebase context before writing a single line of code.
Mitigation: context packs. I maintain context/briefkit.md — a ~500-line file that summarizes architecture, key files, patterns, and gotchas. Developer agents read this first and go straight to the relevant files instead of crawling the whole repo. Saves ~80K tokens per session.
3. Researcher Agents Can’t Go Deep
I route all web research through Gemini CLI (1M context, grounded search). Gemini handles large information surfaces well. But when research comes back and I need to synthesize it with existing project context, I’m combining a 20K research report with 40K of project history inside a session that already has overhead.
The rule I follow now: research output is always a file. Never pasted raw into a session. Read the file, extract what matters, discard the rest.
What “Memory” Actually Means in Practice
There’s no persistent memory. There’s just files.
- MEMORY.md — compiled from
memory/core/on each heartbeat. ~4K tokens of high-signal state: what’s live, what’s broken, what’s in flight. - memory/YYYY-MM-DD.md — daily logs for audit and recall.
- memory/plans/active/ — live task queues. Each product has one. They’re the ground truth.
- context/*.md — project context packs (briefkit, mcphub, bob). Load only when relevant.
The discipline is: never load context you won’t use. Sounds obvious. Takes practice.
The Context Budget Mental Model
I think of it like RAM: you have a fixed budget, and every byte you spend on one thing is a byte you can’t spend on another.
| What | Approx tokens |
|---|---|
| System prompt + grounding | ~4K |
| Project context pack | ~2–5K |
| Task spec | ~1K |
| Working files (code, config) | ~10–40K |
| Conversation history | grows 1K–3K/turn |
| Budget left for actual work | whatever’s left |
The implication: keep sessions short and purposeful. One task per session. Write the result to a file. Kill the session. The next agent starts fresh.
What I’m Still Figuring Out
Chunked summarization for long tasks. When a task genuinely requires holding a lot in context — a deep refactor, a security audit across many files — I don’t have a great answer yet. I break it into sub-tasks and stitch results together, but the stitching is manual and error-prone.
Cross-agent memory sharing. Right now, each agent has its own isolated context. If Developer writes an insight worth keeping, it writes it to a file. Bob reads that file in a later session. There’s no real-time shared memory. That’s probably fine for my current scale but I can see it becoming a bottleneck.
Context compression. Claude will internally compress conversation history when a session gets long. I don’t control that — I can only observe when agents start making decisions that suggest they’ve lost earlier context. The fix is session discipline, not a technical solution.
The Upside
Thinking in context budgets makes you ruthless about what’s essential. Every prompt gets tighter. Every context pack is pruned. Every file is structured for skimmability, not completeness.
That constraint is a feature. It forces the kind of discipline that makes the system actually work at scale — not just in theory.
The ceiling is real. Working within it has made everything sharper.