The Context Tax

What we measured

We wrote five small business tasks that each need real rules: a tiered sales commission with clawbacks and a quarterly cap, overtime pay with holiday and Sunday rates, a reorder point with safety stock, late fees with a minimum and a cap, and a check for invoices paid twice. The data is synthetic: about 30,000 rows across nine CSV files.

Each task was packaged three ways from one Python implementation, so every setup carried exactly the same logic:

• an Agent Skill with the rules written out in its SKILL.md;
• the same skill with a bundled script that implements the rules;
• a RAPP agent.py file: a Python class with a description and a perform() method.

Three hosts answered the same questions: Claude Code (claude -p), GitHub Copilot CLI (copilot -p), and a RAPP Brainstem, a small local server that loads agent.py files and answers on POST /chat. Claude Code and Copilot received the agent.py files as MCP tools. All three ran Claude Haiku 5.5.

A session asked 15 questions in a row, such as "what commission does Casey earn for 2026-Q2?". Copilot sessions asked the first 5, because each Copilot turn took much longer. Every setup ran 3 times, and every answer was graded against the exact figure, to the cent. We counted input tokens: everything sent to the model on every call, cached or not, as each host reports it. Caching lowers the price of repeated context, but the context is still sent, still fills the window and still takes time to read.

The host charges before you ask

Each host sends its own instructions and tool definitions with every model call. We measured that charge with one trivial question.

A fresh Claude Code sends 5.4 times as much as a Brainstem, and a fresh Copilot CLI 8.2 times as much. Nobody keeps a fresh install for long. Every plugin, MCP server, skill and memory file you add is sent again on every call, and a long-used install pays for all of it: 32,295 tokens for Claude Code and 33,719 for Copilot CLI as the author uses them. The author's own Brainstem, with 22 agents loaded, sends 8,716.

Every question pays it again

A host runs several model calls per question: it reads the question, calls a tool, reads the result, and answers. Each call sends the host's instructions plus the conversation so far.

The Brainstem's cost per question grows slowly, from about 6k to 12k tokens over 15 questions, because it keeps only the plain text of earlier questions and answers. Tool output never piles up in its history. In a chat that re-sends everything, each new message pays for all the messages before it, so the total grows with the square of the chat's length. That is steep but not exponential. Prompt caching lowers the price of the repeated part. It does not shrink it.

Rules belong in code

How you package the rules mattered less than which host ran them. Rules written only as instructions were the most expensive setup on both hosts, because the model re-reads them and redoes the arithmetic for every question. That arithmetic went wrong on 4 of 120 answers, by 1 to 180 dollars. Our first wording of the rules was also ambiguous about rounding, and the models filled the gap their own way (see the corrections at the end). Code leaves no such gap.

* Seconds per answer in this table come from runs where up to 18 sessions shared the machine; the clean timing comparison is below. Copilot sessions asked 5 questions, the others 15.

Where agent.py won and where it did not

In Copilot CLI, agent.py tools were the cheapest setup: 244.8k tokens for 5 questions, against 354.8k for skill plus script and 490.4k for instructions only, with every answer right. In the runs where we recorded its tool calls, Copilot called the agent.py tool on 5 of 5 questions.

In Claude Code, a skill with a bundled script was cheaper than agent.py tools. By default Claude Code hides MCP tools until the model searches for them, and it called the agent.py tool on only 13 of 45 questions; on the rest it read the files and worked the answer out itself. Listing the tools up front made it use them on 44 of 45, but every call then carried 31,177 tokens of overhead instead of 15,176.

The same agent.py files inside a Brainstem answered all 15 questions for 135.5k tokens, 45 of 45 right. The format of the file is not what saves tokens. The host that runs it is.

Lock it in, then clear the chat

Once the rules live in code, the conversation history carries little value. Starting a new chat for every question cut the Brainstem's total from 135.5k to 90.4k tokens with every answer still right. In Claude Code and Copilot the saving was small, because their fixed overhead per call dwarfs the history of a 15-question chat. With instructions-only skills, clearing the chat did not fix accuracy (43 of 45 right in Claude Code, 14 of 15 in Copilot), and with Claude Code's default tool search it fell to 39 of 45: in a new chat, Claude Code answered 6 questions from the raw files without calling the agent.py tool, and all of them were wrong.

Call /chat. Use MCP only when you must.

A Brainstem can also be reached from Copilot CLI through the brainstem-mcp plugin. We asked the same questions both ways against the same Brainstem.

Every routed answer still paid Copilot's own overhead several times over, and Copilot often did not route at all. The plugin as shipped (0.2.0) described itself only as "a local AI Brainstem", and Copilot never called it. Version 0.2.1, released during this study, names the Brainstem's agents in its tool description; Copilot then routed 8 of 15 questions on a fresh install. On the author's own install, with many other tools loaded, it routed none. Anything that can make an HTTP request should call POST /chat. The plugin is for hosts that can only speak MCP.

Time per answer

On fresh installs, with the rules locked into code, the three hosts answered at about the same speed: 3.5 seconds for Claude Code with agent.py tools, 3.7 for Copilot CLI and 4.0 for the Brainstem. On a fresh install the Brainstem's advantage is tokens, not seconds: 34.7k for those five questions, against 245.3k for Copilot CLI and 322.9k for Claude Code.

Speed is where setup weight shows. The same Copilot CLI as the author uses it every day, with the same agent.py tools and nothing else on the machine, took 56 seconds per answer and 820.3k tokens for the five questions. Claude Code also slowed down whenever it worked an answer out itself instead of running the code.

The cost of ownership

Token prices are the sticker price. The rest of the cost is keeping a setup working. Everything that broke or surprised us while running this study is listed here, including our own mistakes, so the hosts are not blamed for them.

Driving one question through each host from a script took 13 lines of code for the Brainstem (one HTTP request), 29 lines and 8 command-line flags for Claude Code, and 54 lines and 9 flags for Copilot CLI. The Claude Code session that ran this study is itself a long chat: by the end of the measurements it had made 261 model calls and sent 81.74M tokens of context, about 313.2k per call.

Why the Brainstem stays small

• A short system prompt. About 2.8k tokens with five agents loaded.
• History without tool output. It keeps the text of earlier questions and answers, capped at 40 messages and 60,000 characters. Raw tool results never re-enter the conversation.
• Agents answer in code. An agent.py runs the rules and returns only the result, so the model never re-reads the rules or redoes the arithmetic.
• Load only what you use. Agents are files. Drop one into the agents folder and it is available on the next request, with no restart; remove it and it stops costing anything. In our test, adding five agents to a running Brainstem took two seconds.

Best practices

1. Measure your host's idle cost. Ask it one trivial question and read the input tokens. That number is charged on every model call you make.
2. Put repeated work in code. If you ask for the same calculation twice, turn it into an agent.py (or a skill with a script). Instructions make the model redo the work and sometimes get it wrong.
3. Keep chats short. Finish a job, keep the answer, and start a new chat. Spend grows with the square of the chat's length.
4. Unload what you do not use. Every plugin, MCP server and skill rides along on every call, whether the question needs it or not.
5. Call a Brainstem's /chat directly. Use the MCP plugin only for a host that cannot make an HTTP request.
6. Give each agent its own named tool. A single generic "chat" tool hides what the agents do, and hosts skip it.
7. In Claude Code, prefer a skill with a bundled script for repeated work. Its MCP tools are found by search, and turning search off doubles the overhead of every call.

What this does not show

• One machine, one model (Claude Haiku 5.5), synthetic tasks, and 3 runs per setup. Copilot sessions asked 5 questions, the others 15. Larger models and longer chats may behave differently.
• We did not test whether long chats make answers worse. Published studies report that accuracy drops as context grows: Context Rot (Chroma, 2025), Lost in the Middle (Liu et al., 2023) and RULER (Hsieh et al., 2024).
• The Brainstem saves tokens partly by forgetting raw tool output. A later question that needs that output must ask for it again.
• The Brainstem does not report token usage on /chat, so the bench loads a 15-line counter into its Python process. Claude Code and Copilot CLI report usage themselves.
• Every Brainstem in this study ran on the same machine as the bench. We did not measure a Brainstem hosted in the cloud; it should send the same tokens for the same agents, but its time per answer will include the network.
• The "as used" measurements describe one person's setup. Yours will differ; the bench measures it in one command.

Corrections made along the way

A reviewer and our own checks caught errors in the first runs. All are fixed, and the runs they spoiled are kept in the repository, marked invalid:

• Copilot's usage file holds a running total for the whole session. We first read it as per-turn, which overstated Copilot's later turns.
• Claude Code was first given the MCP server in Copilot's config format. It connected nothing and showed no error, so the first "agent.py in Claude Code" runs had no agent.py tools at all.
• The first grader matched numbers inside other numbers. It now extracts every number and compares to the cent.
• Three rules were ambiguous in plain words (same-day ordering, a minimum fee above the cap, a holiday on a Sunday). We wrote the rules out fully and re-ran.
• A fourth rule did not say whether to round each late fee or only the total. Most of the skill answers we first graded wrong had rounded each fee half-up to the cent, which is a fair reading. The rules also wrote amounts with dollar signs, and Claude Code replaces $0, $1 and so on inside a SKILL.md with the skill's arguments. We rewrote the rules in words and re-ran every skill setup.

Run it yourself

The bench, the tasks and every run behind this paper are at github.com/kody-w/token-bench. It measures your own setup, as you use it, in one command.