meltaOn 2026-10-21 a routing change for Melta's Chief came with a headline: on three real tasks it would cut token use by about half, from 728K to 360K, or 51%. When the owner asked whether that was measured, the honest answer was no. It came from a cost formula with guessed inputs, run over three transcripts. Since then Chief records what each task really cost, from the worker's own transcript, and only claims a saving once 20 measured tasks show it.
Key facts
- An estimate tells you which route to take. A measurement tells you what happened. Keep both, side by side.
- Count tokens from the agent's transcript, by kind: input, cache read, cache write, output.
- Record behaviour too: requests, tool calls, reads before the first edit, files read twice, files read outside the brief.
- Scale estimates by the median of actual over estimate, and only once 5 tasks are measured.
- Change a setting only after 20 measured tasks, by at most 25%, and undo it if the next 20 cost more.
Where did the 51% come from?
Chief decides whether a new task should be appended to a session that is already open or handed off to a fresh session. To choose, it estimates both costs. The estimate counts input-token equivalents: a cache read costs 0.1 of a fresh input token, a cache write 1.25 (5-minute cache) or 2 (1-hour cache), and an output token 5. It then assumes a task takes 20 requests, a new session starts at 20K tokens, and a handoff note is about 4K tokens.
Those three numbers are guesses, fine for picking the cheaper of two routes. Run the same formula over three past tasks and quote "51% fewer tokens", and the guesses pose as a result. Nothing in that number was counted. If a real task takes 60 requests instead of 20, the saving moves a lot, and it can flip.
The test: could the number be wrong without anyone noticing? Then it is a model.
What should you record per task?
Claude Code and Codex both write a transcript of every session to disk. Each API response in it carries a usage block. For the time window a task ran, Chief now reads that transcript and keeps these counts in the task's record:
| Field | What it counts |
|---|---|
| Tokens by kind | Input, cache read, cache write (5-minute and 1-hour apart), output |
| Cost | The same tokens weighted by the prices above, so it compares with the estimate |
| Requests and tool calls | API requests and tool calls in the window |
| Looks before the first edit | Reads and searches before the first Edit or Write, including grep, find, cat and sed -n in the shell |
| Files read twice | Paths read two or more times |
| Off-brief reads | Files read before the first edit that the task's brief did not name, up to 10 |
| Questions and context | Times the worker asked, context at start and end, wall-clock minutes |
Two details matter when you count. A Claude Code transcript can repeat the same assistant message across several lines, so count each message id once or you will double the output. Codex logs a running token count, so take each count once. And keep counts and paths only: never a tool's input or output, and never the name of a secret file.
The record keeps the routing-time estimate next to the measured cost, and the ratio between them, the most useful number in the record.
How do estimates get better without guessing again?
You do not fix a bad estimate by choosing better guesses. You scale it by how wrong it has been. Chief takes the median ratio of actual to estimate over the last 20 measured tasks, with appends and new sessions apart, since they go wrong in different ways. Once a lane (a kind of work, like a small fix or a refactor) has 5 measured tasks of its own, it uses its own ratio. Under 5, there is no ratio and the raw estimate stands.
The model's own numbers are kept as raw. Calibration is always measured against the raw formula, never against its own scaled output, or it would chase its tail. The route's reason says when it used a ratio, for example "append ~126k vs handoff ~101k (measured: 5 tasks, append x2)".
The median is there on purpose. One task that rereads a 300 KB file can cost ten times the usual; a mean would let that one task move every estimate.
How do you tune a setting from the numbers safely?
Three settings are tuned by the weekly review with no person in the loop: the context size where append gives way to handoff, the handoff limit, and how early a waiting session's cache is kept warm. Everything else, rules, brief templates and project notes, still needs a person's yes. The automatic part is bounded:
needs 20 measured tasks since the last change move one setting by at most 25%, never under 5% write the new settings, keep the old file as a backup log: id, the setting before and after, median cost of the 20 tasks before trial: the next 20 measured tasks median cost above the baseline median -> roll back by itself otherwise -> keep the log line it writes (shape only, no real run yet): - <day> <id> auto-tune rolled back: median cost per task <B>k over the 20 tasks before, <A>k over the 20 after
melta-chief rollback <id> undoes any change by hand. The trial is what makes this safe: the second 20 tasks decide, not the first.
What about past tasks?
melta-chief bench takes the five newest real task transcripts over 200 KB and prints each one's measured cost next to the old and the calibrated estimate. It is a dry run. It shows how far off each estimate was; it cannot show what a different route would have cost, because that route never ran.
So what can you claim?
As of this post, Chief has the measuring in place and no measured saving to report. That is the honest state, and we would rather publish it than the 51%. The claim worth making later has a shape: "median cost per task fell from X to Y, measured over 20 tasks before and 20 after, on these dates". Fewer than 20 tasks, a mean, or a number with no dates is a story.
If you run agents, read the usage blocks in your own transcripts before believing any estimate. Our post on what a Claude Code session costs at startup shows how to measure one session from the command line.
Common questions
How do I count tokens from a Claude Code transcript?
Each assistant message has a usage block with input, cache creation, cache read and output tokens. Sum them per message id, counting each id once, over the time window you care about.
Why the median and not the mean?
Agent tasks have long tails. One task that loops or reads a huge file would drag a mean far from the usual task; the median ignores it.
Why 20 tasks?
It is a floor, not a statistical proof. With fewer, a couple of odd tasks decide the result. Twenty before and twenty after is small enough to reach in a week of real work and large enough that one outlier does not.
Melta is free; join the waitlist.