The Caveman skill is built to cut output tokens. It added 17.7%.

A skill is a text file that tells a coding agent how to behave. The Caveman skill tells it to write like a caveman: no articles, no pronouns, short bursts. Fewer words out means fewer output tokens, and output tokens carry the highest price per token, so the claim is that the same work costs less.

We tested it the way you would want it tested. Same tasks, same model, same settings, the only difference being whether the skill was on. Twenty terminal tasks with real build and debug work, plus 20 short coding exercises. The skill compressed the writing exactly as advertised. It still cost more.

Every headline number below is a median. Rank the 20 tasks by how much the skill changed them, and the median is the midpoint: half the tasks did better, half did worse. We report medians rather than averages because a single very long task would otherwise decide the answer on its own. As it happens, one does.

There is a real exception, and it is worth more than the headline. Add up total spend across all 20 tasks and the skill comes out 14.0% cheaper, because one very long session swamps everything else. That session is genuine. It is also one session in twenty.

Output tokens are what the skill exists to reduce. The median task produced 17.7% more of them, and the range runs from 3.2% to 43.3% more. Zero is not in that range. Before running anything we wrote down what would count as success: output tokens had to fall, and the session had to come out at least 10% cheaper. Both moved the other way.

Cost followed. The median task cost 25.4% more, and 15 of the 20 tasks got pricier. If the skill made no difference at all, a split that lopsided would turn up about 4% of the time, so this is not one or two odd tasks dragging the rest. We could not detect any quality difference: 29 of 40 runs passed the tasks’ hidden tests, checks the agent never sees, with the skill on, against 27 of 38 without it. At this sample size only a very large quality gap would have shown up, so read that as no difference detected rather than proof of none. The extra spend bought nothing we can measure.

The interesting part is that the skill was aimed at a real target and still lost. Pooled across all 78 runs, re-reading the conversation is 49.0% of the cost, and input the model is seeing for the first time is another sixth. Output, the part the skill actually touches, is 33.3%, a third of the bill and the second largest line item. A third of the bill is worth attacking. The skill simply did not compress it.

Here is the number that cuts against our own headline. Stop weighting each task equally, add up what all 20 actually cost, and the skill comes out 14.0% cheaper. Both figures are correct. They disagree because one task is enormous and the sum notices while the median does not.

The task is called path-tracing-reverse. Stock Claude Code spent 98,000 output tokens on one attempt and 188,000 on the other. With the skill on, both attempts came in around 50,000. All four runs passed, so this is not an agent giving up early. The skill saved $4.08 on that one task, against a total difference across all 20 tasks of $2.00. Remove it and the remaining 19 tasks cost 25.2% more.

What separates that task from the rest is not how long it ran. It is whether the skill managed to make the agent write less at all. It managed that on six of the 20 tasks, and five of those six came out cheaper. On the 14 tasks where the agent wrote more, all 14 cost more. Put output change next to cost change and the two move together almost perfectly, explaining 95% of the variation between them. Both come out of the same session, so that is a pairing rather than proof one causes the other.

The prize is bigger than it looks, because the conversation transcript is the input. Every turn re-sends everything said so far, so a word the agent never writes is a word nobody pays to re-read on any later turn. Change in input follows change in output at a correlation of 0.94. Output is a third of the bill by itself, and it drags the other two thirds along behind it.

Session length is only a rough stand-in for this. The three longest sessions are all cheaper, though one of them, make-mips-interpreter, failed its tests in both conditions, so that row is cheaper failure rather than cheaper work. Fitting a line puts the crossing point near 58,000 output tokens, but that fit explains only 11% of the variation and the exceptions sit right next to each other in the table. fix-ocaml-gc is 40.2% cheaper at 9,675 output tokens. Length is worth knowing. It is not the thing to plan around.

One row in the table needs a warning. compile-compcert tops it at 240.7% more expensive, but three of its four runs were killed on a time limit, so their token counts are cut off rather than finished. We left it in, because dropping rows once you have seen them is worse than keeping them, but it is not a real 241% penalty. Take it out and the odds that the cost split is just chance rise from 4% to 6%.

The second benchmark is 20 short coding exercises, the kind that take one file and a few minutes. Here the design is stronger: instead of running all the stock attempts and then all the skill attempts, we shuffled the order once and alternated them, so neither condition could benefit from a warm cache the other one paid to fill.

Cost came out 0.8% lower and output tokens 2.0% lower. Both ranges comfortably include zero. This is the shape of a null result, and the honest description is that we could not measure any difference at all, not that the skill saved a little.

One detail from these runs is worth keeping. We counted the characters of the agent’s own commentary, the running narration that the skill is pointed at, and it got 28.1% longer with the skill on. A caveman sentence is shorter than a normal sentence. It seems the agent wrote more of them.

Getting the skill to apply at all took four tries, and if you install it you will hit the same wall. Its own README says it is active from the first message. It is not. Installing the skill and starting work produced ordinary prose every time.

Seeding the agent’s memory folder did not work. Adding a line to the system prompt telling it to use caveman mode did not work either, even though the logs confirmed the prompt change was in place. Pasting the whole skill file into the system prompt not only failed to produce caveman output, it broke the agent’s willingness to act: it started asking permission before writing files. Only an instruction in an actual user message worked.

So that is what the skill runs got: the skill installed plus a user message telling it to write in caveman style. The stock runs got neither, and everything else, the task wording, the model and its settings, was the same on both sides. That extra message is part of what we are measuring, because there is no way to switch the skill on without it. We then read the transcripts to check the voice actually changed. It did, which rules out the boring explanation that the skill never applied.

Which raises a cost we did not measure. Somebody has to read this. The samples below are a few seconds each; a real session is hours of it, and following an agent that has stopped using pronouns is harder work than following one that has not. We have no data on comprehension or on how it wears over a day, so we are not going to claim a number. But it is worth forming your own view before installing something that also costs more, and the samples are there so you can.

Twenty tasks is not many, and task to task variation here is large. The cost range excludes zero, so the direction holds up, but the size of the effect is loosely pinned. Everything ran on one model at one reasoning setting. The arithmetic behind the break-even happens to carry over to the larger model in the same family, since Opus prices every rate at exactly 1.667 times Sonnet’s, so the cost and the saving scale together.

Order matters more than we would have liked. Running one condition before the other leaves a warm cache for whichever goes second, and on the terminal tasks we could only swap which condition went first from one batch of tasks to the next, not run by run. Measuring it directly, that ordering shifts the cost change by about 13 percentage points. Real, and not close to enough to flip the sign.

A result that would change our mind: a team whose typical session runs past 58,000 output tokens, measured the same way with both conditions on the same tasks, coming out cheaper. Our own data points that direction at the long end. We just did not find many sessions that long.

If you are looking at your agent bill and reaching for a skill that makes the model talk less, this is the wrong lever. On normal coding work it costs you about 25% more and buys no correctness we could measure. The idea is appealing because output tokens are priced highest, and output really is a third of the bill, so the room to save is there. It fails for a simpler reason. The running commentary the skill compresses is a sliver of what an agent writes, and on most tasks the skill did not compress anything at all: output went up.

There is one case worth acting on. If your sessions genuinely run tens of thousands of output tokens, long autonomous builds rather than edit and check loops, measure it. Our longest three sessions all got cheaper, one of them by 67%. Run the same comparison on your own tasks, skill on and off, before rolling anything out, because which side of the crossing point you sit on decides the answer, and it flips completely.

For our own work we are not using it. On the sessions we actually run, the saving is not there, and clipped output is a chore to read alongside for hours. Either of those on its own would be arguable. Together they made it an easy call.

The wider lesson is about how these claims get made. A skill that compresses visible text produces an obvious, checkable change in what you see on screen, and that felt improvement is easy to mistake for a cheaper bill. The two are not the same measurement. If you want to know what something costs, run both conditions on the same tasks and read the token counts. That is the kind of question we help teams answer, and you can start with our readiness assessment.