RTK reported 76% less shell output. Did that make Claude Code cheaper to run?
RTK shortens shell output before a coding agent reads it. RTK’s own counter said it removed 76% of that output in our main Opus 5 test. But Claude Code reported 5.8% more total cost across the 18 runs with RTK than across the 18 runs without it. When we compared costs separately for each task, the range included both lower and higher cost. RTK did not produce a reliable cost saving.
RTK’s estimate in the main Opus 5 test
total across 18 runs per setup: $2.66 versus $2.52
main low effort test; all 18 runs without RTK passed
The short answer
In the 18 low effort Opus 5 runs with RTK, rtk gain estimated 76% less shell output for the commands it handled.
Across the 18 runs in each setup, Claude Code reported $2.66 total with RTK and $2.52 total without it. RTK cost 5.8% more across those runs.
Looking task by task, the middle cost change was 2.4% more with RTK. Its 95% range ran from 18.0% less to 33.3% more. The result does not show that RTK reliably raises cost. It shows that the large shell output saving did not produce a reliable cost saving.
Total cost across 18 runs in each setup
5.8% more total with RTKThese totals cover the 18 low effort Opus 5 runs in each setup. Twelve comparisons cost more with RTK and six cost less.
RTK measured only one part of the session
The rtk gain command counts the output from shell commands that RTK processes. It does not count instructions, files read by the agent, the agent’s own responses, or earlier messages sent back to the model.
That difference matters. A smaller command response can save tokens on one model call. It cannot tell us how many calls the agent will make before finishing the task.
The saving was also concentrated. Directory listings produced 91.5% of the tokens RTK said it saved in the main test. The 15 cat commands saved nothing.
What RTK counted
76% less outputThis chart covers only the main Opus 5 test. RTK recorded some saving for 14 of 36 commands. The other 22 showed no saving.
These token counts are RTK’s estimates, not model provider measurements. Claude made 44 shell calls in the RTK runs. Of those, 21 began with rtk. RTK’s database contained 36 processed commands because one shell call can contain more than one command. We did not determine how many other shell calls were eligible for RTK. This result measures RTK as its automatic Claude hook applied it.
The full sessions moved in the other direction
In the main test, RTK runs produced 6.8% more model output and sent 3.9% more new input to the model in total.
Shell calls did not increase. They fell from 48 without RTK to 44 with it. The benchmark therefore does not show that RTK made the agent run more shell commands.
RTK changes the text Claude sees, so it can change what Claude does next. In our runs, Claude produced 6.8% more output and sent 3.9% more new input with RTK, despite making four fewer shell calls. One possibility is that the shorter command output changed how Claude reasoned or what it inspected next. Another is ordinary variation between runs. The cost range includes both savings and an increase, so this experiment cannot distinguish between those explanations.
How we tested it
We used six tasks from Aider Polyglot 1.0. Each comparison started from the same code and used the same model settings. One run used RTK. The other did not. Hidden tests checked whether each agent completed the task.
The benchmark included Claude Opus 5 and three GPT-5.6 model names through Codex CLI. We tested Opus 5 at low and high effort. We tested Sol at low, medium, and high. Terra and Luna received medium and high tests. We did not run every effort level for every model.
Before running the benchmark, we chose how to judge the main result. For each task, we compared the middle result from three attempts with RTK and three without it. We then took the middle change across the six tasks. RTK had to reduce this measure by at least 10% without causing more task failures.
We counted only new input sent to the model, not input reused from the provider’s cache. Both parts of the main target failed. The input reduction was below 10%, and one more RTK run failed the hidden tests.
one each in Python, JavaScript, Go, Rust, Java, and C++
the same task with RTK and without RTK
48 with Claude Code and 84 with Codex CLI
every result was graded with hidden tests
The cost chart and cost range use only the 36 runs from the main low effort Opus 5 test. Pooling low and high effort produces an 8.9% increase. We do not use that as the headline because low effort received three attempts per task while high effort received one.
The input results were also weak
The measure we chose before the test showed 0.5% less new input with RTK. Its 95% range ran from 10.0% less to 25.2% more. Total new input across all 18 runs was 3.9% higher with RTK. Together, those results do not show a clear direction.
Task success did not meet our requirement either. All 18 runs without RTK passed. Seventeen of the 18 RTK runs passed.
95% range: 10.0% less to 25.2% more.
113,687 tokens without RTK and 118,070 with it.
All 18 runs without RTK passed.
We also made 42 comparisons with Codex CLI. No Codex setting reached 10% less input when we compared each task separately using the measure chosen before the benchmark. GPT-5.6 Sol at medium effort came closest, with 7.2% less. Its 95% range ran from 34.9% less to 14.7% more, and 4 of 6 tasks passed in both setups.
Every Codex input range included both less and more input. Some settings also had low task success. Standard pass rates ranged from 1 of 6 to 5 of 6. RTK pass rates had the same range.
A different calculation gives a different view. When we added input across all runs in each setting, Sol at medium effort used 12.6% less and Terra at high effort used 16.5% less with RTK. We did not use those totals for the main test because we had chosen to compare each task separately before the runs.
Codex CLI did not provide comparable dollar costs. We passed the displayed model names to the CLI, but its output did not include a separate model ID from the server, so we could not verify whether the names used different internal model builds. Codex prefixed 65.5% to 95.8% of its shell calls with RTK, depending on the model name and effort.
What we learned
In the main Claude test, RTK claimed 76% less shell output. We did not find a reliable reduction in cost or new model input. Task success was one run lower with RTK.
RTK may still make noisy command output easier for an agent to read. But a shell output counter cannot show whether the agent finishes with fewer model calls, lower cost, or the same task success.
If cost is the goal, test complete tasks with and without the tool. Count failed tasks. Measure the full session. That is more work than showing a compressed shell response, but it answers the question people actually care about.