Caveman skill tested: 38% fewer output tokens, every answer still right, and a higher bill for one-off questions

Summary

If you want shorter answers from Claude Code, the caveman skill delivers them without losing correctness. Expect about a third fewer output tokens, not three quarters, and use it for long sessions, not one-off questions.

The reason is simple: we measured it. We asked 20 developer questions, three times each, in fresh Claude Code sessions on Haiku, with no skill, with caveman and with ponytail, and checked every answer automatically:

• Shorter, still right. Caveman cut output tokens by 38% in total and the median answer from 128 words to 55. All 60 caveman answers were right, the same as with no skill.
• About a third, not three quarters. The median question saved 34%. Savings ran from 76% on a small coding task to 16% more tokens on one git question. An earlier version of the caveman skill promised about 75%.
• One-off questions cost more. Each fresh session pays for the skill’s own instructions, about 1,800 input tokens. On these short answers that outweighed the output saved: caveman’s 60 runs cost 11% more than no skill.
• Ponytail is not a brevity skill. It wrote 50% more output tokens, because it ends every answer with what it skipped and what to check. It did write the least code.

Our picks

1. Turn caveman on for a working session, not for a single question. In one long session the skill’s instructions are cached after the first turn, while the shorter answers keep adding up. In a fresh session for every question, as in our test, you pay the instructions each time.
2. Expect the most savings on code. On the six coding tasks caveman cut output by 55%. It dropped the >>> usage examples that 8 of the 18 plain answers added, and its code answers had a median of 50 words against 126. On explanations and commands it cut 25%.
3. Keep reading the full answer for anything risky. Caveman’s rules switch back to full sentences for security warnings and irreversible steps, and our rm -rf question still got its warning. That is by design; do not strip it out.
4. Use ponytail for code changes, not for chat. Its rules aim at the smallest code change: median code lines fell from 5.5 to 3. Its required closing lines make short answers longer. Ponytail’s own benchmark, on multi-step coding tasks, reports 16% fewer tokens than no skill; on our single questions it was 50% more.
5. Measure on your own work before you believe a percentage. Our median saving was 34%, the README’s example shows 68%, a Reddit review says 65%. The number depends on how wordy the model is without the skill.

What do you mostly ask Claude Code for: explanations you read, or code you paste? Caveman helps the first a little and the second a lot.

How we tested

1. Questions. 20 developer questions: 8 explanations (a Python mutable default, HTTP 401 against 403, a stale React closure), 6 commands (undo a commit, find large files) and 6 small Python functions (parse 1h30m, validate an IPv4 address).
2. Arms. No skill; caveman, with JuliusBrussee/caveman’s skills/caveman/SKILL.md in the project and the prompt starting with /caveman; ponytail, with DietrichGebert/ponytail’s skills/ponytail/SKILL.md and /ponytail. Both files as of November 7, 2026.
3. Runs. Each question three times per arm, each run a fresh claude -p session in an empty folder: 180 runs on Claude Code 2.1.314 with --model haiku and --setting-sources project, so no user settings or hooks loaded. Tokens, cost and time come from Claude Code’s JSON output.
4. Scoring. Explanation and command answers must contain the key facts, for example IS NULL for the SQL question or git reset --soft HEAD~1 for undoing a commit. Code answers must define the function and pass our tests. Two rules misfired on the first pass and were fixed for all arms: code is taken from the first block that defines the function, since plain answers add a block of >>> examples, and deps counts as naming the dependency array. Before the fix, no-skill code looked 50% correct; after, all three arms scored 60 of 60.

Results

All 20 questions, three runs each

	No skill	Caveman	Ponytail
Answers right	60 of 60	60 of 60	60 of 60
Output tokens, total	38,066	23,619 (−38%)	57,108 (+50%)
Median answer	128 words, 439 tokens	55 words, 265 tokens	126 words, 763 tokens
Median code lines	5.5	3	3
Median input tokens per run	18,001	19,823	19,226
Cost, 60 runs	$0.108	$0.120 (+11%)	$0.136 (+25%)
Median time per run	5.4 s	4.6 s	6.9 s

Output tokens by kind of question, total

	No skill	Caveman	Ponytail
Explanations (8)	17,095	12,770 (−25%)	21,459 (+26%)
Commands (6)	4,612	3,480 (−25%)	8,527 (+85%)
Code (6)	16,359	7,369 (−55%)	27,122 (+66%)

Where caveman saved most and least, median of three runs

Question	No skill	Caveman	Change
chunk a list	393	94	−76%
listening ports	440	125	−72%
is_palindrome	590	172	−71%
HTTP 401 against 403	343	169	−51%
is against == for 257	664	756	+14%
undo the last commit	277	321	+16%

Why does a skill that cuts output make a run more expensive? Every Claude Code session starts with about 18,000 input tokens of system prompt and tool definitions, and a fresh session has to write most of them to the prompt cache. Working back from the 180 runs, writing to the cache is 75% of the cost of a no-skill run and output is 18%. Caveman adds about 1,700 tokens to that write and saves about 240 output tokens per answer, and output costs only 2.5 times as much per token as a cache write. In a long session the instructions are written once and then read from the cache at a twentieth of the price, so the balance tips the other way.

What to ignore

• The 75% figure. It came from an earlier caveman description. The current README leads with an example, 63 tokens down to 20, and whole-session input savings from other caveman tools, not this skill.
• Speed claims. Caveman was slightly faster here, 4.6 against 5.4 seconds, but these are single short answers, where network time dominates.
• Ponytail’s extra prose as a flaw. Its closing lines on what was skipped are the point of the skill. Judge it on code changes, which this test did not cover.

Limitations

• Haiku only. We run batches on Haiku. A larger model that writes longer answers without a skill could save more with caveman.
• Single questions in fresh sessions. This is the worst case for any skill’s instruction cost. We did not test long agent sessions with tool calls, where caveman’s rules on status lines also apply.
• Automatic scoring checks key facts and tests, not nuance. An answer can pass and still leave out a caveat a reader would want.
• The skills change. Both repos update often; we used the files as of November 7, 2026, and the data names each one.

Reproduce it

The script is caveman-skill-test-2026-11.py. Download the two SKILL.md files, run python3 caveman-skill-test-2026-11.py run <folder> <caveman SKILL.md> <ponytail SKILL.md> with Claude Code logged in, then score <folder>. It writes caveman-skill-test-2026-11.json: one row per run with tokens, cost, time, code lines and whether the answer was right, plus every summary in this post. Both skills are in our skills directory, and ponytail’s star growth is in our trending skills report.

Sources

1. JuliusBrussee/caveman · official
2. DietrichGebert/ponytail · official
3. Claude Code docs: Agent Skills · official
4. AgentCraft: Claude Code skills directory · blog