Same Code, 73% More Tokens: The Hidden LLM Cost No Pricing Page Shows

Two numbers determine your LLM bill: the tokens your model counts for your input, times the price per token. Pricing pages show the second. The first — decided by the tokenizer — is something almost nobody talks about. A new analysis published July 13 from Playcode reveals that for the same TypeScript file, Claude's tokenizer produces 73% more tokens than GPT's. Before you even multiply by the sticker price.

This is not a bug. It is how tokenizers work. But if you run AI coding agents — where code dominates the workload — this is a hidden cost that no pricing page prints.

A token is not a fixed unit

A token is neither a character nor a word. It is a chunk of text — typically 3-4 characters in English — but every model slices differently. The keyword function might be 1 token under GPT and 2 under Claude. Multiply across hundreds of thousands of lines, and the gap becomes real money.

Playcode measured this exactly: 16 real fixtures — TypeScript, Python, Rust, HTML, JSON schemas, system prompts — counted byte-for-byte through each vendor's official tokenizer. No estimates:

Same TypeScript file (2,888 characters):

Model	Tokenizer	Token count
GPT-5.x	o200k_base	681
Claude Opus 4.6	Old tokenizer	898
Claude Sonnet 5 / Opus 4.8	New tokenizer	1,178

Claude's new tokenizer counts 73% more tokens than GPT's. Even Claude's previous tokenizer was 32% higher. The worst part: TypeScript shows the largest gap in the entire benchmark — and happens to be the language AI coding agents write most today.

The coding agent math

For casual chat API use, the tokenizer gap is modest — English prose shows only ~34% difference. Coding agents are another story. Every time an agent sends context — current file, stack trace, system prompt, tool schemas, previous tool results — everything counts toward the token tally.

Assume your agent sends 50KB of code per request, 200 requests per day:

• GPT-5: ~12,000 tokens/request → ~2.4M tokens/day
• Claude Sonnet 5: ~20,700 tokens/request → ~4.1M tokens/day

Same workload, Claude counts 72% more tokens. Now multiply by price: Sonnet 5 at $3/1M input tokens (introductory $2), GPT-5.6 Sol at $3.75/1M:

	GPT-5.6	Claude Sonnet 5 (intro)	Claude Sonnet 5 (post Aug 31)
Tokens/day	2.4M	4.1M	4.1M
Cost/day	$9.00	$8.20	$12.30
vs GPT	—	9% cheaper	37% more expensive

This is why comparing models by price-per-token alone is meaningless. What matters is real cost per task.

What changed inside the tokenizer

Tokenizers use BPE (Byte-Pair Encoding) — they learn to merge frequently co-occurring byte pairs into a single token. A tokenizer trained primarily on natural text will produce more tokens for code because its vocabulary is not optimized for programming languages.

OpenAI has kept o200k_base frozen since GPT-4o (May 2024) — stable, community-audited. Anthropic changed tokenizers at least twice: the old one (Opus 4.6 and earlier) and the new one (Opus 4.7+, Sonnet 5, Fable 5). Anthropic's own pricing documentation confirms: "Claude Opus 4.7 and later Opus models, Claude Fable 5, Claude Mythos 5, Claude Mythos Preview, and Claude Sonnet 5 use a newer tokenizer."

The direct consequence: upgrading from Opus 4.6 to Opus 4.8 at the same $5/$25 list price increased real code cost by ~31% — purely because the new tokenizer slices code into more pieces. An unannounced price hike.

When does the tokenizer gap not matter?

A more expensive tokenizer is not always a problem. Three cases where this gap becomes irrelevant:

1. When output quality dominates. If the model finishes a task in 1 round-trip instead of 3, total tokens may be lower despite higher per-request counts. Sonnet 5 hits 63.2% on SWE-Bench agent coding, approaching Opus 4.8 (69.2%) at 40% of the price — worth paying extra tokens for better code.

2. When you are not sending code. English prose shows only a ~34% gap. For chatbots and content generation, the tokenizer is not the deciding factor.

3. When you have volume discounts. Batch pricing or reserved capacity can offset tokenizer differences. But you need real numbers to do the math.

The issue is not which tokenizer is "better." The issue is that nobody is transparent about it.

What this means for you

Stop comparing price-per-token. Compare cost-per-task. Build a real benchmark: same prompt, each model's API, read usage.prompt_tokens and usage.completion_tokens. Multiply by price. Five minutes of work saves hundreds per month.

Track token counts across model upgrades. Every time Anthropic ships a new model, the tokenizer may change. A flat price does not mean a flat bill. Compare token counts before and after every upgrade.

Code is the hardest-hit workload. Prose differs by 34%. Code differs by 73%. If your agent sends 70-80% code context, your bill nearly doubles before per-token pricing even applies.

Vendor lock-in via tokenizer is real. Once you optimize prompts for one tokenizer, switching models is not just changing an endpoint. The same prompt may exceed context windows or blow up costs. The less a tokenizer changes, the easier migration becomes.

Bottom line

The number on the pricing page is a starting point, not an answer. Your tokenizer — the component that decides how many pieces your code becomes — is a hidden cost you pay every day. As coding agents go mainstream, this is no longer optional to ignore.

Next time you compare models, do not look at the token price. Open your console, run a real request, and read the actual token count. That is the number you are really paying for.

---

Content assisted by AI (Amy 🌸). Reviewed by the author.