GigaToken: 1000x Faster LLM Tokenization — What Developers Need to Know

24.53 GB/s. That's the text tokenization speed of GigaToken — a Rust project that hit Hacker News on July 22, 2026 and immediately blew up with 279 points and 51 comments.

For context: HuggingFace tokenizers and OpenAI's tiktoken — both multithreaded Rust, both optimized over years by a large community — process the same file at 25–36 MB/s. GigaToken is roughly 1000x faster than HuggingFace and 700x faster than tiktoken. Not on one cherry-picked benchmark — across 20+ tokenizer definitions, on AMD EPYC, Apple M4 Max, and consumer Ryzen.

Orders of Magnitude, Not Increments

When HuggingFace measures in MB/s and GigaToken measures in GB/s, "faster" is the wrong word. This is a unit change:

Tokenizer	GigaToken	HF tokenizers	tiktoken	vs HF	vs tiktoken
GPT-2	24.53 GB/s	24.8 MB/s	36.0 MB/s	989×	681×
Qwen 3	22.16 GB/s	34.2 MB/s	—	648×	—
DeepSeek V4	19.69 GB/s	26.2 MB/s	—	750×	—
Llama 4	20.77 GB/s	72.7 MB/s	—	286×	—

On AMD EPYC 9565 (144 cores). On Apple M4 Max (16 cores), the GPT-2 tokenizer reaches 8.79 GB/s — 1,268× faster than HF.

Author Marcel Rød broke down the technique on HN: replace the regex engine (traditionally handling pretokenization) with SIMD, optimize pretoken mapping cache (look up already-seen words instead of re-tokenizing), minimize branching, and limit Python-to-thread interaction. The cache is the hardest part: pretoken distributions are long-tailed — a few words appear millions of times, most appear only a handful. Building an efficient cache under that distribution is non-trivial.

Tokenization Is <0.1% of Inference. Why Care?

First question on HN — and it deserves a real answer.

Training data pipelines. Tokenizing 1 TB of text at 25 MB/s takes about 11 hours. At 24.53 GB/s: roughly 41 seconds. For ML teams iterating on datasets, cutting the wait from "almost half a day" to "under a minute" fundamentally changes the development loop.

Inference routing. An AI platform operator on HN confirmed: they tokenize early to make routing and rate-limiting decisions before requests hit the GPU. The tokenizer sits on the critical path — every millisecond multiplies across millions of daily requests.

Time-to-first-token (TTFT). For smaller models, tokenization can be a meaningful fraction of TTFT. For latency-focused providers like Groq or Cerebras, microseconds matter.

The real question. If tokenization — something the field accepted as "optimized enough" — still had 1000x headroom, what else in the inference pipeline is leaving similar gains on the table?

How to Use It

Install via pip. Two modes:

import gigatoken as gt

# HuggingFace compatibility — one-line swap
tokenizer = gt.Tokenizer(hf_tokenizer).as_hf()
tokens = tokenizer.encode_batch(["This is a test", "And another one"])

# Native mode — maximum throughput, no Python overhead
tokenizer = gt.Tokenizer("Qwen/Qwen3-8B")
file_source = gt.TextFileSource(["data.txt"], separator=b"[REMOVED_SPECIAL_TOKEN]")
tokens = tokenizer.encode_files(file_source)

Token-for-token identical to HuggingFace output. Supports 20+ tokenizers: GPT-2, Llama 3/4, Qwen 2/3/3.6, DeepSeek V3/R1/V4, Phi-4, OLMo 2/3, Gemma 3/4, GLM 4/5, Kimi K2, Mistral 7B, and more.

The Bigger Picture

There's a striking paradox here. The AI industry pours hundreds of billions into GPUs, novel architectures, and scaling laws — things promising 10–20% gains. Meanwhile, one engineer with Rust and SIMD found a 100,000% speedup in a component everyone agreed was "already optimized."

The pattern keeps repeating. 2023: MLC-LLM used smart compilation to speed up Llama inference on laptops several times over. 2024: GGUF and quantization techniques enabled running 70B models on personal hardware. Breakthrough performance leaps come from low-level software engineering, not from new model architectures.

For developers, GigaToken is a practical new tool — especially useful when building data pipelines, inference platforms, or any system processing massive text volumes before they reach an LLM. But the bigger lesson is about mindset: don't assume anything is "fast enough." There's always headroom — sometimes 1000x.

Takeaways

• GigaToken hits 24.53 GB/s tokenizing GPT-2 on AMD EPYC 9565 — ~1000x faster than HuggingFace, ~700x faster than tiktoken
• Rust + SIMD replacing regex for pretokenization + smart caching + minimal Python overhead
• Drop-in replacement for HuggingFace tokenizers and tiktoken — swap one line of code
• Supports 20+ tokenizers: GPT-2, Llama, Qwen, DeepSeek, Phi, OLMo, Gemma, GLM, Kimi, Mistral...
• Training data prep: 1 TB tokenized in ~41 seconds instead of 11 hours
• Inference routing: tokenizer on the critical path for platforms handling millions of requests/day
• Open question: if tokenization had 1000x headroom, what else in the pipeline is waiting?

Content assisted by AI (Amy 🌸). Reviewed by the author.