Qwen3.8 27B: 4-bit Quantization Holds Up, 1-bit Falls Apart

How much VRAM do you actually need to run Qwen3.8 27B without losing quality? An independent benchmark just answered that with real numbers, not Reddit vibes.

The researcher, writing on the Quesma blog, burned roughly $3,000 renting GPUs on Modal to run three benchmarks — GPQA Diamond, IFBench, and Terminal-Bench 2.1. The tests covered four quantization levels of Qwen3.8 27B: 8-bit (29GB), 4-bit (17GB), 2-bit (10.7GB), and 1-bit (6.2GB). The results confirm something many people suspected but nobody had measured carefully. Compression doesn't degrade quality evenly. It holds steady for a while, then falls off a cliff.

A 55GB Model, a 17GB Copy — No Measurable Difference

The full BF16 model weighs 55GB, out of reach for most consumer hardware. The Q4_K_M (4-bit, 17GB) build from Unsloth on Hugging Face fits neatly on a 24GB card like the RTX 4090, with room left over for around 64,000 tokens of context.

Here's the surprising part: on Terminal-Bench 2.1, an agentic coding benchmark with 89 tasks, the 17GB version scored the same as the full 55GB model. Not "close" — genuinely no measurable gap. The researcher first replicated Qwen's official BF16 scores before running the comparison, so this isn't a sloppy measurement.

GPQA Diamond tells a similar story: almost no difference down to 4-bit. Only the 2-bit version showed a clear drop. On IFBench, which measures instruction-following, even the 2-bit build (under 11GB) kept pace with the original.

2-bit Still Works, 1-bit Doesn't

The UD-Q2_K_XL (2-bit, 10.7GB) build is the interesting edge case. On Terminal-Bench it dropped noticeably compared to 4-bit, but still landed in the range of Opus 4.7 or Gemini 3.1 Pro on the same leaderboard — not frontier-level, but far from useless. On tasks it did solve, it took roughly the same number of turns as the full model, just about a quarter more tokens to get there.

The UD-IQ1_S (1-bit, 6.2GB) build tells a completely different story. On GPQA Diamond it scored around random chance. Worse, cranking reasoning effort up from low to "xhigh" — which normally improves output — made it worse, because the model burns through its token budget thinking and returns an empty answer. Unsloth advertises the 1-bit build as retaining about 72% top-1 accuracy at 89% of the size cut, but the benchmark shows that missing 28% matters far more than the headline number suggests.

Worth noting: this is one independent researcher's benchmark, not an official test from Alibaba or the Qwen team. It's credible — the methodology is transparent and the official scores were replicated first — but it's still one data point, not a final verdict.

What This Means for Developers

• Running locally on a consumer GPU: Q4_K_M (17GB) is the sensible default for an RTX 4090 or similar 24GB card — no meaningful quality trade-off versus the full model.
• Tighter hardware (under 12GB VRAM): the 2-bit UD-Q2_K_XL build still works for simpler tasks, but don't expect full coding-agent performance.
• Skip 1-bit for reasoning-heavy work: for science, logic, or complex coding tasks, the 1-bit build is close to useless — the size savings aren't worth the quality loss.
• Reasoning effort matters more than expected: the low/medium/xhigh setting swings scores more than dropping from 8-bit to 4-bit — worth tuning before switching quantization.

The researcher's takeaway is worth repeating: quantization should be embraced, not feared. The real question isn't whether to compress a model, but how far — and for Qwen3.8 27B, 4-bit is the safe stopping point. 1-bit isn't worth running at all.

---

Content assisted by AI (Amy 🌸). Reviewed by the author.