DeepSeek V4 Flash on a Single AMD MI300X: 304B Params, One GPU

What happened?

Ryan Zhou published a GitHub repo. A Docker Compose file. A 304B-parameter LLM running on one AMD MI300X GPU.

No additional quantization. No weight offloading over PCIe. No 8-GPU cluster. The entire model lives in the card's 192 GB of HBM3.

The result: 168.6 tok/s single-stream decode. Production-grade throughput. This isn't a lab experiment — the repo ships with SHA-256 pins, FP8 patches, AITER tuning tables, and a Caddy reverse proxy. Deploy-ready.

Until now, running a model at this scale meant NVIDIA. Two to eight H100s. NVLink. NCCL configuration. The AMD MI300X, with 192 GB HBM3 and roughly half the list price of an H100, is breaking that formula.

Why this matters

192 GB HBM3 is the strategic advantage

The MI300X carries 2.4× the HBM of an H100 SXM5: 192 GB vs 80 GB. DeepSeek V4 Flash uses mixed FP4/FP8 precision — the full weights (156.67 GiB) plus KV cache fit comfortably on a single card.

The result: the hardest problem in large model deployment — multi-GPU interconnect — simply vanishes. No NVLink bottleneck. No NCCL deadlocks. No load balancing. One GPU, one model, one process.

CUDA is no longer an absolute wall

This is the first time a frontier model has been production-tuned on a single AMD GPU. vLLM now has mature ROCm support. The community is writing CDNA3 kernels by hand. AITER tuning fills the gaps. Three puzzle pieces that, together, produce something unthinkable a year ago.

A separate repo from AgntroAI demonstrates hand-written CDNA3 kernels achieving 3.1× speedup (20.7 → ~64 tok/s) over stock vLLM. Lossless — GSM8K holds at 96.3%.

The road was paved with sharp edges

FP8: two standards, two worlds

The biggest trap. The MI300X (CDNA3) uses FNUZ E4M3 — an AMD/Graphcore variant. MI325X and newer use standard OCP FP8.

A kernel that reads FNUZ bytes as OCP produces a factor-of-two scale error. The model outputs garbage but doesn't crash — exactly the kind of silent failure that's hardest to catch. Zhou had to write format-specific overlays: float8e4b8 with FP8_MAX=224.0, plus a 16×16 preshuffle for AITER.

MoE at batch=1: 15/16 computations wasted

Stock vLLM routes MoE through Triton tile-GEMMs. The hard tile is M=16, but batch=1 only needs M=1. Fifteen out of sixteen calculations are discarded. AgntroAI replaced the entire path with custom dequant-GEMV kernels — the primary source of their 3.1× speedup.

The speculative decoding paradox

DeepSeek ships DSpark-7 — a speculative decoder that predicts 7 tokens ahead to accelerate generation. But on the MI300X with optimized kernels, DSpark-7 is slower than pure single-stream decode. Why? Speculative verification runs at M≥2, exactly the slow tile-GEMM path. AgntroAI audited thoroughly: acceptance rate ~2.4, below break-even. Verdict: turn it off to go faster.

An interesting question: if the model's built-in acceleration mechanism becomes a bottleneck on different hardware, is "optimized for NVIDIA" becoming an implicit barrier to hardware diversity?

What developers should know

1. Inference economics are about to shift

When a 304B model runs on a ~$10-15K card instead of a $100K+ cluster, the cost equation flips. Inference API providers gain more hardware options — competitive pressure drives prices down. Developers benefit first.

2. Self-hosting becomes realistic

One MI300X costs roughly half an H100. A small startup buys 1-2 cards, runs their own model, cuts the OpenAI API dependency. No rate limits. No per-token variable costs. ROI in months at sufficient volume.

3. AMD doesn't need to win at training

This is the most overlooked take. NVIDIA dominates training through 15 years of CUDA ecosystem accumulation. But inference — 70-80% of total model operating cost — is a different game. Memory bandwidth matters more than compute. HBM capacity matters more than MFMA throughput. And on these two specs, the MI300X has structural advantages.

AMD doesn't need to beat the H100 at training. They just need to win at inference — where 192 GB HBM naturally dominates 80 GB. That's a realistic path to market share.

4. Software is still the barrier

Clone-and-run? Not yet. You need to understand FP8 dialects, debug kernels, and patch race conditions in vLLM (issue #47282 — the CPU→GPU KV restore fix hasn't been upstream-merged). But for teams with systems expertise, the barrier is manageable — and the payoff is real.

The numbers

Metric	Stock vLLM	+ Hand-written kernels	+ AITER tuning
Single-stream decode	20.7 tok/s	58.9 tok/s	~64 tok/s
Speedup	—	2.85×	3.1×
Accuracy (GSM8K)	96.3%	96.3%	96.3%

With DSpark-7 + tuned kernels (Zhou):

• 1 stream: 168.6 tok/s median
• 8 streams: 542 tok/s aggregate
• 64-stream burst: 830 tok/s, no OOM
• Prefill: 7.9–8.5K tok/s
• Validated max context: 256K (architecture supports 1M)

Caveats

HBM headroom is razor-thin. High-water mark: 204.5 of 205.8 GB. Bumping KV cache to 30 GB crashes with HSA_STATUS_ERROR_OUT_OF_RESOURCES. Multi-session throughput (batch>1) remains occupancy-bound — the CUs sit mostly idle because MoE compute intensity is low. This isn't a high-concurrency serving solution.

But for single-stream use — chatbots, coding agents, internal tools — this is a real leap forward. From 20.7 to 168.6 tok/s on the same GPU: an 8× improvement over stock vLLM.

Bottom line

On August 4, 2026, an engineer dropped a Docker Compose file on GitHub. Inside: a 304B-parameter model, one AMD GPU, 168 tok/s.

Not a blog post. Not a slide deck. Working code.

If your team is spending thousands per month on inference APIs — it's time to run the self-hosting numbers with AMD. You might be surprised.

---

Content assisted by AI (Amy 🌸). Reviewed by the author.