Is Speculative Decoding Worth It for Local LLMs in 2026?
Speculative decoding took a Qwen3.6-27B setup from 25 to 45 tok/s. It also took a mismatched Gemma 4 setup from 60 down to 45 tok/s. Acceptance rate decides which result you get.
Short answer
Speculative decoding is worth it when your draft acceptance rate is above roughly 60–70%. Below that, you usually lose speed.
- Best case measured: Qwen3.6-27B + MTP draft on a 24GB GPU: ~25 → ~45 tok/s (+78%).
- Worst case measured: Gemma 4 with a mismatched int4 draft at 41% acceptance: 60 → 45 tok/s (-25%).
Same technique, opposite results. The rest of this guide is how to land on the right side.
How it works, in one paragraph
A small “draft” predicts several tokens ahead. The big target model verifies them in one pass. Accepted tokens are free speed; rejected tokens are wasted work. The target model verifies every token, so output quality never changes — only speed does. That makes this the rare free lunch, but only when the draft guesses well.
The 2026 numbers (runaihome)
| Setup | Before | After | Change |
|---|---|---|---|
| Qwen3.6-27B + MTP, A10G 24GB | ~25 tok/s | ~45 tok/s | +78% |
| Qwen3.6-27B + MTP, Apple Silicon (82% acceptance) | ~7 tok/s | ~16 tok/s | ~2.3x |
| Gemma 4 + MTP coding agent, Apple Silicon MLX | — | — | ~1.9x |
| Llama 3.1 8B + 1B draft (draft length 5) | — | — | 1.83x |
| Gemma 4, mismatched int4 draft (41% acceptance) | 60 tok/s | 45 tok/s | -25% |
The pattern: code and structured output sit at the top of the 1.5–3x band, because code is predictable and drafts guess it well. Creative and open-ended chat often falls below break-even.
The 2026 llama.cpp flags (the old ones are gone)
llama.cpp renamed its speculative decoding flags. The old --draft, --draft-n, and --draft-max now exit with an error: “the argument has been removed. use —spec-draft-n-max”. Verified against the current llama.cpp source (common/arg.cpp).
| Old flag | New flag |
|---|---|
--draft-max / --draft-n / --draft | --spec-draft-n-max |
--draft-min | --spec-draft-n-min |
| (new) | --spec-type |
--spec-type selects the method, including draft (classic separate draft model), draft-mtp, draft-eagle3, draft-dflash, and several ngram modes.
The winning A10G config from the benchmark above:
llama-server -m qwen3.6-27b-q4.gguf \
--spec-type draft-mtp \
--spec-draft-n-max 2Starting points that measured well:
- MTP or creative text:
--spec-draft-n-max 2to4 - Standalone draft model on code: start at
--spec-draft-n-max 8
MTP vs a separate draft model
Multi-Token Prediction (MTP) heads are the 2026 shortcut: the model drafts for itself, so there is no second model to pick, match, or fit in VRAM.
- Gemma 4 on the MLX runner enables speculative decoding by default (and Ollama v0.30.5 added Gemma 4 MTP spec decode on Macs, per runaihome). You may already be running it.
- Qwen3.6 still needs a separate MTP GGUF (for example the unsloth Qwen3.6-27B-MTP-GGUF build) loaded alongside the main model.
If you use a classic separate draft, it must share the target’s tokenizer and family. The -25% Gemma 4 failure above came from a mismatched int4 draft — acceptance collapsed to 41%.
How to test if it pays on your machine
- Record baseline decode tok/s on your real prompts.
- Enable spec decode with
--spec-type draft-mtp --spec-draft-n-max 2. - Watch the acceptance rate in the server log. Above ~70%: keep it. Below ~60%: turn it off.
- Re-test per workload. Code loops and chat sessions give different acceptance rates.
- Check VRAM. A separate draft model steals memory from context — see KV cache quantization: Q8 vs Q4 if you get tight.
The +78% Qwen3.6-27B result above ran on a 24 GB GPU. That is the RTX 3090/4090 class: enough VRAM for a 27B quant plus its MTP draft and context.
Decision table
| Your workload | Verdict |
|---|---|
| Coding agent / structured output | Enable it — top of the 1.5–3x band |
| Qwen3.6-27B on a 24GB GPU | Enable with MTP GGUF (+78% measured) |
| Gemma 4 on Mac (MLX) | Already on by default |
| Creative writing / open chat | Test first — often below break-even |
| Mismatched or wrong-quant draft | Do not — this is the -25% case |
| VRAM already full | Skip, or use MTP instead of a second model |
Final recommendation
Enable speculative decoding for code and agent workloads — MTP first, since it needs no second model. Measure acceptance, and turn it off below ~60%. It never changes your output, so the only question is whether your workload is predictable enough to pay for the drafts.
Sources:
- runaihome: speculative decoding benchmarks 2026
- llama.cpp repository (arg parser: flag renames and —spec-type values)
Related guides
Need OpenClaw fixed live?
Remote rescue sessions for gateway, auth, tunnel, VPS, and model access problems.
See Rescue Session