# Speculative decoding

**Speculative decoding** is a family of inference methods for autoregressive language models in which a smaller or cheaper drafter proposes several future tokens and the large target model verifies all of them in a single forward pass, accepting the tokens it agrees with and resampling from a corrected distribution at the first disagreement. The procedure is a draft-then-verify paradigm: draft multiple future tokens, then verify them in parallel with the target LLM to speed up inference.<sup>[1](https://aclanthology.org/2024.findings-acl.456.pdf)</sup> For the original speculative-decoding procedure, a defining property is losslessness: the output distribution is guaranteed identical to the target model's for any choice of draft model, without retraining or architecture changes.<sup>[2](https://arxiv.org/html/2211.17192v2)</sup>

| Key fact | Detail |
|---|---|
| Core loop | A drafter proposes γ tokens; the target model scores all of them in parallel, accepting consistent ones and sampling a correction token on the first rejection.<sup>[2](https://arxiv.org/html/2211.17192v2)</sup> |
| Losslessness | A modified rejection-sampling rule (accept with min(1, p/q), resample from the residual) provably reproduces the target distribution.<sup>[2](https://arxiv.org/html/2211.17192v2)</sup><sup> • </sup><sup>[3](https://proceedings.neurips.cc/paper_files/paper/2024/file/e7349e785900b93d8b4971a3f2c1cefe-Paper-Conference.pdf)</sup> |
| Typical speedups | 2–2.5× for draft models on Chinchilla 70B; Medusa 2.2–2.3×; EAGLE 2.9–3.1×; EAGLE-3 up to 6.5× on HumanEval.<sup>[4](https://arxiv.org/abs/2302.01318)</sup><sup> • </sup><sup>[1](https://aclanthology.org/2024.findings-acl.456.pdf)</sup><sup> • </sup><sup>[5](https://neurips2026-speculative-decoding.vercel.app/)</sup> |
| Acceptance rate | Per-token acceptance α = 1 − ½‖p − q‖₁; expected output tokens emitted in a round of k drafts is (1 − α^(k+1))/(1 − α).<sup>[6](https://ai-math-theory.vercel.app/session/ch10-inference-decoding/10.4-speculative-decoding)</sup> |
| Batch-size decay | EAGLE speedup falls from up to 1.96× at batch size 1 to 1.21× at batch 128; sources disagree on the threshold (about 32 vs about 64) where speculation stops helping.<sup>[7](https://specdecode-bench.github.io/)</sup><sup> • </sup><sup>[8](https://pytorch.org/blog/hitchhikers-guide-speculative-decoding/?hss_channel=tw-776585502606721024)</sup><sup> • </sup><sup>[6](https://ai-math-theory.vercel.app/session/ch10-inference-decoding/10.4-speculative-decoding)</sup> |
| Cost profile | Target-model verification dominates execution time, from about 42% to about 95% of it; rejection sampling itself is under 1.7%.<sup>[7](https://specdecode-bench.github.io/)</sup> |
| Serving support | vLLM ships EAGLE, MTP, draft models, PARD, MLP, n-gram, and suffix decoding, with method gain ratings by query rate.<sup>[9](https://docs.vllm.ai/en/stable/features/speculative_decoding/)</sup> |

## How it works: draft, verify, accept

The loop has three stages. First, an efficient approximation model M_q generates γ draft completions, typically 3 to 8 tokens.<sup>[2](https://arxiv.org/html/2211.17192v2)</sup><sup> • </sup><sup>[5](https://neurips2026-speculative-decoding.vercel.app/)</sup> Second, the target model M_p evaluates all the guesses and their probabilities in parallel, in one forward pass that costs about the same as decoding a single token.<sup>[2](https://arxiv.org/html/2211.17192v2)</sup><sup> • </sup><sup>[5](https://neurips2026-speculative-decoding.vercel.app/)</sup> Third, drafted tokens are accepted left to right under a modified rejection-sampling rule, and generation resumes from the first rejected position, with the target supplying a correction token drawn from an adjusted distribution.<sup>[2](https://arxiv.org/html/2211.17192v2)</sup><sup> • </sup><sup>[10](https://research.ibm.com/blog/speculative-decoding)</sup>

The acceptance rule is the heart of the method. A draft token sampled from q(x) is kept if q(x) ≤ p(x), where p is the target distribution; when q(x) > p(x) it is rejected with probability 1 − p(x)/q(x), and a replacement is drawn from the residual distribution p′(x) = norm(max(0, p(x) − q(x))). This provably yields samples from p(x).<sup>[2](https://arxiv.org/html/2211.17192v2)</sup> Equivalently, a drafted token x is accepted with probability min(1, p(x)/q(x)).<sup>[11](https://prakashkagitha.github.io/llm-stack-book/07-inference-serving/06-speculative-decoding.html)</sup> At temperature 0 the test reduces to comparing argmaxes, so greedy decoding is lossless by construction: only tokens matching the target's own argmax are ever accepted.<sup>[11](https://prakashkagitha.github.io/llm-stack-book/07-inference-serving/06-speculative-decoding.html)</sup>

**Why verification is cheap.** Decoding in a transformer is memory-bound: generating one token requires streaming the full weight set from high-bandwidth memory, so a forward pass over γ+1 positions costs little more than one over a single position. Each parallel run of the target model therefore produces at least one new token, so the number of serial target-model runs never exceeds standard autoregressive decoding, while up to γ+1 tokens can be produced per run.<sup>[2](https://arxiv.org/html/2211.17192v2)</sup> In concrete terms, if 3 tokens of lookahead are correct, three round trips to high-bandwidth memory are saved, offsetting the increased arithmetic.<sup>[8](https://pytorch.org/blog/hitchhikers-guide-speculative-decoding/?hss_channel=tw-776585502606721024)</sup>

## Drafting strategies

Methods divide by where the drafts come from.<sup>[1](https://aclanthology.org/2024.findings-acl.456.pdf)</sup>

**Independent draft models.** A separate, smaller model from the same family proposes tokens. One reference puts the sweet spot at a drafter 10–50× smaller than the target, for example a Llama-1B or 7B draft for a Llama-70B target;<sup>[11](https://prakashkagitha.github.io/llm-stack-book/07-inference-serving/06-speculative-decoding.html)</sup> the original paper found drafters around two orders of magnitude smaller usually performed best, balancing acceptance rate against drafting cost.<sup>[2](https://arxiv.org/html/2211.17192v2)</sup> A 2025 study of over 350 experiments with LLaMA-65B and OPT-66B confirmed that speedup depends heavily on the choice of draft model.<sup>[12](https://aclanthology.org/2025.naacl-long.328.pdf)</sup>

**Self-speculation.** The target model drafts for itself. Medusa adds trained speculation heads to the existing model; the PyTorch implementation made the heads hierarchical, each stage predicting one token and feeding the next, with 3–4 heads working well for language models and 6–8 for code models.<sup>[8](https://pytorch.org/blog/hitchhikers-guide-speculative-decoding/?hss_channel=tw-776585502606721024)</sup> Attaching and training such heads was found more effective than a separate smaller model in both model quality and latency gains.<sup>[8](https://pytorch.org/blog/hitchhikers-guide-speculative-decoding/?hss_channel=tw-776585502606721024)</sup> Layer-skipping self-speculation reaches 1.4–1.7× on LLaMA-2 13B–70B.<sup>[1](https://aclanthology.org/2024.findings-acl.456.pdf)</sup> EAGLE uses an autoregression head with knowledge distillation at the feature level, reaching 2.9–3.1× on Vicuna 7B–33B.<sup>[1](https://aclanthology.org/2024.findings-acl.456.pdf)</sup> NVIDIA's Model Optimizer documents a self-speculative variant in which the draft model is the target model itself, with candidates combined through tree-based attention and a typical-acceptance scheme picking the longest plausible prefix.<sup>[13](https://nvidia.github.io/Model-Optimizer/guides/5_speculative_decoding.html)</sup>

**Model-free drafting.** Suffix and n-gram methods reuse patterns in the token stream, including the prompt, and add no model at all.<sup>[14](https://developer.nvidia.com/blog/co-designing-ai-models-using-speculative-decoding-for-faster-llm-inference/)</sup> Lookahead decoding (Fu et al., 2024) needs no draft model and no trained heads: it uses Jacobi iteration to generate and verify n-grams simultaneously and is lossless under greedy decoding.<sup>[11](https://prakashkagitha.github.io/llm-stack-book/07-inference-serving/06-speculative-decoding.html)</sup>

**Tree-based verification.** Instead of a single chain, tree drafting proposes many candidate continuations arranged as a token tree, for example the top-2 choices at depth 1 and for each of those the top-2 at depth 2, and verifies the entire tree in a single target forward pass, accepting the longest surviving path. This raises the expected accepted length per pass and is essential to Medusa and EAGLE.<sup>[11](https://prakashkagitha.github.io/llm-stack-book/07-inference-serving/06-speculative-decoding.html)</sup> Measured on Qwen3-8B at batch size 1, widening from a chain of k=3 to a tree of k=21 lifted EAGLE speedup from 1.65× to 1.85×; at batch 64, however, the k=21 tree fell below 1× on all workloads.<sup>[7](https://specdecode-bench.github.io/)</sup>

## By the numbers

Reported speedups cluster by method family: Blockwise Decoding 1.7–3.0× on Transformer-big (213M), Medusa 2.2–2.3× on Vicuna 7B–13B, Self-Speculative 1.4–1.7× on LLaMA-2 13B–70B, Lookahead 1.5–2.3× on LLaMA-2 7B–70B, and EAGLE 2.9–3.1× on Vicuna 7B–33B.<sup>[1](https://aclanthology.org/2024.findings-acl.456.pdf)</sup> SpecTr, which frames the problem via optimal transport, achieved a 2.13× wall-clock speedup on large language models, a further 1.37× over prior methods.<sup>[15](https://proceedings.neurips.cc/paper_files/paper/2023/file/6034a661584af6c28fd97a6f23e56c0a-Paper-Conference.pdf)</sup> The original speculative-sampling result with [Chinchilla](https://www.edgepedia.org/chinchilla) (70B parameters) was 2–2.5× in a distributed setup.<sup>[4](https://arxiv.org/abs/2302.01318)</sup>

**Acceptance and draft length.** The per-token acceptance probability is α = E_{Ỹ∼q}[min(1, p(Ỹ)/q(Ỹ))], with the identity α = 1 − ½‖p − q‖₁, so acceptance rises as draft and target distributions agree.<sup>[6](https://ai-math-theory.vercel.app/session/ch10-inference-decoding/10.4-speculative-decoding)</sup> In a round of k drafts the expected number of output tokens emitted is the truncated geometric sum (1 − α^(k+1))/(1 − α), and expected speedup is this divided by (1 + k·c_draft/c_target); with k = 5 and a draft costing one ninth of the target, the ceiling is about 3.86×.<sup>[6](https://ai-math-theory.vercel.app/session/ch10-inference-decoding/10.4-speculative-decoding)</sup> Speedup approaches 1/(1−α) as drafting cost vanishes, but a larger γ only pays off once α is high enough; at low acceptance, longer draft chains waste passes on rejected tokens.<sup>[11](https://prakashkagitha.github.io/llm-stack-book/07-inference-serving/06-speculative-decoding.html)</sup> Empirically, average speedup plateaus or degrades with larger K; on XSum with nucleus sampling, latency was minimized at K = 3.<sup>[4](https://arxiv.org/abs/2302.01318)</sup> Two rules of thumb from the cost model: if the draft-to-target cost ratio exceeds roughly 0.3, overhead exceeds savings even at high acceptance, and α below about 0.4 typically means a net loss.<sup>[6](https://ai-math-theory.vercel.app/session/ch10-inference-decoding/10.4-speculative-decoding)</sup>

**Where the time goes.** Verification by the target model dominates execution time, from about 42% (draft-model drafting on Qwen3-8B at batch 1, where drafting itself takes about 47%) to about 95% (n-gram on Llama-3-70B at batch 512, where drafting is essentially free); rejection sampling is under 1.7% in all measured settings.<sup>[7](https://specdecode-bench.github.io/)</sup>

## When it helps and when it hurts

Speculative methods convert spare compute into fewer sequential steps, so they help when decode is memory-bound, at small-to-moderate batch sizes, and hurt once the GPU is compute-bound; production engines therefore gate speculation on batch size.<sup>[11](https://prakashkagitha.github.io/llm-stack-book/07-inference-serving/06-speculative-decoding.html)</sup> The original paper states the same condition: latency improves through increased concurrency at the cost of more arithmetic operations, so the method is not helpful where additional compute resources are unavailable.<sup>[2](https://arxiv.org/html/2211.17192v2)</sup>

Measured decay is gradual rather than a cliff: EAGLE reaches up to 1.96× on Llama-3-70B at batch size 1, falling to 1.21× for Llama-3.1-8B on GSM8K at batch 128.<sup>[7](https://specdecode-bench.github.io/)</sup> Where the cliff sits is contested. One account places the onset of throughput reduction beyond a batch size of 64, which it calls rare in practice;<sup>[8](https://pytorch.org/blog/hitchhikers-guide-speculative-decoding/?hss_channel=tw-776585502606721024)</sup> another holds that at batch sizes well above 32 decode is compute-bound and speculation makes throughput worse, casting the method as primarily a latency optimization for small-batch or single-stream serving, with continuous batching the better tool for throughput.<sup>[6](https://ai-math-theory.vercel.app/session/ch10-inference-decoding/10.4-speculative-decoding)</sup> At high batch sizes with short-to-medium contexts, verification overhead can outpace gains and slow serving.<sup>[16](https://redis.io/en/blog/speculative-decoding-llm/)</sup>

**The long-context exception.** With sizable batches and lengthy sequences, the [KV cache](https://www.edgepedia.org/kv-cache) emerges as the limiting factor for memory, a situation that can bring back the circumstances under which speculative decoding becomes beneficial.<sup>[16](https://redis.io/en/blog/speculative-decoding-llm/)</sup>

**Failure modes.** Larger draft length increases the variance of sequence generation time, which matters for P90 and P99 latency targets.<sup>[4](https://arxiv.org/abs/2302.01318)</sup> Memory overheads differ sharply by drafting family: n-gram adds zero GPU memory, EAGLE and EAGLE-3 add under 10% to static weights and per-token KV cache, and draft-model methods cost the most, with pairing Qwen3-0.6B to Qwen3-8B increasing per-token memory by 1.77×.<sup>[7](https://specdecode-bench.github.io/)</sup> Rejected drafts also force KV-cache rollback and re-quantization when the cache is compressed, slowing decoding unless recent tokens stay in full precision.<sup>[17](https://www.stat.berkeley.edu/~mmahoney/pubs/9485_QuantSpec_Self_Speculativ.pdf)</sup> More broadly, realized speedup often falls short of theoretical potential.<sup>[18](https://dl.acm.org/doi/10.1145/3846171)</sup>

## Deployment and serving systems

vLLM supports model-based methods (EAGLE, MTP, draft models, PARD, MLP), which give the best latency reduction, and simpler n-gram and suffix decoding, which give modest speedups without increasing workload during peak traffic.<sup>[9](https://docs.vllm.ai/en/stable/features/speculative_decoding/)</sup> Its method table rates EAGLE as high gain at low query rates and medium-to-high gain at high query rates, draft models as high gain at low query rates and medium at high, and n-gram/suffix as low-to-medium at low rates and medium at high.<sup>[9](https://docs.vllm.ai/en/stable/features/speculative_decoding/)</sup> It also exposes acceptance-length telemetry (num_drafts, num_draft_tokens, num_accepted_tokens) for tuning.<sup>[9](https://docs.vllm.ai/en/stable/features/speculative_decoding/)</sup>

KV-cache management is central. After each step, caches must be rolled back to the accepted length, evicting rejected positions; O(1) cache truncation is described as the single largest source of bugs in speculative-decode implementations.<sup>[11](https://prakashkagitha.github.io/llm-stack-book/07-inference-serving/06-speculative-decoding.html)</sup> Paged attention divides KV sequences into smaller memory blocks summoned as needed, letting the speculator generate multiple candidates per predicted word without duplicating the full cache for each.<sup>[10](https://research.ibm.com/blog/speculative-decoding)</sup> PyTorch's implementation modifies the vLLM paged-attention kernel for efficient cache maintenance across speculative heads and adjusts attention masks to verify the (N+1)th token while preserving the original model's output.<sup>[8](https://pytorch.org/blog/hitchhikers-guide-speculative-decoding/?hss_channel=tw-776585502606721024)</sup> Ragged acceptance across batched requests interacts with continuous batching, requiring padding to the maximum accepted length or variable-length kernels.<sup>[11](https://prakashkagitha.github.io/llm-stack-book/07-inference-serving/06-speculative-decoding.html)</sup> Dynamic batching can raise throughput four- to five-fold, but that benefit shrinks when speculation competes for GPU memory.<sup>[10](https://research.ibm.com/blog/speculative-decoding)</sup>

## Recent developments

The field has moved from separate draft models toward feature-level self-speculation with dynamic draft trees. EAGLE-3 reaches up to 6.5× speedup over vanilla decoding on [HumanEval](https://www.edgepedia.org/humaneval), about 1.4× over EAGLE-2, with native support in SGLang and vLLM.<sup>[5](https://neurips2026-speculative-decoding.vercel.app/)</sup> EAGLE 3.1 has landed in vLLM as a config-driven extension of EAGLE 3, merged into main and available in the v0.22.0 release with backward compatibility with existing EAGLE 3 checkpoints; benchmarking the [Kimi K2](https://www.edgepedia.org/kimi-k2).6 EAGLE 3.1 draft on Kimi-K2.6-NVFP4 with vLLM (TP=4, GB200) showed 2.03× higher per-user output throughput at concurrency 1, 1.71× at C=4, and 1.66× at C=16 on the SPEED-Bench coding dataset.<sup>[19](https://vllm-project.github.io/2026/05/26/eagle-3-1.html)</sup> Hardware support has broadened: EAGLE3 speculative decoding is enabled on AMD Instinct GPUs with training and serving support via vLLM and AMD Quark.<sup>[20](https://vllm.ai/blog/2026-07-13-eagle-3-amd-instinct)</sup>

**The lossless-vs-lossy debate.** SGLang's acceptance thresholds default to 1.0 for strict lossless verification; lowering them trades quality for speed and ends losslessness.<sup>[5](https://neurips2026-speculative-decoding.vercel.app/)</sup> vLLM distinguishes theoretical losslessness (up to hardware numerics), algorithmic losslessness (validated by convergence tests), and output stability, which is not promised, since a change in batch size can shift logprobs and with them the output distribution.<sup>[5](https://neurips2026-speculative-decoding.vercel.app/)</sup> Its documentation likewise notes that floating-point precision differences across hardware and batch-size-dependent numerical instability in batched operations can produce slight output discrepancies despite the lossless design.<sup>[9](https://docs.vllm.ai/en/stable/features/speculative_decoding/)</sup>

## Theory and guarantees

The distributional guarantee has been formalized: for any output sequence x1:T, the joint distribution over outputs under speculative decoding equals the target model's distribution, for any output length T.<sup>[3](https://proceedings.neurips.cc/paper_files/paper/2024/file/e7349e785900b93d8b4971a3f2c1cefe-Paper-Conference.pdf)</sup> The same work gives the first characterization of the expected rejection rate over a sequence of length T for a draft distribution and target distribution.<sup>[3](https://proceedings.neurips.cc/paper_files/paper/2024/file/e7349e785900b93d8b4971a3f2c1cefe-Paper-Conference.pdf)</sup> It also proves ([Proposition](https://www.edgepedia.org/proposition) 1) that some rejection is unavoidable no matter how many draft responses are sampled from the draft model, which implies that very large draft batches do not necessarily yield significant speedups over small ones.<sup>[3](https://proceedings.neurips.cc/paper_files/paper/2024/file/e7349e785900b93d8b4971a3f2c1cefe-Paper-Conference.pdf)</sup> The acceptance-rate model sets a theoretical ceiling: with α near 1 and cheap drafting, speedup tends toward 1/(1−α).<sup>[6](https://ai-math-theory.vercel.app/session/ch10-inference-decoding/10.4-speculative-decoding)</sup><sup> • </sup><sup>[11](https://prakashkagitha.github.io/llm-stack-book/07-inference-serving/06-speculative-decoding.html)</sup>

## References

1. [Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding (ACL Findings 2024)](https://aclanthology.org/2024.findings-acl.456.pdf)
2. [Fast Inference from Transformers via Speculative Decoding (Leviathan et al., arXiv)](https://arxiv.org/html/2211.17192v2)
3. [A Theoretical Perspective for Speculative Decoding Algorithm (NeurIPS 2024)](https://proceedings.neurips.cc/paper_files/paper/2024/file/e7349e785900b93d8b4971a3f2c1cefe-Paper-Conference.pdf)
4. [Accelerating Large Language Model Decoding with Speculative Sampling (Chen et al., 2023)](https://arxiv.org/abs/2302.01318)
5. [Speculative Decoding: How It Evolved, When It Stays Lossless, and What's Next (workshop overview)](https://neurips2026-speculative-decoding.vercel.app/)
6. [AI / ML Theory, Ch. 10.4 — Speculative decoding](https://ai-math-theory.vercel.app/session/ch10-inference-decoding/10.4-speculative-decoding)
7. [SpecDecode-Bench: Speculative Decoding — Performance or Illusion?](https://specdecode-bench.github.io/)
8. [A Hitchhiker's Guide to Speculative Decoding (PyTorch blog)](https://pytorch.org/blog/hitchhikers-guide-speculative-decoding/?hss_channel=tw-776585502606721024)
9. [Speculative Decoding — vLLM documentation](https://docs.vllm.ai/en/stable/features/speculative_decoding/)
10. [Speculative decoding: cost-effective AI inferencing (IBM Research)](https://research.ibm.com/blog/speculative-decoding)
11. [Speculative Decoding: Draft Models, Medusa, EAGLE & Lookahead — The LLM Stack](https://prakashkagitha.github.io/llm-stack-book/07-inference-serving/06-speculative-decoding.html)
12. [Decoding Speculative Decoding (NAACL 2025)](https://aclanthology.org/2025.naacl-long.328.pdf)
13. [Speculative Decoding — NVIDIA Model Optimizer documentation](https://nvidia.github.io/Model-Optimizer/guides/5_speculative_decoding.html)
14. [Co-Designing AI Models Using Speculative Decoding (NVIDIA developer blog)](https://developer.nvidia.com/blog/co-designing-ai-models-using-speculative-decoding-for-faster-llm-inference/)
15. [SpecTr: Fast Speculative Decoding via Optimal Transport (NeurIPS 2023)](https://proceedings.neurips.cc/paper_files/paper/2023/file/6034a661584af6c28fd97a6f23e56c0a-Paper-Conference.pdf)
16. [Speculative decoding: How it works, when it helps & where it fits (Redis)](https://redis.io/en/blog/speculative-decoding-llm/)
17. [QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cache (UC Berkeley)](https://www.stat.berkeley.edu/~mmahoney/pubs/9485_QuantSpec_Self_Speculativ.pdf)
18. [Towards Optimal Speculative Decoding: A Comprehensive Survey (ACM Computing Surveys)](https://dl.acm.org/doi/10.1145/3846171)
19. [EAGLE 3.1 in vLLM](https://vllm-project.github.io/2026/05/26/eagle-3-1.html)
20. [EAGLE3 Speculative Decoding on AMD Instinct GPUs](https://vllm.ai/blog/2026-07-13-eagle-3-amd-instinct)

---
*Topic: Encyclopedia › Technology and the built world › Computing and digital systems › Modern AI: foundation models, generative AI, and the AI industry › Foundation-model methods and training › Inference, serving, and efficiency of foundation models*

*Initially written Sep 17, 2026 · Reviewed: Sep 30, 2026; Oct 15, 2026 · Edited: Sep 30, 2026; Oct 15, 2026 · Last review: Oct 15, 2026*

*Copyright 2026 EdgeChat AI, a subsidiary of Biostate AI.*

License: Edgepedia Community License 1.0, https://www.edgepedia.org/edgepedia/license