Explainer

Speculative decoding: when it pays and when it does not

When speculative decoding speeds up LLM inference: acceptance rate, draft length, batch size, long context, EAGLE-3 in vLLM and SGLang, and hybrid Mamba models.

By Debarshi Das. Published 18 October 2026.

17 min read

LLM inference
Speculative decoding
GPU infrastructure
KV cache

Speculative decoding makes a large language model produce several tokens per forward pass instead of one. A cheap drafter guesses the next few tokens, and the target model checks all of the guesses in one pass. The output does not change. What changes is the work: the GPU computes tokens that it may throw away, in exchange for fewer sequential steps.

The usual summary is that speculative decoding pays at low batch and loses as batch grows. The published measurements mostly agree, with two corrections: better drafters keep a gain at batch 64, and at long context the gain can grow with batch size. On hybrid models, each draft token also costs a copy of the recurrent state. This essay gives the mechanism, the math, the published measurements, and when to turn it on or off.

What does speculative decoding trade?

A normal decode step reads every weight of the model to produce one token per request. Speculative decoding splits the step in two:

Draft. A drafter proposes $k$ tokens. The drafter can be a small model, extra heads on the target model (Medusa), a small head that reads the target's hidden features (EAGLE), the model's own multi-token prediction (MTP) layers, or a text match against the prompt (n-gram or prompt lookup).
Verify. The target model runs once over all $k$ draft tokens. It keeps the longest prefix of drafts that it agrees with, and adds one token of its own at the first disagreement, or after the last draft if all of them pass.

A one-layer draft guessed 3 tokens; the 32-layer target kept 2, rejected 1 and added its own: 3 tokens in one pass. — EAGLE-3 on Llama 3.1 8B: a one-layer draft guesses three tokens, and one pass of the 32-layer target checks all three. Two are kept, the third is replaced by the target's own token: three tokens for one pass.

The verification rule, rejection sampling, keeps the output distribution exactly the same as plain decoding. Leviathan et al. prove this and report "identical outputs" on T5-XXL. Chen et al. report benchmark scores close to plain sampling on Chinchilla 70B. So speculative decoding is a performance setting, not a quality setting. The only question is whether it is faster.

How much faster can it be?

Two numbers describe the drafter. The acceptance rate $a$ is the chance that the target accepts a draft token, given that it accepted the ones before it. The draft length $k$ is the number of tokens proposed per step. Leviathan et al. derive the expected number of tokens produced per target pass:

E = \frac{1 - a^{k+1}}{1 - a}

E = \frac{1 - a^{k+1}}{1 - a}

$E$ is what engines report as the mean acceptance length. It is never more than $k + 1$ , and it stops growing once $a^{k}$ is small, so long drafts help only when the acceptance rate is high.

The drafter is not free. Let $c$ be the time of one drafter step divided by the time of one target step. If the target can check $k + 1$ tokens in the time it takes to produce one, the expected speedup in wall time is:

S = \frac{1 - a^{k+1}}{(1 - a)(k c + 1)}

S = \frac{1 - a^{k+1}}{(1 - a)(k c + 1)}

A worked example with published inputs: Chen et al. drafted for Chinchilla 70B with a 4B model at 1.8 ms per token, against 14.1 ms per token for the target, so $c \approx 0.128$ . With $k = 4$ :

at $a = 0.8$ , $E = 3.36$ tokens per pass and $S \approx 2.2$ ;
at $a = 0.6$ , $E = 2.31$ and $S \approx 1.5$ ;
the gain disappears near $a = 0.34$ , where $E$ falls to $k c + 1 = 1.51$ .

These are outputs of the formula, not measurements. Chen et al. measured 1.92 to 2.46 times at $k = 4$ and batch size 1, which sits inside that range.

The formula carries one assumption that decides everything else: that checking $k + 1$ tokens costs the same time as producing one. Leviathan et al. state it directly. Their method "is not helpful for configurations where additional computation resources are not available."

Why does batch size take the gain away?

At batch size 1, a decode step is limited by memory bandwidth. Llama 3.1 8B has 8,030,261,248 parameters, 16.1 GB in BF16. An H100 SXM reads 3.35 TB/s, so one step needs at least 4.8 ms just to read the weights: at most about 208 tokens per second. SGLang measured 158 tokens per second for this model on one H100 at batch 1. The tensor cores spend most of that time waiting.

A matrix multiply over $n$ tokens does about 2 floating-point operations per parameter per token, and reads each 2-byte BF16 weight once. So it does about $n$ operations per byte read. The H100 SXM does 989 TFLOPS of dense BF16 work and reads 3.35 TB/s, so it can do about 295 operations per byte before compute, not memory, sets the pace. In theory, the weight matrix multiplies stay memory-bound until a step carries about 295 tokens.

Every token in the batch counts toward that number, drafts included. With $k = 3$ , each request brings 4 tokens to the verification pass: the last token it produced, which the model has not read yet, and the 3 drafts. vLLM's scheduler plans for exactly these $k + 1$ tokens per request, although its documentation rounds the verification batch to batch size times $k$ . So 74 requests already fill the 295. Past that point, verifying $k + 1$ tokens costs more time than producing one, and the cost grows with every draft.

128 requests with 3 drafts each fill one H100 decode step to 512 tokens; those past the 295-token line are ember. — With three drafts, each request brings four tokens to every step. On an H100, about 295 tokens per step is where reading the weights stops being the limit; 128 requests bring 512.

The 295 is a scale, not a threshold. It ignores attention, which reads each request's own KV cache and stays memory-bound at any batch size, and it assumes kernels reach peak compute, which they do not at small sizes.

To see the effect in the formula, add a factor $v$ : the time of the verification pass divided by the time of a plain decode step.

S = \frac{1 - a^{k+1}}{(1 - a)(k c + v)}

S = \frac{1 - a^{k+1}}{(1 - a)(k c + v)}

When the step is memory-bound, $v \approx 1$ and this is the formula above. When it is fully compute-bound, $v$ approaches $k + 1$ , because the pass now does $k + 1$ times the work. Then $S$ is below 1 for any acceptance rate under 1, because $E$ is never more than $k + 1$ . In the worked example, $a = 0.8$ gives 2.2 times at $v = 1$ and 0.6 times at $v = 5$ . This extension is mine; real steps sit between the two ends, and the only way to find $v$ is to measure it.

What the literature measured at larger batch

The EAGLE-3 paper reports throughput by batch size for Llama 3.1 8B Instruct on MT-Bench, with a chain of drafts and no tree. The SGLang team ran it on one H100 with SGLang v0.4.4 and a chain length of 3; the baseline is SGLang without speculation.

Batch size	2	8	16	24	32	48	64
EAGLE	1.40x	1.23x	1.02x	0.93x	0.94x	0.88x	0.99x
EAGLE-3	1.81x	1.62x	1.48x	1.39x	1.32x	1.38x	1.38x

The same paper reports a vLLM run with a chain length of 2. There EAGLE falls from 1.30 times at batch 2 to 0.71 times at batch 56, and EAGLE-3 falls from 1.75 times to 1.01 times. The paper names the GPU for that run as an RTX 3090 in the text and an A100 in the table caption.

The gain falls with batch size in both engines, as the thesis says. A better drafter moves the break-even point: EAGLE's gain was gone by batch 24 in SGLang. EAGLE-3, whose drafts are accepted more often, still gave 1.38 times at batch 64, where a step carries 256 tokens, below the H100's 295. The engine and hardware matter too: the same methods lost their gain earlier in the vLLM run.

The vLLM team saw the same shape on a larger model. On Llama 3 70B on four H100 GPUs, a 0.5B draft model gave up to 1.5 times on ShareGPT at 1 query per second (QPS), and n-gram lookup gave up to 2.8 times on CNN/DailyMail. At high QPS, the same settings were 1.4 and 1.8 times slower. The post does not say which rate counts as high.

vLLM now builds the conclusion into its configuration. Its dynamic speculative decoding takes a table of draft lengths by batch size. The documentation explains why: "As BS increases, the effective BS becomes BS*K which increases the compute requirement during verification." Its EAGLE-3 example for Llama 3.1 8B uses 5 drafts at batch 1 to 16, 4 at 17 to 32, 3 at 33 to 64, 1 at 65 to 128, and none above 128. That is an example, not a tuned result. TurboSpec, built on vLLM, sets the amount of speculation at run time from measured goodput, because speculation "may degrade LLM serving performance if added naively without tuning."

Does long context change the answer?

Yes. At long context, a decode step spends most of its time reading the KV cache, not the weights. Each request reads its own cache once per step whether it verifies one token or four, so the extra tokens ride along on memory traffic the step pays anyway. The size of that cache per token is set by the model, as the KV cache essay shows: 128 KiB per token for Llama 3.1 8B in BF16, so a 32,768-token request reads 4 GiB of cache in every step, about a quarter of the model's weights. At batch 32, the caches are 128 GiB and the weights 15 GiB. That is more than one 80 GB GPU holds, so a setup like this spreads the work over several GPUs, as MagicDec does below. The share stays the same: 90 percent of every step's reads are cache, and checking 4 tokens instead of 1 does not read any more of them.

Llama 3.1 8B at batch 32 and 32,768 tokens: 32 KV caches, 128 GiB, dwarf 15 GiB of weights; cache is 90% of reads. — Llama 3.1 8B at batch 32: at 1,024 tokens each, the caches are 4 GiB of a step's reads; at 32,768, they are 128 GiB, 90 percent. Checking 4 tokens instead of 1 reads the same bytes.

MagicDec measured this. It drafts with the target model itself, using a small fixed window of the KV cache (StreamingLLM, 512 tokens), so the drafter's cost does not grow with context. For Llama 3.1 8B on 8 A100 GPUs, with a draft length of 2:

Prompt length	Batch 32	Batch 64	Batch 128
4,000 tokens	0.90x	0.81x	0.76x
32,000 tokens	1.22x	1.28x	1.35x

At 4,000 tokens, speculation lost, and lost more as batch grew. At 32,000 tokens it won, and won more as batch grew. On 8 H100 GPUs at 32,000 tokens, the same method with its best draft length gave 1.08 times at batch 16 and 1.85 times at batch 128. The authors call the length where the trend reverses the critical sequence length. In the A100 table, Llama 3.1 8B breaks even near 8,000 tokens. The authors report that this length is lower on H100, which has more compute per byte of bandwidth, and higher for models with grouped-query attention, which have a smaller cache per token.

This contradicts the simple form of the thesis. Batch size alone does not decide; batch size and context length together do. The thesis holds for short and medium requests, which is most chat traffic. For long-context serving, such as document analysis or agents with long histories, speculation can raise throughput at large batch, given a drafter whose cost stays small at long context.

What does a draft token cost on a hybrid model?

Hybrid models replace most attention layers with layers that keep a fixed-size recurrent state: Mamba layers in Nemotron-H, gated delta networks in Qwen3-Next and Qwen3.5. That state is what makes them cheap at long context. It also makes rejection hard.

An attention layer rolls back a rejected draft by dropping the last cache entries. A recurrent layer has no entries to drop: each token overwrites the state. To return to the state after the last accepted token, the engine must keep the state as it was after each draft token. In vLLM 0.31.0, a Mamba layer's cache specification sets num_speculative_blocks to num_speculative_tokens. Its memory per request is one state page plus one per draft token; with prefix caching on, at least two pages plus one per draft token. The only exception is Kimi K3's KDA layers, through an option named RecoverSSM.

The cost is easy to compute from the numbers in the KV cache essay. Nemotron-H-8B keeps 97.4 MiB of state per request in vLLM's default FP32, and its 4 attention layers add 16,384 bytes of cache per token. An 8,192-token request holds 225.4 MiB. With $k = 3$ , it holds 4 copies of the state: 389.6 MiB of state, 517.6 MiB in all, 2.3 times as much. The 3 extra copies equal 18,702 tokens of the model's own cache. On a pure attention model such as Qwen3-8B, the same 3 drafts add 3 tokens of cache, 432 KiB.

A Nemotron-H-8B request with a 128 MiB cache and a 97.4 MiB state; 3 draft tokens add 3 state copies: 517.6 MiB. — Nemotron-H-8B at 8,192 tokens, FP32 state: three draft tokens add three copies of the recurrent state, from 225.4 MiB to 517.6 MiB per request.

Memory is half the cost. The other half is traffic: each copy is written in every step. A 2026 post on Tri Dao's blog, introducing ReplaySSM, calls the per-draft-token state "the common workaround, used for example in vLLM", and says it "adds $T$ times more memory traffic per decoding step" for a window of $T$ drafts. It measured the baseline verification kernel at 4.85 times the plain decode kernel at $T = 6$ , on Qwen3.5-122B-A10B in NVFP4 on one B300. It reports that at a window of 4 the preallocated states cut the maximum decode batch by roughly 4 times. Its alternative caches the small inputs of recent steps and rebuilds the state only when needed, for up to 2.14 times the throughput of vLLM's speculative path at large batch.

The Mamba in the Llama (NeurIPS 2024) took another route: one fused kernel that keeps a single snapshot and recomputes the state after a rejection. Its authors report that a naive implementation gave "no speedup at all on H100s", and their kernel 1.71 to 2 times on H100 for pure Mamba models of 2.8B and 7B parameters. STree extends tree verification to hybrid models.

So on hybrids, the thesis holds in a sharper form: speculation takes more memory from the batch, so the batch where it stops paying comes earlier. Check how your engine stores draft states before you enable speculation on a hybrid model.

Which drafter should you use?

EAGLE-3 is the strongest general method in its authors' comparisons, ahead of EAGLE and EAGLE-2. It needs a trained head for the exact target model; heads for common models are public.
MTP uses prediction layers that ship with the model, such as DeepSeek-V3's and Qwen3-Next's. DeepSeek reports that its second-token prediction is accepted 85 to 90 percent of the time, for 1.8 times the tokens per second.
A separate draft model works with any target that shares its tokenizer. Over more than 350 experiments, Yan et al. found that the drafter's latency matters more than its quality as a language model.
N-gram or prompt lookup needs no model. It copies spans of the prompt, so it pays where outputs repeat inputs: summarization, retrieval answers, code editing. Apoorv Saxena reports an average 2.4 times on summarization and context QA with Mistral-7B on one A100. Lookahead decoding, which has the target guess its own n-grams, reports up to 1.8 times on MT-Bench.
Trees verify several candidate branches per step. SGLang's documentation notes that a wider tree "increases memory/compute consumption", so trees spend the spare compute fastest and lose first at high batch.

When to turn it on, and when to turn it off

Turn it on when:

requests are latency-bound and concurrency per GPU is low, such as interactive chat, coding assistants and agents waiting on one call;
the drafter's acceptance length on your traffic is well above $k c + 1$ ;
contexts are long, and the drafter's cost does not grow with them.

Turn it off, or shorten the draft, when:

the GPU runs large batches of short requests for throughput, such as offline batch jobs or the decode pool of a disaggregated deployment sized for tokens per second;
acceptance is low, as on high-temperature sampling or text unlike the drafter's training data;
the model is a hybrid and memory, not compute, limits the batch.

Between those cases, make the draft length depend on load, as vLLM's dynamic speculative decoding does, and set the table from a measurement.

How to find the break-even batch on your hardware

The published data does not give the batch size where the gain disappears for your model, GPU and engine version. Measure it. The method below uses vLLM 0.31.0; SGLang's bench_serving works the same way.

Bash

# 1. Start the baseline server, with no speculation.
vllm serve meta-llama/Llama-3.1-8B-Instruct --port 8000 --no-enable-prefix-caching

# 2. Sweep concurrency. MT-Bench text gives a realistic acceptance rate;
#    random tokens give almost none.
for c in 1 2 4 8 16 32 64 128 256; do
  vllm bench serve --model meta-llama/Llama-3.1-8B-Instruct --port 8000 \
    --dataset-name hf --dataset-path philschmid/mt-bench \
    --num-prompts $((c * 8)) --max-concurrency "$c" --temperature 0 \
    --save-result --result-filename "base-c$c.json"
done

# 3. Restart with EAGLE-3 and three drafts per step, then run the same sweep
#    with --result-filename "eagle3-c$c.json".
vllm serve meta-llama/Llama-3.1-8B-Instruct --port 8000 --no-enable-prefix-caching \
  --speculative-config '{"method": "eagle3", "model": "yuhuili/EAGLE3-LLaMA3.1-Instruct-8B", "num_speculative_tokens": 3}'

For each concurrency, compare output tokens per second and the median time per output token. The break-even batch is the first concurrency where the speculative run is slower. The benchmark also prints the acceptance rate and length, $a$ and $E$ for your traffic. Repeat with your own prompts and context lengths. For a hybrid model, also compare the "Maximum concurrency" line in the server's startup log with and without --speculative-config: the difference is the batch that the state copies take.

Limits

This essay reports public measurements and computes from public configurations and code. I did not run a sweep. Read every number with these limits:

No single setup. The measurements come from different models, GPUs, engines and versions: TPU v4 for the first two papers, A100 and H100 for MagicDec, one H100 with SGLang v0.4.4 for EAGLE-3. Engines have changed since, so the break-even batch in a current release can differ.
Some details are unclear. The EAGLE-3 paper names the GPU for its vLLM run as an RTX 3090 in the text and an A100 in the table; the SGLang team, not the authors, ran the SGLang table. The vLLM post does not give the request rate it calls high.
MagicDec's setup is specific. It uses 8 GPUs with tensor parallelism and drafts with the target model on a compressed cache. A separate small model with a full cache does not keep its cost flat at long context.
The 295 tokens per step is a roofline estimate from the H100's datasheet rates. Real kernels leave the memory-bound region earlier, and attention moves it later.
The hybrid memory numbers are computed, not measured, from Nemotron-H-8B's config and vLLM 0.31.0's source. Engines can pad state pages, so the real number can be higher. SGLang's documentation does not state how it stores draft states.
ReplaySSM is a blog post, not a peer-reviewed paper, and its figures and text disagree on a few details. It is the only measurement of speculation on hybrid models at large batch that I found.

Questions engineers ask

Does speculative decoding change the output?

No. With the standard verification rule, the tokens follow the same distribution as plain decoding. Leviathan et al. prove it and measured it at temperature 0 and at temperature 1.

Does it raise throughput or only lower latency?

At low batch it does both, because the GPU had idle compute. At high batch with short contexts it lowers throughput, because the drafts take compute that other requests needed. At long context it can raise throughput at high batch.

What acceptance rate do I need?

Enough that $E$ exceeds $k c + v$ . With Chen et al.'s 4B drafter at $k = 4$ and a memory-bound step, $E$ must exceed 1.51. A cheaper drafter lowers $c$ and the bar with it. Measure $E$ on your own traffic; the benchmark prints it.

Is MTP the same as speculative decoding?

MTP is a way to train extra prediction layers into the model. Used for serving, those layers are the drafter, and the target verifies their tokens in the same way.

References

Leviathan, Kalman and Matias, Fast Inference from Transformers via Speculative Decoding, ICML 2023.
Chen et al., Accelerating Large Language Model Decoding with Speculative Sampling, DeepMind, 2023.
Cai et al., Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads.
Li et al., EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty.
Li et al., EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees.
Li et al., EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test, tables 3 and 5 for batch size.
Sadhukhan et al., MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding, tables 3 and 6.
Fu et al., Break the Sequential Dependency of LLM Inference Using Lookahead Decoding.
Yan, Agarwal and Venkataraman, Decoding Speculative Decoding, NAACL 2025.
Liu et al., TurboSpec: Closed-loop Speculation Control System for Optimizing LLM Serving Goodput.
Apoorv Saxena, Prompt lookup decoding.
DeepSeek-AI, DeepSeek-V3 Technical Report, section 5.4.3, for MTP acceptance.
Wang et al., The Mamba in the Llama: Distilling and Accelerating Hybrid Models, NeurIPS 2024.
Wu et al., STree: Speculative Tree Decoding for Hybrid State-Space Models.
Tri Dao's blog, ReplaySSM: Cache SSM Inputs, Not State, 2026.
vLLM team, How Speculative Decoding Boosts vLLM Performance by up to 2.8x, October 2024.
vLLM 0.31.0, Speculative decoding and Dynamic speculative decoding.
vLLM 0.31.0 source, MambaSpec memory per request and the Mamba layer's speculative slots.
SGLang, Speculative decoding, including the Llama 3.1 8B throughput on one H100.
NVIDIA, H100 specifications, for the BF16 rate (1,979 TFLOPS with sparsity, half without) and 3.35 TB/s.
Llama 3.1 8B Instruct model card, for the parameter count.
Nemotron-H-8B config, for the state and cache sizes, computed in the KV cache essay.