Large language models generate text one token at a time in an autoregressive loop: each token depends on all previous tokens, creating a sequential bottleneck that limits throughput and increases latency. Speculative decoding breaks this pattern by introducing a faster draft model that predicts multiple tokens ahead, allowing the main model to verify several candidates in parallel rather than generating them sequentially.

The Sequential Bottleneck

Standard autoregressive generation is inherently serial. A 70B parameter model must perform a full forward pass for each token, loading billions of weights from memory. Even with optimized inference engines, memory bandwidth becomes the limiting factor—the model spends most of its time waiting for data rather than computing. For a 1,000-token response, that means 1,000 sequential forward passes with no opportunity for parallelization within the sequence.

The observation behind speculative decoding is that GPUs are underutilized during this process. Modern accelerators can handle massive parallelism, but autoregressive generation forces them to process tokens one at a time. If we could predict several tokens and verify them together, we could make better use of available compute.

Draft and Verify

Speculative decoding introduces a two-stage process. A small, fast draft model—often 10x to 100x smaller than the target model—generates a sequence of candidate tokens. The draft model uses the same vocabulary and produces a probability distribution over tokens, but runs quickly because of its reduced size. It might draft 4-8 tokens in the time the main model would generate one.

The main model then verifies all draft tokens in a single forward pass. Because transformer attention is parallel across the sequence dimension, the large model can process multiple positions simultaneously. For each position, it compares its own probability distribution against the draft model’s prediction. Tokens are accepted if they fall within the main model’s distribution, using a sampling strategy that preserves the target distribution exactly.

When the draft is accurate, multiple tokens are accepted at once, reducing the number of main model forward passes. When the draft diverges, verification catches it immediately and the process continues from the rejection point. The key insight is that verification is essentially free—the main model was going to run a forward pass anyway, and checking multiple positions in parallel adds minimal overhead.

Maintaining Output Quality

Speculative decoding is lossless when implemented correctly. The final output distribution matches what the main model would have produced on its own, token for token. This works through careful acceptance sampling: at each position, a token is accepted with probability proportional to how well it aligns with the main model’s distribution.

If the draft model suggests a token with 30% probability and the main model assigns it 40%, it’s accepted. If the draft assigns 40% but the main model assigns 30%, it’s accepted with 75% probability, and a correction sample is drawn if rejected. This scheme ensures the aggregate distribution over many generations matches the main model exactly.

The draft model doesn’t need to be trained specifically for this task. Any smaller model from the same family works, including earlier checkpoints, distilled versions, or quantized variants. Some implementations use the main model itself with reduced precision or fewer layers. The only requirement is speed—the draft must be fast enough that generating multiple candidates plus verification is cheaper than standard generation.

Practical Speedups

Speedup depends on draft accuracy. If the draft model predicts the main model’s output well, acceptance rates are high and latency drops significantly. Reports show 2x to 3x speedups for well-matched model pairs on typical workloads. Gains are larger for tasks where a small model can approximate the large model’s behavior—code completion, structured output, and domains where the draft model was trained on similar data.

The technique works best when the draft model is genuinely fast relative to the main model. A 7B draft model paired with a 70B main model sees good results. A 30B draft model paired with a 70B model offers less benefit because the draft itself becomes expensive. Memory bandwidth still dominates, so the draft must be small enough to stay in faster cache tiers or require fewer memory round trips.

Batching complicates speculative decoding. Standard inference batches multiple requests together to amortize memory costs, but speculative decoding requires coordinating draft and verification across the batch. Different sequences may accept different numbers of tokens, leading to ragged batches and synchronization overhead. Implementations must balance batch size, draft length, and acceptance rates to maintain throughput.

Engineering Tradeoffs

Speculative decoding increases system complexity. The inference stack must manage two models, coordinate draft generation and verification, and handle rejection and resampling. Memory footprint grows because both models must be loaded, though the draft model’s small size limits the impact. Some deployments use shared KV caches or pipeline the draft and main models to reduce memory overhead.

The technique shifts the bottleneck from latency to compute cost. Generating draft tokens and running verification passes consume additional FLOPs, even when tokens are rejected. In high-throughput scenarios where GPUs are already saturated, speculative decoding may reduce per-request latency but lower overall throughput. The tradeoff favors latency-sensitive applications over batch processing workloads.

Speculative decoding is one of several emerging techniques to accelerate autoregressive generation without sacrificing quality. It reveals that the memory-bound nature of LLM inference leaves room for speculative work—using cheap computation to reduce the number of expensive operations. As models grow larger and latency requirements tighten, techniques that exploit this asymmetry will become essential infrastructure.