The rapid proliferation of Large Language Models (LLMs) into production environments\u2014spanning generative chatbots, code assistants, and automated reasoning agents\u2014has precipitated a fundamental shift in the architectural priorities of inference systems. In the early stages of the “generative AI” era, the primary optimization metric was <\/span>Time-to-First-Token (TTFT)<\/b>, a latency-centric measure reflecting the responsiveness of the system to a single user. However, as deployment scales to millions of concurrent requests, the economic viability of these systems is increasingly dictated by <\/span>Throughput<\/b> (tokens generated per second per dollar) and <\/span>Hardware Utilization<\/b>.<\/span><\/p>\n This transition has brought two powerful optimization paradigms into direct conflict: <\/span>Continuous Batching<\/b> and <\/span>Speculative Decoding<\/b>. Continuous batching, popularized by systems like vLLM and Orca, maximizes the utilization of High-Bandwidth Memory (HBM) by dynamically scheduling requests at the token level, effectively saturating the Graphics Processing Unit (GPU) compute cores. Conversely, Speculative Decoding (SD) was originally conceived as a latency-optimization technique that leverages “idle” GPU cycles\u2014cycles that exist because memory bandwidth limits the rate at which a single sequence can be processed\u2014to verify multiple future tokens in parallel.<\/span><\/p>\n The central thesis of this report, supported by research emerging in late 2024 and 2025, is that <\/span>batching has effectively compressed the available idle computing power<\/b>, rendering traditional speculative decoding strategies inefficient or even deleterious in high-load scenarios.<\/span>1<\/span> As batch sizes increase to maximize throughput, the “idle” gaps that SD relies on disappear, leading to resource contention where drafting and verification compete for the same saturated arithmetic units.<\/span><\/p>\n This analysis provides an exhaustive examination of this contention and the subsequent wave of architectural innovations designed to resolve it. We explore <\/span>SpecFormer\u2019s<\/b> unification of attention mechanisms to eliminate expensive prefix trees <\/span>1<\/span>, <\/span>Falcon\u2019s<\/b> semi-autoregressive drafting for training stability <\/span>4<\/span>, <\/span>Mirror Speculative Decoding\u2019s<\/b> utilization of heterogeneous hardware to physically separate draft and verify stages <\/span>5<\/span>, and the <\/span>EqSpec\/EXSpec<\/b> frameworks that solve the critical “ragged tensor” problem to ensure mathematical correctness in batched speculation.<\/span>6<\/span> By synthesizing these developments, we articulate a new standard for high-performance, cost-effective LLM inference.<\/span><\/p>\n To understand why batching and speculative decoding are currently at odds, one must first rigorously define the physical constraints of the hardware executing these models. The performance of any Deep Learning workload is governed by the <\/span>Roofline Model<\/b>, which plots performance (FLOPs\/second) against <\/span>Arithmetic Intensity<\/b> (FLOPs\/byte of memory transferred).<\/span>7<\/span><\/p>\n Generative LLMs are autoregressive: the generation of token $t$ depends on token $t-1$. This sequential dependency forces the hardware to load the model’s entire weight matrix $\\Theta$ from HBM to the on-chip SRAM for <\/span>every single token<\/span><\/i> generated.<\/span><\/p>\n For a model like Llama-3-70B (approx. 70 billion parameters, FP16 precision), the weights constitute roughly 140 GB of data. Generating one token requires moving this 140 GB across the memory bus. If the GPU is an NVIDIA H100 with a memory bandwidth of roughly 3.35 TB\/s, the theoretical minimum time to load weights is:<\/span><\/p>\n <\/p>\n $$T_{load} = \\frac{140 \\text{ GB}}{3350 \\text{ GB\/s}} \\approx 41.8 \\text{ ms}$$<\/span><\/p>\n However, the computation required is merely a matrix-vector multiplication (the active vector being the single token embedding). The arithmetic intensity is extremely low\u2014approximately 1 FLOP per byte transferred. Since modern GPUs are capable of nearly 1,000 TFLOPs (FP16), the compute units (Tensor Cores) finish the calculation in microseconds, spending the vast majority of the 41.8 ms cycle waiting for memory. This state is defined as being <\/span>Memory-Bound<\/b>.<\/span>8<\/span><\/p>\n Speculative Decoding (SD) exploits this massive inefficiency. Since the Tensor Cores are idle for >90% of the cycle in a single-batch setting, SD proposes using a smaller “draft” model to guess the next $K$ tokens (e.g., $K=4$). The large target model then verifies all $K$ tokens in a single forward pass.<\/span><\/p>\n Verification is a matrix-matrix operation (checking 4 tokens simultaneously). This increases the arithmetic intensity by a factor of $K$ without significantly increasing the memory transfer time (since the weights are loaded once for the batch of $K$ tokens). In the low-batch regime ($B=1$), SD effectively converts memory-bound latency into compute-bound work, utilizing the “idle” resources to accelerate generation.<\/span>1<\/span><\/p>\n Batching is the standard solution to the memory wall. By grouping $B$ user requests together, the system loads the weights once and applies them to $B$ tokens simultaneously. The arithmetic intensity scales linearly with $B$:<\/span><\/p>\n <\/p>\n $$\\text{Intensity}_{Batched} \\approx B \\times \\text{Intensity}_{Single}$$<\/span><\/p>\n As $B$ increases, the workload moves from the memory-bound region of the roofline graph toward the compute-bound region. In production systems serving thousands of users, schedulers drive $B$ as high as memory capacity allows (often $B > 128$). At these levels, the Tensor Cores are fully saturated processing the main batch. The “idle computing power” that SD requires is no longer idle; it has been harvested to serve other users. This is the crux of the problem: <\/span>Batching compresses available idle computing power<\/b>, turning SD from an optimization into a burden.<\/span>1<\/span><\/p>\n The industry standard for handling this high-throughput requirement is <\/span>Continuous Batching<\/b> (often referred to as cellular or iteration-level batching), popularized by the vLLM and Orca systems.<\/span>11<\/span> Understanding its mechanism is crucial to understanding why it is so hostile to traditional speculative decoding.<\/span><\/p>\n In traditional <\/span>Static Batching<\/b>, the server waits for $B$ requests to accumulate. It then processes them in lockstep. If one request finishes early (generating a short answer), it must wait for the longest request in the batch to complete, wasting compute slots on “padding” tokens.<\/span><\/p>\n Continuous Batching<\/b> fundamentally alters this by operating at the granularity of a single iteration.<\/span>11<\/span><\/p>\n This mechanism is extremely effective at maximizing <\/span>GPU Utilization<\/b>. Metrics from industry deployments indicate that continuous batching can improve throughput by factors of 10$\\times$ to 20$\\times$ over static batching.<\/span>13<\/span> Consequently, any “slack” in the system is aggressively filled with new revenue-generating work.<\/span><\/p>\n Continuous batching is physically enabled by <\/span>PagedAttention<\/b>.<\/span>9<\/span> Traditional attention mechanisms require contiguous memory blocks for the Key-Value (KV) cache, leading to fragmentation and limiting the maximum batch size. PagedAttention applies the concept of virtual memory paging to the KV cache, allowing memory to be allocated in non-contiguous blocks. This dramatically reduces memory waste, allowing the scheduler to fit even more concurrent sequences into HBM, further increasing the batch size $B$ and pushing the system deeper into the compute-bound regime.<\/span><\/p>\n While continuous batching optimizes global throughput, recent 2025 research has highlighted a critical flaw: <\/span>Unfairness<\/b>. Stall-free schedulers (like those in vLLM) prioritize the “decode” phase of active requests to minimize Inter-Token Latency (ITL). This creates a “head-of-line blocking” effect for new requests waiting in the “prefill” (prompt processing) phase, leading to high Time-to-First-Token (TTFT) tail latencies.<\/span>14<\/span><\/p>\n FairBatching<\/b> 14<\/span> is a novel scheduler developed to address this. It introduces an adaptive batch capacity mechanism that dynamically adjusts the computational budget. By acknowledging the non-monotonic nature of Time-Between-Tokens (TBT), FairBatching reclaims resources from bursting decode tasks to serve prefill surges. This nuance is critical: even highly optimized schedulers are now dynamically trading off resources between prefill and decode, leaving absolutely zero margin for the overhead of inefficient speculative decoding.<\/span><\/p>\n When Speculative Decoding is superimposed onto a Continuously Batched system, two distinct failures modes emerge: <\/span>Compute Contention<\/b> and <\/span>Memory\/Tensor Misalignment<\/b>.<\/span><\/p>\n In a batch of size $B=128$, the GPU is calculating logits for 128 tokens per step. If we enable Speculative Decoding with a draft length $K=3$, the verification step now requires processing $128 \\times (3+1) = 512$ tokens per step.<\/span><\/p>\n If the GPU was already near saturation at 128 tokens (a common scenario with large models on H100s using FP8), demanding 512 tokens’ worth of compute forces the operation to slow down linearly. The verification time $T_{verify}$ exceeds the time it would have taken to just generate the tokens autoregressively. This is the Batching-Speculation Contention: batching fills the compute budget with width (more users), while speculation attempts to fill it with depth (more tokens per user). In a finite compute budget, these are mutually exclusive.1<\/span><\/p>\n A more subtle, systems-level failure occurs in memory management. In standard decoding, every sequence in the batch advances by exactly one token. The data structures are regular.<\/span><\/p>\n In Speculative Decoding, Sequence A might accept 3 draft tokens, Sequence B might accept 0, and Sequence C might accept 5. The batch becomes “ragged”.16<\/span><\/p>\n To survive in this high-contention, large-batch environment, Speculative Decoding must fundamentally change its verification cost structure. It cannot afford to verify large “draft trees” (e.g., checking 64 possible branches) because the batch size multiplier makes the compute cost prohibitive.<\/span><\/p>\n SpecFormer<\/b> 1<\/span>, introduced in research accepted for AAAI 2026, proposes a novel architecture designed specifically for this constraint.<\/span><\/p>\n The authors argue that the reliance on complex verification trees is a symptom of weak draft models. If the draft model is accurate enough, a single candidate sequence is sufficient. SpecFormer aims to maximize the quality of the draft sequence while maintaining extreme computational efficiency.<\/span>2<\/span><\/p>\n Standard autoregressive models use Unidirectional (Causal) Attention: token $t$ can only see $t-1 \\dots 0$. This is necessary for generation but limits the model’s ability to understand the local context of the phrase it is building.<\/span><\/p>\n BERT-style models use Bidirectional Attention, seeing the whole sequence, but cannot generate text.<\/span><\/p>\n SpecFormer integrates both:<\/span><\/p>\n This fused attention mechanism allows the draft model to capture strong local correlations (e.g., “New” implies “York” strongly in both directions locally) that a pure causal model might miss or be less confident about.<\/span><\/p>\n Parallel to SpecFormer, the <\/span>Falcon<\/b> framework (AAAI 2025) attacks the problem via <\/span>Semi-Autoregressive Drafting<\/b> and advanced distillation techniques.<\/span>4<\/span><\/p>\n Pure Non-Autoregressive (NAR) models generate all tokens simultaneously (independent of each other). This is fast but usually results in incoherent text (“The cat sat on the mat” -> “The sat cat mat on”).<\/span><\/p>\n Falcon employs a Semi-Autoregressive approach. It generates tokens in blocks, but maintains dependencies within the block. This strikes a balance: it is faster than token-by-token generation (latency reduction) but more accurate than pure NAR (quality preservation).4<\/span><\/p>\n The core innovation of Falcon is a training technique called <\/span>Coupled Sequential Glancing Distillation<\/b>.<\/span>4<\/span><\/p>\n Falcon also implements a dynamic decoding tree that adjusts based on confidence. Unlike SpecFormer\u2019s “no tree” approach, Falcon uses a “smart tree.” It can dynamically expand or contract the search space. If the draft model is highly confident (low entropy), it generates a single chain. If it is uncertain, it branches. This adaptivity is crucial for resource-constrained batching, as it allocates compute only where it is statistically likely to yield a return.<\/span>4<\/span><\/p>\n While SpecFormer and Falcon optimize the software, <\/span>Mirror Speculative Decoding (Mirror-SD)<\/b> argues that the hardware architecture itself is the bottleneck. It proposes solving resource contention by physically separating the contestants.<\/span>5<\/span><\/p>\n Modern inference servers and Edge SoCs (like the Apple M-series or NVIDIA Grace-Hopper) often contain heterogeneous accelerators:<\/span><\/p>\n Standard approaches leave the NPU idle. Mirror-SD offloads the <\/span>Draft Model<\/b> to the NPU while keeping the <\/span>Target Model<\/b> on the GPU.<\/span>5<\/span><\/p>\n By running the draft on the NPU, the GPU is completely freed from the sequential overhead of drafting.<\/span><\/p>\n2. The Physics of Interference: The Roofline Model and the Memory Wall<\/b><\/h2>\n
2.1 The Agony of Autoregression<\/b><\/h3>\n
2.2 The “Free Lunch” of Speculative Decoding<\/b><\/h3>\n
2.3 The Impact of Batching on Arithmetic Intensity<\/b><\/h3>\n
3. The Hegemony of Continuous Batching<\/b><\/h2>\n
3.1 From Static to Continuous Scheduling<\/b><\/h3>\n
\n
3.2 The Enabler: PagedAttention<\/b><\/h3>\n
3.3 The “Unfairness” of Stall-Free Scheduling<\/b><\/h3>\n
4. The Conflict: Resource Contention and the “Ragged Tensor”<\/b><\/h2>\n
4.1 Compute Contention<\/b><\/h3>\n
4.2 The Ragged Tensor Problem<\/b><\/h3>\n
\n
5. Architectural Solution A: SpecFormer and the Unification of Attention<\/b><\/h2>\n
5.1 The Philosophy: Better Drafts, No Trees<\/b><\/h3>\n
5.2 Unidirectional and Bidirectional Fusion<\/b><\/h3>\n
\n
5.3 Mechanism and Benefits<\/b><\/h3>\n
\n
6. Architectural Solution B: Falcon and Semi-Autoregressive Drafting<\/b><\/h2>\n
6.1 Semi-Autoregressive Generation<\/b><\/h3>\n
6.2 Coupled Sequential Glancing Distillation<\/b><\/h3>\n
\n
6.3 Custom-Designed Decoding Tree<\/b><\/h3>\n
7. Architectural Solution C: Mirror-SD and Heterogeneous Offloading<\/b><\/h2>\n
7.1 The GPU-NPU Dichotomy<\/b><\/h3>\n
\n
7.2 Breaking the Serial Barrier<\/b><\/h3>\n
\n