The proliferation of Large Language Models (LLMs) supporting context windows extending from 128,000 to over 10 million tokens has fundamentally altered the computational and architectural requirements of inference systems. As of late 2025, the primary bottleneck in serving these “infinite context” models has shifted from compute capabilities (FLOPS) to memory capacity and bandwidth, specifically regarding the Key-Value (KV) cache. The KV cache\u2014the intermediate state required to avoid redundant computation in autoregressive decoding\u2014has become the dominant consumer of GPU memory, often exceeding the aggregate High Bandwidth Memory (HBM) capacity of entire server clusters.<\/span><\/p>\n This report provides an exhaustive, expert-level analysis of the distributed systems, algorithms, and hardware architectures developed to manage this “Memory Wall.” We analyze the transition from single-node execution to disaggregated serving, detailing the mechanisms of middleware solutions like <\/span>LMCache<\/b>, <\/span>DistKV-LLM<\/b>, and <\/span>NVIDIA Dynamo<\/b>, alongside algorithmic innovations such as <\/span>Ring Attention<\/b>, <\/span>TokenRing<\/b>, and <\/span>DeepSpeed Ulysses<\/b>. Furthermore, we explore the integration of <\/span>Compute Express Link (CXL)<\/b> and Near-Data Processing (NDP) as the critical hardware substrate for next-generation inference. The analysis indicates that the future of LLM serving lies in a disaggregated, memory-centric architecture where the KV cache is treated not as a transient buffer, but as a persistent, distributed asset managed by intelligent fabrics spanning the data center.<\/span><\/p>\n To comprehend the necessity of distributed KV cache management, one must first rigorously quantify the resource demands of modern LLM inference. The core mechanism of the Transformer architecture involves the self-attention layer, where each token attends to all previous tokens in the sequence. In the autoregressive generation process, the model predicts the next token based on the entire history. To prevent the computationally prohibitive re-processing of this history for every new token generation, the Key (K) and Value (V) matrices computed for previous tokens are cached in GPU memory.<\/span><\/p>\n The memory footprint of the KV cache does not scale linearly with model size alone, but rather with the sequence length ($L$) and batch size ($B$). For a model with $N_{layers}$, hidden dimension $H$, number of heads $N_{heads}$, head dimension $d_{head}$, and element size $P$ (e.g., 2 bytes for FP16), the cache size $M_{KV}$ is governed by the equation:<\/span><\/p>\n <\/p>\n $$M_{KV} = 2 \\cdot B \\cdot L \\cdot N_{layers} \\cdot N_{heads} \\cdot d_{head} \\cdot P$$<\/span><\/p>\n For a 70-billion parameter model (e.g., Llama-3-70B) serving a batch of requests with a 1-million token context window, the KV cache requirements are staggering. A single request of 1 million tokens, assuming standard FP16 precision and typical architectural hyperparameters, can require hundreds of gigabytes of memory.<\/span>1<\/span> This far exceeds the HBM capacity of a single NVIDIA H100 (80GB) or even a standard node equipped with 8 GPUs. As context lengths expand to 10M tokens, as seen in models from Google and others, the KV cache becomes the dominant contributor to memory consumption, reducing the space available for model weights and activation buffers to a negligible fraction.<\/span>3<\/span><\/p>\n This scaling behavior introduces a critical “Memory Wall.” While GPU compute throughput has increased exponentially, memory capacity and bandwidth have lagged. The KV cache footprint scales linearly with context length, but the attention mechanism’s computational complexity scales quadratically ($O(L^2)$) during the prefill phase and linearly during decoding. However, in the regime of long contexts, the sheer volume of data that must be moved from HBM to the Tensor Cores during the decode phase saturates memory bandwidth, making the system bandwidth-bound rather than compute-bound.<\/span>5<\/span><\/p>\n Inference workloads are characterized by two distinct phases with opposing resource profiles, a dichotomy that drives modern system design:<\/span><\/p>\n This divergence has led to the architectural paradigm of <\/span>disaggregated serving<\/b>. In legacy systems, a single GPU handled both phases. In disaggregated architectures, the cluster is divided into “prefill instances” (optimized for compute) and “decode instances” (optimized for memory capacity and bandwidth). The prefill instances generate the initial KV cache, which is then transferred to decode instances for the long-tail generation process. This separation allows for independent scaling of resources but introduces a new bottleneck: the network bandwidth required to transport terabytes of KV cache data between nodes.<\/span>6<\/span><\/p>\n While hardware scaling addresses capacity, the computational complexity of attention over millions of tokens requires specialized algorithmic parallelism. Single-GPU execution is impossible for these lengths; thus, the sequence itself must be partitioned across multiple devices. The year 2024-2025 has seen the maturation of three primary strategies: Ring Attention, TokenRing, and DeepSpeed Ulysses.<\/span><\/p>\n Ring Attention<\/b> represents the foundational breakthrough for infinite-context processing. It circumvents the memory constraints of individual devices by performing self-attention and feedforward computations in a blockwise fashion, distributing the sequence dimension across a logical ring of devices.<\/span>8<\/span><\/p>\n In a Ring Attention setup, the input sequence is partitioned into blocks. Each GPU manages a specific block of Queries (Q), Keys (K), and Values (V). The calculation of attention scores requires every Query to interact with every Key in the global sequence. To achieve this without gathering the full sequence on every device (which would violate memory constraints), the devices pass their KV blocks to their neighbor in the ring.<\/span><\/p>\n The process operates in steps. In step $i$, a GPU computes the partial attention scores between its local Query block and the currently resident KV block. Simultaneously, it sends this KV block to the next GPU in the ring and receives a new KV block from the previous GPU. This “stream-and-compute” approach ensures that the full KV cache never needs to be materialized on a single device. The quadratic memory cost is effectively distributed across the cluster.<\/span>10<\/span><\/p>\n The critical efficiency of Ring Attention stems from its ability to overlap communication with computation. While the GPU’s compute units are calculating attention for the current block, the interconnect (e.g., NVLink or InfiniBand) is transmitting the data for the next block. Ideally, if the computation time exceeds the transmission time, the communication latency is completely hidden. However, as the number of devices ($N$) increases, the sequence chunk size per device decreases, potentially reducing the computation time per step while communication overhead remains constant or decreases linearly, leading to efficiency degradation at extreme scales.<\/span>11<\/span><\/p>\n While Ring Attention allows for context scaling, it relies on unidirectional peer-to-peer (P2P) communication, typically sending data only to the “next” device. This leaves the reverse bandwidth of full-duplex interconnects (like NVLink) idle. <\/span>TokenRing<\/b> addresses this inefficiency by leveraging bidirectional communication to further maximize the overlap of data transmission and computation.<\/span>11<\/span><\/p>\n TokenRing partitions the Q, K, and V tensors along the token dimension but fundamentally alters the data flow. Instead of merely rotating KV blocks, TokenRing enables the concurrent transmission of Query blocks and block outputs\u2014specifically block_out (the partial attention output) and block_lse (log-sum-exp statistics required for the Softmax normalization).<\/span><\/p>\n By utilizing a fully connected mesh topology or optimized ring variants, TokenRing allows a device to send its Query block to a neighbor while simultaneously receiving a Query block from another, or exchanging partial results. This effectively doubles the utilized bandwidth compared to standard Ring Attention. The algorithm is particularly effective in reducing the “bubble” time\u2014the idle time associated with the fill-up and drain phases of the pipeline. Experimental results demonstrate that TokenRing significantly enhances throughput and reduces communication latency, particularly in scenarios where the communication-to-computation ratio is high.<\/span>11<\/span><\/p>\n DeepSpeed Ulysses<\/b> adopts a fundamentally different parallelism strategy. Instead of rotating data in a ring, it utilizes collective communication primitives to partition the computation by attention heads rather than by sequence blocks during the attention phase.<\/span>13<\/span><\/p>\n In the Ulysses architecture, the input sequence is initially partitioned across GPUs (Sequence Parallelism). When the attention operation begins, the system triggers an <\/span>All-to-All<\/b> collective communication operation. This “transpose” operation reshuffles the data such that each GPU receives the full sequence but only for a specific subset of attention heads.<\/span><\/p>\n For example, if there are 8 GPUs and 64 attention heads, each GPU receives the full sequence for 8 heads. This allows the GPU to execute the standard FlashAttention kernel (or any optimized local attention kernel) without modification, as it has all the necessary K and V data for its assigned heads locally. Once the attention computation is complete, another All-to-All transpose returns the output to the original sequence-partitioned layout for the subsequent Feed-Forward Network (FFN) layers.<\/span>14<\/span><\/p>\n The primary advantage of Ulysses is its kernel agnosticism; it does not require specialized “ring” kernels and can leverage the highly optimized FlashAttention-3 immediately upon release. However, its scalability is strictly limited by the number of attention heads. The parallelism degree cannot exceed the number of heads, which can be a constraint for certain model architectures (e.g., those using Grouped Query Attention with few KV heads). Furthermore, the All-to-All operation places immense stress on the network bisection bandwidth. Unlike Ring Attention, which utilizes neighbor-to-neighbor links (efficient on linear topologies), Ulysses requires a high-bandwidth, fully connected fabric (like NVSwitch) to perform efficiently. In bandwidth-constrained environments (e.g., Ethernet), Ulysses often underperforms Ring Attention.<\/span>13<\/span><\/p>\n The choice between these algorithms depends heavily on the underlying hardware topology and the specific model architecture.<\/span><\/p>\n Table 1:<\/b> Comparative Analysis of Distributed Sequence Parallelism Strategies for Long-Context Inference.<\/span><\/p>\n While distributed attention algorithms solve the compute problem, the management of the KV cache data itself\u2014its storage, retrieval, and movement\u2014requires a sophisticated middleware layer. The transition to disaggregated serving, where prefill and decode occur on different nodes, necessitates a “storage-centric” view of inference.<\/span><\/p>\n LMCache<\/b> has emerged as a critical open-source solution designed to abstract the complexities of KV cache management. It functions as a middleware layer sitting between the inference engine (e.g., vLLM, SGLang) and the backend storage hierarchy. Its primary innovation is transforming the KV cache from an internal engine state into a first-class storage primitive that can be shared across engines and queries.<\/span>16<\/span><\/p>\n LMCache is architected to handle the rapid evolution of inference engines, where internal memory layouts change frequently (e.g., 15-20 new models released weekly). It employs a <\/span>Connector<\/b> pattern:<\/span><\/p>\n LMCache implements several performance-critical optimizations. <\/span>Layer-wise pipelining<\/b> allows the transfer of KV cache for layer $N+1$ to occur simultaneously with the computation of layer $N$. This hides the significant latency of fetching data from remote storage. Additionally, it supports <\/span>delayed decode storing<\/b>, where small, granular page updates from the decode phase are aggregated into larger chunks before being committed to storage, reducing the I\/O operations per second (IOPS) overhead on the storage backend.<\/span>16<\/span> The system also utilizes <\/span>zero-copy<\/b> mechanisms to move data between the network interface card (NIC) and GPU memory, bypassing the CPU to minimize latency.<\/span>16<\/span><\/p>\n For enterprise-grade, high-performance deployments, <\/span>NVIDIA Dynamo<\/b> provides a comprehensive framework for distributed inference. It addresses the “difficult UX” of manually managing distributed state by providing a unified control plane.<\/span><\/p>\n The engine of Dynamo is the <\/span>NVIDIA Inference Transfer Library (NIXL)<\/b>. NIXL is a specialized communication library optimized for the bursty, latency-sensitive traffic patterns of inference. Unlike generic MPI or TCP stacks, NIXL is aware of the GPU memory hierarchy. It abstracts the underlying physical transport (NVLink, InfiniBand, Ethernet) and supports a plugin architecture for various storage backends.<\/span>6<\/span><\/p>\n A key feature of NIXL is its support for <\/span>GPUDirect Storage<\/b> and <\/span>RDMA<\/b>. This enables <\/span>zero-copy<\/b> transfers where KV blocks are streamed directly from a storage appliance (e.g., a WEKA data grid) to the GPU HBM, completely bypassing the host CPU. This architecture eliminates the “jitter” caused by CPU OS scheduling and frees up the CPU to handle the complex logic of the Dynamo Planner. NIXL essentially turns the storage layer into an extension of the GPU memory.<\/span>19<\/span><\/p>\n Dynamo employs a <\/span>Smart Router<\/b> that fundamentally changes how requests are distributed. Traditional load balancers use Round Robin or Least Connections algorithms. The Smart Router, however, is <\/span>KV-cache aware<\/b>. It maintains a global index of which KV blocks reside on which GPU workers, structured as a <\/span>Radix Tree<\/b>. When a new request arrives, the router queries this tree to find the worker that already holds the longest matching prefix (e.g., a cached system prompt or document). It then routes the request to that specific worker. This maximizes the cache hit rate, drastically reducing the need for the prefill phase and lowering the Time To First Token (TTFT).<\/span>20<\/span><\/p>\n DistKV-LLM<\/b> proposes a decentralized approach to memory management, viewing the entire cluster’s memory as a unified pool.<\/span><\/p>\n DistKV-LLM segments the KV cache into <\/span>rBlocks<\/b> (replicated blocks), which are manageable sub-units that can be independently stored, migrated, and retrieved. The system implements a <\/span>coherence protocol<\/b> similar to CPU cache coherency (MESI), ensuring that when a KV block is updated during generation, all distributed copies are invalidated or updated. This allows the system to support complex beam search or parallel sampling patterns where multiple sequences diverge from a common prefix.<\/span>7<\/span><\/p>\n Unlike systems that statically partition memory, DistKV-LLM allows an instance facing a memory deficit (e.g., due to a surprisingly long generation) to proactively seek and reserve memory on less burdened instances. This dynamic elasticity ensures that a single long-context query does not crash a node due to Out-Of-Memory (OOM) errors, but rather spills over transparently to available resources elsewhere in the datacenter.<\/span>7<\/span><\/p>\n llm-d<\/b> provides the bridge between these high-performance inference techniques and the standard cloud-native operating model. It integrates deep into the Kubernetes networking stack.<\/span><\/p>\n llm-d introduces an <\/span>External Processing Pod (EPP)<\/b> that hooks into the Kubernetes Gateway API. The EPP inspects the headers and body of incoming HTTP requests to extract prompt metadata. It then queries the cluster state to score available pods based on “cache warmth”\u2014a metric indicating how much of the required KV cache is already resident. This routing logic happens at the ingress layer, ensuring that requests are directed to the optimal pod before they even reach the model server.<\/span>22<\/span><\/p>\n A standout feature of llm-d is its support for heterogeneous hardware. It can orchestrate a cluster where NVIDIA H100s are dedicated to the compute-intensive prefill phase, while older A100s or even CPU-based nodes handle the memory-bound decode phase or store cold KV blocks. This tiered architecture allows organizations to optimize Total Cost of Ownership (TCO) by matching hardware capabilities to the specific physics of each inference phase.<\/span>24<\/span><\/p>\n As software pushes the limits of existing interconnects, hardware architectures in 2025 are undergoing a revolution centered on <\/span>Compute Express Link (CXL)<\/b>. This open standard allows for the disaggregation of memory from the compute node, addressing the fundamental capacity limitations of HBM.<\/span><\/p>\n CXL enables GPUs to access remote memory with load\/store semantics, just like local DRAM, but over a PCIe-based fabric. Systems like <\/span>Beluga<\/b> leverage this to create a massive, shared KV cache pool.<\/span><\/p>\n In the Beluga architecture, a CXL memory pool is attached to the GPU server via a CXL switch. This pool appears to the OS and the GPU as a vast extension of physical memory. When the GPU HBM is full, KV blocks are evicted to the CXL pool. Crucially, because CXL supports memory semantics (load\/store), the GPU can access this data at cache-line granularity without the high overhead of block-based IO calls required for NVMe SSDs. Beluga demonstrates that CXL memory can reduce the Time-To-First-Token (TTFT) by nearly 90% compared to swapping to local SSDs, effectively breaking the memory capacity barrier for single-node serving.<\/span>25<\/span><\/p>\n While CXL solves the capacity problem, the bandwidth of the CXL link (typically PCIe Gen5 or Gen6 speeds) is significantly lower than HBM bandwidth. Moving terabytes of KV cache across the CXL link for every token generation can saturate the bus and stall the GPU. <\/span>CXL-NDP (Near-Data Processing)<\/b> solves this by moving the compute to the data.<\/span><\/p>\n In a CXL-NDP architecture, the CXL memory module is equipped with lightweight compute units (e.g., RISC-V cores or small accelerators). Instead of fetching the entire Key matrix to the GPU to compute attention scores, the GPU sends the Query vector to the CXL device. The CXL device computes the dot products between the Query and the locally stored Keys and returns only the resulting attention scores (or the top-k highest scores) to the GPU.<\/span><\/p>\n This operation reduces the data movement by orders of magnitude. The massive Key matrix never leaves the CXL module; only the tiny Query vector and the resulting scores traverse the interconnect. This architecture allows the system to scale to millions of tokens without being bottlenecked by the CXL bandwidth.4<\/span><\/p>\n Advanced CXL-NDP controllers also implement <\/span>transparent lossless compression<\/b>. As KV blocks are written to the CXL memory, a hardware engine compresses them using algorithms like LZ4 or ZSTD. They are decompressed on the fly when read. Furthermore, <\/span>precision-scalable bit-plane layouts<\/b> allow for dynamic quantization. For example, the system might retrieve only the most significant bits of the Value vectors for the initial attention pass, fetching the full precision only for the top-ranking tokens. This effectively amplifies the available bandwidth, improving throughput by over 40% in memory-constrained scenarios.<\/span>27<\/span><\/p>\n While distributed systems manage the “supply” of memory, algorithmic research focuses on reducing the “demand.” Innovations in 2025 are moving away from brute-force caching toward smarter, compressed representations of context.<\/span><\/p>\n Infini-attention<\/b> introduces a modification to the standard Transformer block to support effectively infinite context within a bounded memory footprint. It integrates a <\/span>compressive memory module<\/b> into the attention mechanism.<\/span><\/p>\n Instead of storing the discrete KV pairs for the entire history, Infini-attention maintains a fixed-size buffer. As the attention window slides forward, old KV states are not discarded but are “compressed” into this memory buffer using a linear attention mechanism. The model reuses the Query, Key, and Value states to update this compressed representation. During retrieval, the model attends to both the local, high-resolution context (standard attention) and the global, compressed context (linear attention) simultaneously. This allows the model to recall information from millions of tokens back without maintaining a multi-terabyte cache.<\/span>29<\/span><\/p>\n Jamba 1.5<\/b> represents a shift toward <\/span>Hybrid SSM-Transformer<\/b> architectures. Pure Transformers scale quadratically (or linearly with KV cache). State Space Models (SSMs) like <\/span>Mamba<\/b> scale linearly with constant memory state\u2014they do not grow a KV cache at all.<\/span><\/p>\n Jamba interleaves Mamba layers with standard Transformer layers. For example, a model might have one Transformer layer for every seven Mamba layers. The Mamba layers handle the bulk of the temporal processing with a fixed, small memory state. The sparse Transformer layers provide the “associative recall” capability that SSMs sometimes lack. This hybrid design allows Jamba-1.5-Large to achieve an effective context length of 256K tokens while fitting on a single 8-GPU node, a feat that would require massive clusters for a pure Transformer of equivalent parameter count. The KV cache footprint is reduced to a fraction (e.g., 1\/8th) of a standard model.<\/span>32<\/span><\/p>\n For pure Transformers, <\/span>eviction policies<\/b> determine which tokens can be safely discarded. 2025 has seen a move from simple heuristics to semantic-aware eviction.<\/span><\/p>\n1. The Physics of Long-Context Inference and the Memory Wall<\/b><\/h2>\n
1.1 The Scaling Laws of KV Memory<\/b><\/h3>\n
1.2 The Prefill-Decode Dichotomy and Disaggregation<\/b><\/h3>\n
\n
2. Distributed Attention Algorithms<\/b><\/h2>\n
2.1 Ring Attention: The Blockwise Foundation<\/b><\/h3>\n
2.1.1 Mechanism and Data Flow<\/b><\/h4>\n
2.1.2 Overlapping Communication and Computation<\/b><\/h4>\n
2.2 TokenRing: Optimizing Bidirectional Bandwidth<\/b><\/h3>\n
2.2.1 Concurrent Transmission Architecture<\/b><\/h4>\n
2.3 DeepSpeed Ulysses: The All-to-All Approach<\/b><\/h3>\n
2.3.1 The Transpose Mechanism<\/b><\/h4>\n
2.3.2 Trade-offs and Constraints<\/b><\/h4>\n
2.4 Comparative Analysis of Parallelism Strategies<\/b><\/h3>\n
\n\n
\n Feature<\/b><\/td>\n Ring Attention<\/b><\/td>\n TokenRing<\/b><\/td>\n DeepSpeed Ulysses<\/b><\/td>\n<\/tr>\n \n Partitioning Axis<\/b><\/td>\n Sequence Dimension<\/span><\/td>\n Sequence Dimension<\/span><\/td>\n Sequence (Input) $\\rightarrow$ Heads (Compute)<\/span><\/td>\n<\/tr>\n \n Communication Pattern<\/b><\/td>\n P2P Ring (Unidirectional)<\/span><\/td>\n P2P Mesh\/Ring (Bidirectional)<\/span><\/td>\n All-to-All Collective<\/span><\/td>\n<\/tr>\n \n Network Requirement<\/b><\/td>\n Neighbor-to-Neighbor (linear)<\/span><\/td>\n Mesh \/ Bidirectional Ring<\/span><\/td>\n High Bisection Bandwidth (NVSwitch)<\/span><\/td>\n<\/tr>\n \n Overlap Potential<\/b><\/td>\n High (KV comms hidden by compute)<\/span><\/td>\n Very High (Utilizes bidirectional BW)<\/span><\/td>\n Moderate (Harder to hide large collectives)<\/span><\/td>\n<\/tr>\n \n Scalability Limit<\/b><\/td>\n Device Count (Context Length)<\/span><\/td>\n Device Count<\/span><\/td>\n Number of Attention Heads<\/span><\/td>\n<\/tr>\n \n Kernel Complexity<\/b><\/td>\n High (Requires custom Ring kernels)<\/span><\/td>\n High (Custom communication schedule)<\/span><\/td>\n Low (Standard FlashAttention)<\/span><\/td>\n<\/tr>\n \n Primary Bottleneck<\/b><\/td>\n Latency of P2P steps<\/span><\/td>\n Orchestration complexity<\/span><\/td>\n Network Bisection Bandwidth<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n 3. Middleware Architectures and Disaggregated Serving<\/b><\/h2>\n
3.1 LMCache: The Unified Storage Substrate<\/b><\/h3>\n
3.1.1 Architectural Decomposition<\/b><\/h4>\n
\n
3.1.2 Optimization Mechanisms<\/b><\/h4>\n
3.2 NVIDIA Dynamo and NIXL<\/b><\/h3>\n
3.2.1 The NIXL Transport Layer<\/b><\/h4>\n
3.2.2 The Smart Router and Radix Trees<\/b><\/h4>\n
3.3 DistKV-LLM: Dynamic Memory Orchestration<\/b><\/h3>\n
3.3.1 The Coherence Protocol and rBlocks<\/b><\/h4>\n
3.3.2 Proactive Memory Seeking<\/b><\/h4>\n
3.4 llm-d: Kubernetes-Native Integration<\/b><\/h3>\n
3.4.1 The External Processing Pod (EPP)<\/b><\/h4>\n
3.4.2 Heterogeneous Hardware Orchestration<\/b><\/h4>\n
4. Hardware Acceleration: CXL and Near-Data Processing<\/b><\/h2>\n
4.1 CXL-Enabled Memory Expansion (Beluga)<\/b><\/h3>\n
4.2 CXL-NDP: Processing-Near-Memory<\/b><\/h3>\n
4.2.1 Offloading Attention Scores<\/b><\/h4>\n
4.2.2 Transparent Hardware Compression<\/b><\/h4>\n
5. Algorithmic Compressions and Hybrid Architectures<\/b><\/h2>\n
5.1 Infini-attention and Compressive Memory<\/b><\/h3>\n
5.2 Hybrid SSM-Transformer Models (Jamba)<\/b><\/h3>\n
5.3 Advanced Eviction Policies<\/b><\/h3>\n
\n