The latter half of 2025 marks a definitive inflection point in the trajectory of artificial intelligence. While the previous decade was defined by the race for parameter count\u2014scaling from millions to trillions of weights\u2014the current epoch is characterized by the “Context Window Explosion.” The ability to process, reason over, and retain vast amounts of information in a single forward pass has shifted from a theoretical capability to a production imperative. We are witnessing the transition of Large Language Models (LLMs) from stateless text processors, constrained by the sliding window of their immediate input, into stateful reasoning engines capable of ingesting entire codebases, legal archives, and video libraries.<\/span><\/p>\n By late 2025, the industry standard for enterprise-grade foundation models has firmly settled at the one-million-token mark, with vanguard architectures pushing orders of magnitude beyond. This shift is not merely quantitative; it represents a fundamental change in the utility function of generative AI.<\/span><\/p>\n The landscape is currently dominated by a diverse array of architectures, each targeting specific modalities of long-context understanding:<\/span><\/p>\n The explosion of context windows challenges the prevailing orthodoxy of Retrieval-Augmented Generation (RAG). For years, RAG has been the standard solution to the context limit: chunking documents, embedding them into vector databases, and retrieving top-k chunks based on semantic similarity. While efficient, this process is inherently lossy. It shreds global context, breaks cross-document reasoning chains, and relies on the imperfect proxy of vector similarity to determine relevance.<\/span><\/p>\n With 1M+ context windows, we are seeing the emergence of <\/span>“Many-Shot Learning”<\/b> or <\/span>“Long-Context Prompting.”<\/b> Instead of retrieving snippets, the system ingests the <\/span>entire<\/span><\/i> knowledge base\u2014the full manual, the complete case history, the whole codebase\u2014into the prompt. This allows the model’s native attention mechanism to perform reasoning over the raw data, identifying subtle connections and contradictions that vector search would inevitably miss.<\/span>4<\/span><\/p>\n Furthermore, this capacity is the foundational enabler for <\/span>Agentic Workflows<\/b>. An autonomous agent operating over days or weeks generates a massive trail of observations, tool outputs, and internal monologues. In a 4k token world, this history had to be aggressively summarized or discarded, forcing the agent to effectively “forget” its past. With million-token windows, agents can maintain a persistent “stream of consciousness,” recalling a specific error message from three days ago to inform a current debugging decision.<\/span>3<\/span><\/p>\n However, this capability comes at a steep physical cost. The move from 4k to 1M tokens increases the memory requirements for the Key-Value (KV) cache by factor of 250x. In a standard Transformer, this memory demand scales linearly with sequence length, while the compute required for attention scales quadratically. This creates a “Memory Wall” where the bottleneck for inference shifts decisively from compute (FLOPS) to memory capacity and bandwidth.<\/span><\/p>\n Sustaining these context lengths in production requires a sophisticated, hierarchical approach to memory management. We can no longer treat the GPU’s High Bandwidth Memory (HBM) as the sole repository for state. Instead, we must architect systems that treat GPU VRAM, CPU DRAM, and local NVMe SSDs as a unified, tiered address space, orchestrated by intelligent algorithms that move data just in time.<\/span><\/p>\n To understand the necessity of complex memory tiering, one must first analyze the arithmetic intensity and memory footprint of the Key-Value (KV) cache in modern Transformers. In an autoregressive model, generating the next token requires the model to attend to the hidden states (Keys and Values) of <\/span>all<\/span><\/i> preceding tokens. Recomputing these states for the entire history at every step would be computationally prohibitive ($O(N^3)$ complexity). Therefore, these tensors are cached in GPU memory, growing with every token generated.<\/span><\/p>\n The memory footprint of the KV cache ($M_{KV}$) for a standard Transformer is governed by the following relationship:<\/span><\/p>\n <\/p>\n $$M_{KV} = 2 \\times n_{layers} \\times n_{heads} \\times d_{head} \\times L_{seq} \\times P_{bytes}$$<\/span><\/p>\n Where:<\/span><\/p>\n Consider a production-grade model like <\/span>Llama-3-70B<\/b>. It features 80 layers ($n_{layers}$), and while it uses Grouped Query Attention (GQA) to reduce the number of KV heads, the memory footprint remains substantial. For a standard configuration without GQA reduction, a single request with a 1 million token context would generate a KV cache in the range of hundreds of gigabytes\u2014far exceeding the 80GB capacity of an NVIDIA H100 or even the 141GB of an H200.<\/span>6<\/span><\/p>\n Variable Cost Analysis:<\/span><\/p>\n It is crucial to distinguish between the Fixed Cost and the Variable Cost of inference memory.<\/span><\/p>\n The challenge of massive context is bifurcated into two distinct phases with opposing hardware bottlenecks: <\/span>Prefill<\/b> and <\/span>Decode<\/b>.<\/span><\/p>\n When a user submits a 1 million token prompt, the model processes all tokens in parallel to generate the initial KV cache and the first output token. This is a massive matrix multiplication workload.<\/span><\/p>\n Once the first token is generated, the model enters the autoregressive decode loop. To generate token $T_{N+1}$, the GPU must load the Key and Value vectors for tokens $T_1$ through $T_N$ from memory to the compute units.<\/span><\/p>\n This creates the <\/span>“Memory Wall.”<\/b> As context length ($L_{seq}$) increases, the time to load the KV cache increases linearly, eventually exceeding the time required to compute the token. When the KV cache exceeds local VRAM capacity and spills to system RAM (DRAM) or SSD, the bandwidth drops precipitously:<\/span><\/p>\n Moving a 100GB cache over PCIe Gen 5 (128 GB\/s max theoretical) would take nearly a second <\/span>per token<\/span><\/i>, rendering the application unusable for real-time interaction. This necessitates the complex tiering and “hiding” strategies discussed in this report.<\/span><\/p>\n To manage these massive, dynamic memory requirements, the software layer of the inference stack has undergone a revolution. The monolithic memory allocation strategies of 2023 have been replaced by sophisticated virtualization techniques borrowed from operating system theory.<\/span><\/p>\n The foundational technology enabling modern long-context inference is <\/span>PagedAttention<\/b>, popularized by the vLLM engine. Before PagedAttention, inference engines required contiguous memory allocation for the Key and Value tensors. If a user requested a 1M token window, the system had to pre-allocate a contiguous block of VRAM sufficient for 1M tokens.<\/span><\/p>\n This led to two forms of waste:<\/span><\/p>\n The Paging Solution:<\/span><\/p>\n PagedAttention divides the KV cache into fixed-size blocks (pages), typically containing the keys and values for 16 or 32 tokens. These blocks do not need to be contiguous in physical memory. A Block Manager (analogous to an OS Page Table) maintains the mapping between logical token indices and physical block addresses.9<\/span><\/p>\n While PagedAttention solved the basic fragmentation problem, the diversity of model architectures in 2025\u2014specifically the rise of Mixture-of-Experts (MoE) and models with varying embedding sizes\u2014introduced new complexities. A fixed block size that is optimal for one layer or expert might be suboptimal for another.<\/span><\/p>\n Jenga<\/b>, a memory allocation framework introduced in 2025 and implemented on top of vLLM, addresses this <\/span>heterogeneity<\/b>.<\/span>12<\/span><\/p>\n Despite these optimizations, a 100M token request will eventually exhaust VRAM. Production systems in late 2025 utilize sophisticated swapping mechanisms managed by the Block Manager.<\/span><\/p>\n The software virtualization described above relies on a robust physical substrate. The memory hierarchy for AI inference has formalized into three distinct tiers, each with specific bandwidth and capacity characteristics.<\/span><\/p>\n This is the “Hot” tier where active computation occurs.<\/span><\/p>\n This is the “Warm” tier.<\/span><\/p>\n This is the “Cold” tier, but in 2025 it has become an “Active” tier thanks to interface advancements.<\/span><\/p>\n1.1 The Million-Token Standard and Beyond<\/b><\/h3>\n
\n
1.2 The Obsolescence of RAG and the Rise of Many-Shot Learning<\/b><\/h3>\n
1.3 The Infrastructure Imperative<\/b><\/h3>\n
2. The Physics of the Memory Wall: Arithmetic of the KV Cache<\/b><\/h2>\n
2.1 The KV Cache Formula<\/b><\/h3>\n
\n
\n
2.2 The Bandwidth Bottleneck: Prefill vs. Decode<\/b><\/h3>\n
2.2.1 The Prefill Phase (Compute Bound)<\/b><\/h4>\n
\n
2.2.2 The Decode Phase (Memory Bandwidth Bound)<\/b><\/h4>\n
\n
\n
3. Memory Virtualization and Management: The Software Layer<\/b><\/h2>\n
3.1 vLLM and PagedAttention: The Foundation<\/b><\/h3>\n
\n
\n
3.2 Jenga: Heterogeneous Memory Allocation<\/b><\/h3>\n
\n
3.3 Handling Out-Of-Memory (OOM) in Production<\/b><\/h3>\n
\n
4. The Hardware Hierarchy: Tiering for Infinite Context<\/b><\/h2>\n
4.1 Tier 1: High Bandwidth Memory (HBM)<\/b><\/h3>\n
\n
4.2 Tier 2: Host Memory (DRAM)<\/b><\/h3>\n
\n
4.3 Tier 3: Local Storage (NVMe SSD)<\/b><\/h3>\n
\n