https:\/\/uplatz.com\/course-details\/bundle-combo-sap-s4hana-sales-and-s4hana-logistics\/509<\/a><\/p>\n1.1. Initial Architecture: The Single-Model, Single-Request Paradigm<\/b><\/h5>\n
The earliest serving patterns for LLMs, which emerged from the natural language processing (NLP) research community, mirrored the development environment by prioritizing simplicity and ease of implementation over performance.<\/span>1<\/span> The typical architecture involved a pre-trained model, such as an early Generative Pre-trained Transformer (GPT) or even an LSTM-based network, encapsulated within a basic web server framework like Flask or FastAPI.<\/span>2<\/span><\/p>\nThe operational workflow was linear and uncomplicated. An HTTP endpoint would receive a request containing a text prompt. The server would then process this request individually, tokenizing the input text, feeding it through the model, detokenizing the generated output, and returning the result as an HTTP response.<\/span>1<\/span> Concurrency was handled, if at all, through rudimentary process-level parallelism, where multiple instances of the server process would run, each handling one request at a time. This single-model, single-request paradigm was a direct translation of a research script into a service, a functional but profoundly inefficient approach to deployment.<\/span><\/p>\n <\/p>\n
1.2. Analysis of Core Deficiencies: The Triad of Unsustainability<\/b><\/h5>\n
<\/p>\n
As LLMs began to be explored for real-world applications, the limitations of this monolithic architecture became starkly apparent, manifesting as a triad of interconnected and unsustainable problems.<\/span><\/p>\nFirst, the <\/span>prohibitive cost<\/b> of this model made it non-viable for any application at scale. LLM inference is an extraordinarily resource-intensive process, demanding powerful and expensive hardware like GPUs or TPUs, which in turn consume substantial amounts of energy.<\/span>3<\/span> When processing requests sequentially, each on-demand use of this expensive hardware yielded a minuscule amount of work, resulting in an abysmal return on investment. The GPU would be active for a request, then sit idle waiting for the next one. This inefficient utilization meant that the operational costs, both for hardware amortization and energy consumption, could quickly spiral out of control, making the widespread adoption of the technology economically unsustainable.<\/span>3<\/span><\/p>\nSecond, the architecture delivered <\/span>unacceptable latency<\/b>. The process of generating text token by token, known as autoregressive decoding, is inherently sequential and slow.<\/span>5<\/span> Each new token depends on all previously generated tokens. Without any optimizations, the time it took to generate the first output token (Time-to-First-Token, or TTFT) and the time between subsequent tokens (Time-Per-Output-Token, or TPOT) were high.<\/span>6<\/span> For interactive applications such as chatbots or virtual assistants, where users expect near-instantaneous feedback, these long delays created a frustrating and unusable experience.<\/span>6<\/span><\/p>\nThird, the system demonstrated a fundamental <\/span>inability to scale<\/b>. The single-request processing model meant that as the number of concurrent users increased, requests would simply form a queue, waiting for the single model instance to become available. This led to cascading latency, where the wait time for users grew linearly with the number of people ahead of them in the queue. The architecture lacked any sophisticated mechanisms for load balancing, intelligent request scheduling, or dynamic scaling of resources to meet fluctuating demand, rendering it incapable of supporting production-level traffic.<\/span>7<\/span><\/p>\n <\/p>\n
1.3. The Inefficiency of Autoregressive Generation on Parallel Hardware<\/b><\/h5>\n
<\/p>\n
The root cause of these deficiencies was not a simple software engineering oversight but a fundamental mismatch between the computational pattern of the algorithm and the design of the underlying hardware. LLM inference for text generation is predominantly a <\/span>memory-bandwidth-bound<\/b> problem, not a compute-bound one.<\/span>9<\/span><\/p>\nModern GPUs are marvels of parallel computation, equipped with thousands of processing cores (e.g., CUDA cores, Tensor Cores) designed to perform trillions of floating-point operations per second (FLOPs) simultaneously.<\/span>12<\/span> However, in each step of the autoregressive generation loop, the system must load the model’s entire set of parameters\u2014often many gigabytes of data\u2014from the GPU’s high-bandwidth memory (HBM) into its faster on-chip cache just to perform the calculations necessary to generate a single output token.<\/span>11<\/span><\/p>\nThis process leaves the vast parallel processing capabilities of the GPU severely underutilized. The compute cores spend the majority of their time idle, waiting for the data transfer to complete.<\/span>9<\/span> This profound inefficiency\u2014the mismatch between a sequential, memory-intensive algorithm and a parallel, compute-rich hardware architecture\u2014is the central challenge that all subsequent innovations in LLM serving have sought to address. The initial failure of the simple API model was therefore not merely a flawed implementation but a symptom of a deeper architectural crisis. It became clear that to make LLM serving viable, the entire paradigm had to shift away from a “one prompt, one process” mindset, which treated the hardware as a single-use resource, toward a “many prompts, one shared hardware state” model that could keep the expensive parallel processors continuously fed with work. This realization was the genesis of the modern LLM serving stack.<\/span><\/p>\nSection 2: Unlocking GPU Potential: The Critical Role of Batching<\/b><\/h4>\n
<\/p>\n
In response to the critical inefficiency of the single-request paradigm, batching emerged as the first and most essential optimization. By grouping multiple requests together, serving systems could begin to leverage the parallel nature of GPU hardware, amortizing the high cost of loading model weights and dramatically improving throughput. However, the unique characteristics of LLM workloads\u2014specifically, their variable and unpredictable output lengths\u2014meant that simple batching strategies were insufficient. This section traces the evolution of batching from naive, static approaches to the sophisticated, dynamic techniques that now form the bedrock of high-performance LLM inference.<\/span><\/p>\n <\/p>\n
2.1. From Static to Dynamic Batching: A Necessary but Insufficient Step<\/b><\/h5>\n
<\/p>\n
The first logical step to combat GPU underutilization was <\/span>static batching<\/b>. In this approach, the serving system waits until a fixed number of requests have arrived, groups them into a single “batch,” and processes them simultaneously in one parallel operation.<\/span>2<\/span> This allows the GPU to perform the same matrix multiplications across multiple input sequences at once, sharing the cost of loading the model’s parameters from memory and thereby improving computational efficiency and throughput.<\/span>2<\/span><\/p>\nHowever, static batching introduced a significant new problem: head-of-line blocking. Because the batch is treated as an atomic unit, the entire group of requests must wait until the single longest-running sequence has finished generating its output. Only then can the results for all requests be returned and a new batch begin processing.<\/span>9<\/span> This leads to two major inefficiencies. First, requests that generate short responses are forced to wait unnecessarily for longer ones, increasing average and tail latency. Second, as shorter sequences complete, their corresponding slots in the GPU’s computational grid become idle, leading to wasted compute cycles, often depicted as “white blocks” in utilization diagrams.<\/span>9<\/span> Due to these latency issues, static batching is only practical for offline workloads, such as batch document processing, where immediate responsiveness is not a requirement.<\/span>13<\/span><\/p>\nTo mitigate the latency problems of static batching, systems evolved to use <\/span>dynamic batching<\/b>. This approach is more flexible; instead of waiting for a fixed batch size, the server collects incoming requests for a predetermined period of time (a time window) or until a certain batch size is reached, whichever occurs first.<\/span>12<\/span> This helps to strike a better balance between throughput and latency, ensuring that requests that arrive early are not held indefinitely waiting for a full batch to assemble.<\/span>12<\/span><\/p>\nDespite this improvement, dynamic batching still suffers from the same fundamental limitation as its static counterpart: it operates at the request level. The batch, once formed, is still processed as a single unit and is gated by its slowest member. Shorter requests must still wait for the longest one to finish, and GPU resources remain underutilized as sequences complete at different times.<\/span>12<\/span> It was a necessary step forward but ultimately an insufficient solution for the highly variable nature of LLM workloads.<\/span><\/p>\n <\/p>\n
2.2. The Breakthrough of Continuous Batching: Maximizing Throughput<\/b><\/h5>\n
<\/p>\n
The true breakthrough in maximizing GPU utilization and throughput came with the development of <\/span>continuous batching<\/b>, also known as in-flight batching or iteration-level scheduling.<\/span>9<\/span> This technique represents a fundamental paradigm shift. Instead of treating the batch as a static group of requests, it manages the batch composition dynamically at each individual token generation step.<\/span><\/p>\nThe mechanism is elegant in its efficiency. The serving system maintains a running batch of active requests. In each iteration, it performs a forward pass to generate one token for every sequence currently in the batch. As soon as any single sequence completes its generation (by emitting an end-of-sequence token), its slot in the batch is immediately freed. The system’s scheduler then instantly inserts a new, waiting request into that empty slot for the very next iteration.<\/span>9<\/span> This process is analogous to a modern assembly line where a finished product is immediately replaced by new raw materials, ensuring the line never stops and operates at maximum capacity.<\/span>12<\/span><\/p>\nThis approach, first detailed in the research paper on the Orca serving system <\/span>9<\/span>, decouples the lifecycles of the individual requests within the batch. No request has to wait for any other. This keeps the GPU’s parallel processors constantly supplied with work, dramatically increasing utilization from the 30-60% range typical of older methods to over 80-95%.<\/span>15<\/span> The resulting gains in throughput are substantial, with reports of 10-20x improvements over dynamic batching.<\/span>16<\/span> Today, continuous batching is the state-of-the-art method and a cornerstone feature of all major high-performance serving frameworks, including vLLM, SGLang, TensorRT-LLM, and Hugging Face’s Text Generation Inference (TGI).<\/span>12<\/span><\/p>\n <\/p>\n
2.3. Technical Deep Dive: Prefill vs. Decode in Continuous Batching<\/b><\/h5>\n
<\/p>\n
The implementation of continuous batching is more complex than the high-level model suggests, primarily because LLM inference is not a single, uniform process but consists of two distinct computational phases.<\/span>9<\/span><\/p>\n\n- Prefill (Prompt Processing):<\/b> When a new request arrives, its input prompt must first be processed. This is done in a single, highly parallel forward pass where the model computes the attention states (the Key-Value cache, discussed in the next section) for all tokens in the prompt simultaneously. This phase is typically compute-intensive and benefits from large parallel operations.<\/span>19<\/span><\/li>\n
- Decode (Token Generation):<\/b> After the prefill phase, the model enters the autoregressive loop, generating subsequent output tokens one by one. Each step in this phase is a small forward pass that processes a single token and is memory-bandwidth-bound.<\/span>19<\/span><\/li>\n<\/ol>\n
These two phases have starkly different computational profiles, making it challenging to batch them together efficiently. A large, compute-heavy prefill operation can stall the generation of single tokens for other requests in the decode phase. Modern continuous batching frameworks employ sophisticated schedulers to manage this complexity. They use heuristics and configurable hyperparameters, such as a waiting_served_ratio, to dynamically balance the allocation of GPU resources between new requests needing prefill and existing requests in the decode stage, thereby optimizing for both low latency and high throughput.<\/span>9<\/span><\/p>\nThis evolution in batching strategy reflects a critical shift in the conceptual model of the serving system’s core scheduling unit. Early systems operated on a coarse-grained, request-level unit, an approach borrowed from other machine learning domains where task lengths are more uniform. This model was fundamentally misaligned with the reality of LLM inference, where output length is highly variable and unknown in advance.<\/span>12<\/span> The system was perpetually held back by the longest-running task, leading to immense inefficiency.<\/span>9<\/span> Continuous batching succeeded because it abandoned the “batch as an atomic unit” concept. It recognized that the only true, consistent synchronization point in autoregressive generation is the single-token iteration. By redesigning the server’s scheduler to operate at this fine-grained, iteration level, it aligned the system’s architecture with the intrinsic properties of the workload. This was not merely a better batching algorithm; it was a re-architecting of the server’s core logic, which is why it delivered a step-function improvement in performance rather than an incremental one.<\/span><\/p>\n <\/p>\n
2.4. Comparison of Batching Strategies<\/b><\/h5>\n
<\/p>\n
The following table provides a comparative summary of the batching strategies discussed, highlighting their mechanisms, performance characteristics, and ideal use cases.<\/span><\/p>\n <\/p>\n
\n\n\n| Strategy<\/b><\/td>\n | Mechanism<\/b><\/td>\n | Granularity<\/b><\/td>\n | GPU Utilization<\/b><\/td>\n | Latency Profile<\/b><\/td>\n | Pros<\/b><\/td>\n | Cons<\/b><\/td>\n | Ideal Workload<\/b><\/td>\n<\/tr>\n\n| Static Batching<\/b><\/td>\n | Waits for a fixed number of requests (N) to arrive, then processes them as a single unit.<\/span><\/td>\n | Request-Level<\/span><\/td>\n | Low to Medium<\/span><\/td>\n | High average and tail latency due to head-of-line blocking.<\/span><\/td>\n | Simple to implement; improves throughput over no batching.<\/span><\/td>\n | Inefficient for variable-length sequences; high latency; wasted GPU cycles.<\/span><\/td>\n | Offline, non-interactive tasks (e.g., document summarization).[2, 12, 13]<\/span><\/td>\n<\/tr>\n\n| Dynamic Batching<\/b><\/td>\n | Collects requests that arrive within a time window or until a size limit is reached.<\/span><\/td>\n | Request-Level<\/span><\/td>\n | Medium<\/span><\/td>\n | Improved average latency over static, but still high tail latency.<\/span><\/td>\n | Balances throughput and latency better than static batching.<\/span><\/td>\n | Still gated by the longest sequence in the batch; GPU underutilization persists.<\/span><\/td>\n | Workloads with short or uniform-length responses.[12, 14, 16]<\/span><\/td>\n<\/tr>\n\n| Continuous Batching<\/b><\/td>\n | Manages batch composition at each token generation step. Finished sequences are immediately replaced with new ones.<\/span><\/td>\n | Iteration-Level<\/span><\/td>\n | High (80-95%+)<\/span><\/td>\n | Low average and tail latency; enables high throughput.<\/span><\/td>\n | Maximizes GPU utilization; dramatically increases throughput; fair scheduling.<\/span><\/td>\n | More complex to implement; requires sophisticated scheduling for prefill\/decode.<\/span><\/td>\n | Online, interactive, and high-throughput applications (e.g., chatbots, APIs).<\/span>9<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\nPart II: Core Architectural Innovations for High-Performance Serving<\/b><\/h3>\n <\/p>\n With batching established as the foundational method for improving GPU utilization, the focus of architectural innovation shifted to addressing the next major bottleneck: memory. The sheer size of LLMs and their associated state, particularly the Key-Value (KV) cache, presented a formidable challenge that limited context lengths, constrained batch sizes, and ultimately capped system throughput. This part of the report provides a deep dive into the specific technologies developed to solve these core problems. It examines the critical role of memory management, the strategies for distributing inference across multiple GPUs, and the cutting-edge optimizations that are pushing the boundaries of latency and efficiency.<\/span><\/p>\nSection 3: Taming the Memory Beast: The KV Cache and PagedAttention<\/b><\/h4>\n <\/p>\n Memory is the single most critical and constrained resource in LLM serving. The quest for longer context windows and higher concurrency is fundamentally a battle for efficient memory management. This section explores the dual nature of the KV cache\u2014both an essential performance accelerator and a primary memory bottleneck\u2014and details the revolutionary impact of PagedAttention, a technique that applied principles from classical operating systems to solve the GPU memory crisis.<\/span><\/p>\n <\/p>\n 3.1. The KV Cache Explained: An Accelerator and a Bottleneck<\/b><\/h5>\n <\/p>\n The attention mechanism, a core component of the Transformer architecture, allows a model to weigh the importance of different tokens in the input sequence when generating a new token. In its naive form, this mechanism has a computational complexity that scales quadratically with the sequence length ($O(n^2)$), where $n$ is the number of tokens. For long sequences, this quadratic scaling becomes computationally prohibitive.<\/span>10<\/span><\/p>\nThe <\/span>Key-Value (KV) cache<\/b> is an optimization designed to overcome this bottleneck. During autoregressive generation, the key (K) and value (V) vectors, which are projections of the input tokens, remain constant for all previously processed tokens. Instead of recomputing these vectors at every single generation step, they are computed once and stored\u2014or “cached”\u2014in the GPU’s memory. For each new token, the model only needs to compute the K and V vectors for that single new token and can then retrieve the vectors for all previous tokens from the cache. This simple but powerful technique transforms the attention computation from quadratic to linear ($O(n)$) complexity, making generation for long sequences feasible.<\/span>10<\/span><\/p>\nHowever, this performance gain comes at a steep price: memory consumption. The KV cache must store two vectors (K and V) for every token in the sequence, for every attention head, and for every layer in the model. Its size grows linearly with both the batch size and the total sequence length.<\/span>10<\/span> For a large model like Llama-2 7B, the KV cache can consume around 0.5 MB per token, while for a 176B parameter model, this can be as high as 4 MB per token.<\/span>10<\/span> In many scenarios, the total memory required for the KV cache of a large batch of requests can easily exceed the memory required for the model weights themselves, becoming the primary factor limiting how many requests can be processed concurrently and how long the context window can be.<\/span>5<\/span><\/p>\n <\/p>\n 3.2. The Memory Crisis: Fragmentation and Over-allocation<\/b><\/h5>\n <\/p>\n Early serving systems managed this burgeoning KV cache with a simple and highly inefficient strategy: they would pre-allocate a single, large, contiguous block of GPU memory for each incoming request. This block had to be large enough to accommodate the maximum possible sequence length supported by the model (e.g., 4096 or 8192 tokens).<\/span>5<\/span> This approach led to a severe memory crisis characterized by two classic forms of memory waste.<\/span><\/p>\n\n- Internal Fragmentation:<\/b> Since most prompts and their generated outputs are significantly shorter than the maximum possible length, a large portion of each pre-allocated memory block would go unused. For a request that only used 500 tokens in a system with a 4096-token context window, nearly 88% of its reserved memory would be wasted. This wasted space could not be used by any other request.<\/span>23<\/span><\/li>\n
- External Fragmentation:<\/b> The requirement for <\/span>contiguous<\/span><\/i> memory blocks created another problem. Over time, as requests of different sizes were allocated and freed, the GPU memory would become fragmented into many small, non-contiguous free spaces. Even if the total amount of free memory was large, the system might be unable to serve a new request because it could not find a single <\/span>contiguous<\/span><\/i> block large enough to satisfy the allocation, stranding the available memory.<\/span>25<\/span><\/li>\n<\/ol>\n
This rampant inefficiency in memory management directly capped the number of concurrent requests a system could support, thereby limiting its overall throughput and making it prohibitively expensive to serve applications requiring long context windows.<\/span>5<\/span><\/p>\n <\/p>\n 3.3. PagedAttention: A Paradigm Shift in GPU Memory Management<\/b><\/h5>\n <\/p>\n The solution to this memory crisis came from an unexpected source: the decades-old principles of virtual memory and paging used in modern operating systems. The <\/span>PagedAttention<\/b> algorithm, introduced as the core innovation of the vLLM serving engine, fundamentally re-architected GPU memory management for LLM inference.<\/span>5<\/span><\/p>\nThe core insight was to recognize the analogy between the problems of LLM serving and traditional OS memory management: a request’s KV cache is like a process’s memory space, individual tokens are like bytes, and the fixed-size memory chunks are like pages.<\/span>5<\/span> Based on this, PagedAttention abandons the contiguous allocation model. Instead, it partitions the KV cache of each sequence into small, fixed-size <\/span>“pages”<\/b> or <\/span>“blocks.”<\/b> These blocks can be stored anywhere in physical GPU memory, in non-contiguous locations. A per-request <\/span>block table<\/b> maintains the mapping between the logical blocks of a sequence (i.e., the first block, second block, etc.) and their actual physical addresses in memory.<\/span>25<\/span><\/p>\nThis seemingly simple abstraction had profound benefits:<\/span><\/p>\n\n- Near-Zero Fragmentation:<\/b> By allocating small blocks on demand as a sequence grows, PagedAttention virtually eliminates internal fragmentation. Since all blocks are the same size, it also completely eliminates external fragmentation. This leads to near-optimal memory utilization, with studies showing memory waste reduced to as low as 4%.<\/span>23<\/span><\/li>\n
- Efficient Memory Sharing:<\/b> The paged abstraction enables far more sophisticated and granular memory sharing. In use cases like parallel sampling (generating multiple outputs for one prompt) or beam search, the different candidate sequences can share the physical memory blocks for their common prefix. When a sequence diverges, a <\/span>copy-on-write<\/b> mechanism is employed: a new physical block is allocated, and only the divergent data is copied, while the shared prefix remains untouched. This dramatically reduces the memory overhead for complex decoding strategies.<\/span>25<\/span> This same mechanism also provides a highly efficient foundation for prompt caching across different user requests, which will be discussed later.<\/span>29<\/span><\/li>\n<\/ul>\n
The development of PagedAttention was not just a clever optimization; it was a necessary paradigm shift. It transformed the problem from a brute-force challenge of “how much memory can we provision?” to a more sophisticated one of “how efficiently can we manage the memory we have?” This fundamental change in approach unlocked a new level of performance and altered the economic calculus of LLM serving, making high-concurrency services with large context windows economically viable for the first time.<\/span><\/p>\n <\/p>\n 3.4. Advanced KV Cache Strategies: Eviction, Offloading, and Compression<\/b><\/h5>\n <\/p>\n Building on the foundation of efficient memory management provided by PagedAttention, several other advanced strategies have been developed to further optimize KV cache usage, especially for extremely long contexts.<\/span><\/p>\n\n- Eviction Policies:<\/b> When a sequence is so long that its KV cache cannot fit entirely in GPU memory, the system must decide which parts of the cache to discard or “evict.” These policies can be static, such as <\/span>sliding window attention<\/b>, which only keeps the cache for the most recent tokens, or more nuanced approaches that always retain the initial tokens, as they often contain important global context.<\/span>10<\/span> Dynamic policies are more sophisticated, using runtime information like attention scores to identify and evict the least important tokens, thereby preserving the most salient context.<\/span>20<\/span><\/li>\n
- KV Cache Offloading:<\/b> For applications with intermittent user interaction, such as a chatbot session that might be idle for several minutes or hours, it is inefficient to keep the entire KV cache resident in expensive GPU memory. <\/span>KV cache offloading<\/b> is a technique where the cache for inactive sessions is moved from the GPU to more abundant and cheaper CPU RAM or even disk storage. When the user re-engages, the cache is loaded back into the GPU, avoiding the need to recompute the entire conversation history from scratch and significantly reducing the time-to-first-token for resumed interactions.<\/span>22<\/span><\/li>\n
- Quantization:<\/b> Another effective technique is to reduce the memory footprint of the KV cache itself. By <\/span>quantizing<\/b> the floating-point values in the key and value vectors to lower-precision formats like FP8 or INT8, the size of the cache can be reduced by half or more. This allows a larger effective batch size to fit within the same amount of GPU memory, directly increasing system throughput.<\/span>30<\/span><\/li>\n<\/ul>\n
Section 4: Scaling Giant Models: Architectures for Distributed Inference<\/b><\/h4>\n <\/p>\n As the parameter counts of state-of-the-art LLMs escalated into the hundreds of billions, a new architectural challenge emerged: many models became too large to fit within the memory of a single GPU. A 70-billion parameter model like Llama 3.1, for instance, requires approximately 140 GB of VRAM for its weights alone, far exceeding the capacity of even top-tier enterprise GPUs.<\/span>32<\/span> This necessitated the development of distributed inference techniques, collectively known as <\/span>model parallelism<\/b>, which partition a single model across a cluster of interconnected GPUs.<\/span><\/p>\n <\/p>\n 4.1. The Need for Model Parallelism<\/b><\/h5>\n <\/p>\n Beyond the model weights, the memory requirements for the KV cache and intermediate activations during inference also scale with model size. For a large model processing a long context request, the total memory footprint can easily reach hundreds of gigabytes.<\/span>32<\/span> Model parallelism addresses this by splitting the model itself, allowing a group of GPUs to collaborate on processing a single inference request, with each GPU holding only a fraction of the total model state.<\/span>33<\/span> There are two primary strategies for achieving this: tensor parallelism and pipeline parallelism.<\/span><\/p>\n <\/p>\n 4.2. Tensor Parallelism (TP): Intra-Layer Parallelization<\/b><\/h5>\n <\/p>\n Tensor parallelism<\/b> is a technique that parallelizes the computations <\/span>within<\/span><\/i> each layer of the model. It focuses on the most computationally intensive operations in a Transformer block: the large matrix multiplications. In this approach, the large weight matrices of a layer are partitioned or “sharded” across multiple GPUs.<\/span>33<\/span><\/p>\n | | | |
![\"\"](\"https:\/\/uplatz.com\/blog\/wp-content\/uploads\/2025\/11\/LLM-Serving-Architecture-1-1024x576.jpg\")
<\/p>\n