learning-path-sap-cloud By Uplatz<\/a><\/h3>\nII. The Foundational Challenge: Deconstructing Token Efficiency and the KV Cache Bottleneck<\/b><\/h2>\n
<\/p>\n
The core challenge in LLM inference stems from the Transformer architecture itself. To generate a new token, the model must attend to all previous tokens in the sequence. To avoid recomputing these, their internal representations\u2014the Key (K) and Value (V) tensors\u2014are cached in GPU memory.<\/span>19<\/span> This “KV cache” is the primary bottleneck for token efficiency, creating two compounding problems that vLLM and TRT-LLM are designed to solve.<\/span><\/p>\n <\/p>\n
Internal vs. External Fragmentation and GPU Memory Waste<\/b><\/h3>\n
<\/p>\n
Traditional inference systems employed static, reservation-based memory allocators.<\/span>5<\/span> For every incoming request, a contiguous block of GPU memory was reserved to hold the KV cache for the <\/span>maximum possible<\/span><\/i> sequence length (e.g., 32,768 tokens).<\/span>19<\/span><\/p>\nThis approach leads to catastrophic <\/span>internal fragmentation<\/span><\/i>. If a user’s actual sequence is only 2,000 tokens long, the memory reserved for the additional 30,768 tokens is wasted.<\/span>19<\/span> Analyses of such systems show that this “pre-allocation” model wastes between 60% and 80% of the total KV cache memory.<\/span>22<\/span> This memory waste directly limits the number of requests that can be processed concurrently, forcing smaller batch sizes, which in turn leads to lower GPU utilization and reduced throughput.<\/span>22<\/span> In contrast, the newer, dynamic methods reduce this waste to under 4%.<\/span>22<\/span><\/p>\n <\/p>\n
The Fallacy of Static Batching: Head-of-Line Blocking and GPU Underutilization<\/b><\/h3>\n
<\/p>\n
The second problem is scheduling. To improve GPU utilization, servers group multiple requests into a “batch”.<\/span>23<\/span> In a “static” or “naive” batching system, the server processes this batch and must wait for <\/span>every<\/span><\/i> sequence in the batch to finish generating its output before the <\/span>entire<\/span><\/i> batch is cleared and a new one can begin.<\/span>5<\/span><\/p>\nThis creates a “Head-of-Line Blocking” scenario.<\/span>24<\/span> A request generating a short, 50-token response is trapped, waiting for the slowest request in its batch, which might be generating 2,000 tokens. During this wait, the GPU is either idle or performing useless computation on “padding” tokens for the already-completed sequences.<\/span>5<\/span><\/p>\nThese two problems are inextricably linked. A dynamic scheduler that could evict finished requests is useless if the memory allocator is static and cannot immediately reuse the fragmented, freed memory. Conversely, a dynamic memory manager is sub-optimal if the scheduler is static and fails to fill the newly available memory blocks. The true innovation, therefore, was the co-design of a dynamic memory manager <\/span>and<\/span><\/i> a dynamic scheduler that work in concert.<\/span>24<\/span><\/p>\n <\/p>\n
III. vLLM: A Specialized Engine for High-Throughput Inference<\/b><\/h2>\n
<\/p>\n
vLLM, which originated from research at UC Berkeley <\/span>2<\/span>, directly addresses the dual challenges of memory fragmentation and inefficient scheduling. It was introduced in the 2023 paper, “Efficient Memory Management for Large Language Model Serving with Paged Attention”.<\/span>5<\/span><\/p>\n <\/p>\n
Core Innovation 1: PagedAttention and the Virtualization of KV Cache Memory<\/b><\/h3>\n
<\/p>\n
PagedAttention solves the memory fragmentation problem by borrowing a core concept from operating system design: virtual memory.<\/span>19<\/span> Instead of allocating a single, large, contiguous block for each sequence, PagedAttention partitions the KV cache into non-contiguous, fixed-size “blocks” (analogous to “pages” in an OS).<\/span>8<\/span><\/p>\nThe mechanism works as follows:<\/span><\/p>\n\n- Block Allocation:<\/b> The KV cache for a sequence is stored in these fixed-size blocks, which are physically non-contiguous in GPU memory.<\/span><\/li>\n
- Block Tables:<\/b> A “block table” (analogous to a “page table”) is created for each request. This table maps the <\/span>logical<\/span><\/i> token-level addresses of the sequence to the <\/span>physical<\/span><\/i> addresses of the blocks in GPU memory.<\/span>21<\/span><\/li>\n
- On-Demand Allocation:<\/b> As a sequence generates new tokens, blocks are allocated <\/span>on-demand<\/span><\/i>.<\/span>8<\/span> This “just-in-time” allocation completely eliminates internal fragmentation, as a sequence only ever uses the exact number of blocks it requires.<\/span>20<\/span><\/li>\n
- Efficient Sharing:<\/b> This architecture allows for advanced memory sharing. For example, in parallel sampling (where multiple outputs are generated from one prompt), the blocks for the initial prompt can be shared, with new blocks allocated only for the divergent, newly-generated tokens.<\/span>20<\/span><\/li>\n<\/ol>\n
<\/p>\n
Core Innovation 2: Continuous Batching and Iteration-Level Scheduling<\/b><\/h3>\n
<\/p>\n
With a dynamic memory system in place, vLLM introduces “Continuous Batching” to solve the scheduling problem.<\/span>24<\/span> This is an <\/span>iteration-level<\/span><\/i> or <\/span>token-level<\/span><\/i> scheduler.<\/span>28<\/span><\/p>\nIn contrast to static batching, the Continuous Batching process is dynamic:<\/span><\/p>\n\n- At <\/span>each<\/span><\/i> decoding step, the scheduler checks if any sequences in the currently processing batch have completed (i.e., generated an end-of-sequence token).<\/span>29<\/span><\/li>\n
- When a sequence finishes, its memory blocks are <\/span>immediately<\/span><\/i> freed and returned to the global pool via the PagedAttention memory manager.<\/span>5<\/span><\/li>\n
- The scheduler <\/span>immediately<\/span><\/i> fills this newly available GPU slot by pulling a new request from the waiting queue.<\/span>5<\/span><\/li>\n<\/ol>\n
This process ensures the GPU is never idle as long as requests are in the queue. It “absorbs latency variance” between different requests <\/span>28<\/span>, eliminates head-of-line blocking, and dramatically increases GPU utilization, leading to state-of-the-art throughput\u2014up to 24x higher than systems like HuggingFace Transformers.<\/span>5<\/span><\/p>\n <\/p>\n
System Architecture: The AsyncLLMEngine and Standalone Server<\/b><\/h3>\n
<\/p>\n
vLLM is architected with a core LLMEngine that handles the scheduling and model execution, and an AsyncLLMEngine wrapper that uses asyncio to manage concurrent requests for online serving.<\/span>30<\/span> This engine is famously easy to deploy, often requiring just a pip install vllm command.<\/span>19<\/span><\/p>\nIt provides a standalone, OpenAI-compatible API server out-of-the-box, which can be launched with a simple vllm serve <model> command.<\/span>19<\/span> This, combined with its optimized CUDA kernels (including integration with FlashAttention) <\/span>2<\/span>, speculative decoding, and parallelism support, makes it a powerful and accessible specialized engine.<\/span><\/p>\n <\/p>\n
IV. NVIDIA Triton and the TensorRT-LLM Engine: An Enterprise Platform for Accelerated Inference<\/b><\/h2>\n
<\/p>\n
The NVIDIA ecosystem presents a more complex, multi-component architecture. The comparison is not with “Triton” alone, but with “Triton serving a TensorRT-LLM engine.”<\/span><\/p>\n <\/p>\n
Triton’s Philosophy: A General-Purpose Server for Heterogeneous Workloads<\/b><\/h3>\n
<\/p>\n
NVIDIA Triton Inference Server is an “industrial-grade” <\/span>18<\/span> serving <\/span>platform<\/span><\/i> designed for enterprise-scale AI deployment.<\/span>3<\/span> Its primary philosophy is <\/span>versatility<\/span><\/i>. Triton is framework-agnostic, capable of serving models from virtually any framework, including PyTorch, TensorFlow, ONNX, and custom C++ or Python backends.<\/span>3<\/span><\/p>\nThis makes Triton the ideal solution for <\/span>mixed workloads<\/span><\/i>.<\/span>4<\/span> A single Triton instance can concurrently serve a vision model, a speech-to-text model, and multiple LLMs.<\/span>16<\/span> It provides a suite of enterprise-grade features, including:<\/span><\/p>\n\n- Concurrent model execution (loading multiple models on one or more GPUs) <\/span>16<\/span><\/li>\n
- A form of request-level dynamic batching <\/span>16<\/span><\/li>\n
- HTTP\/REST and GRPC endpoints <\/span>16<\/span><\/li>\n
- Liveness and readiness health endpoints <\/span>16<\/span><\/li>\n
- Utilization and performance metrics for monitoring <\/span>16<\/span><\/li>\n
- Deep integration with Kubernetes for scaling <\/span>1<\/span><\/li>\n<\/ul>\n
<\/p>\n
The Engine: Understanding the TensorRT-LLM Backend<\/b><\/h3>\n
<\/p>\n
To achieve high performance for LLMs <\/span>within<\/span><\/i> Triton, NVIDIA provides <\/span>TensorRT-LLM (TRT-LLM)<\/b>. TRT-LLM is an open-source <\/span>library<\/span><\/i> that accelerates and optimizes LLM inference.<\/span>9<\/span> It is not a server itself.<\/span><\/p>\nUsing its Python API, a developer <\/span>compiles<\/span><\/i> a standard LLM (e.g., from Hugging Face) into a highly optimized “engine”.<\/span>36<\/span> This compilation process applies state-of-the-art optimizations like layer fusion, kernel tuning, and advanced quantization.<\/span>31<\/span><\/p>\nThe <\/span>Triton TensorRT-LLM Backend<\/b> is the C++ component that allows the Triton server to load and serve these optimized TRT-LLM engines.<\/span>36<\/span> This backend is what implements the critical LLM-specific serving features.<\/span><\/p>\n <\/p>\n
Technological Convergence: TRT-LLM’s Adoption of Paged KV Caching and In-Flight Batching<\/b><\/h3>\n
<\/p>\n
The core memory and scheduling techniques of vLLM were so effective that they have become the industry standard, adopted by NVIDIA. The TensorRT-LLM backend includes:<\/span><\/p>\n\n- Paged KV Caching:<\/b> A memory management system conceptually identical to vLLM’s PagedAttention, designed to mitigate fragmentation by allocating memory in fixed-size blocks or “pages”.<\/span>6<\/span><\/li>\n
- In-Flight Batching (IFB):<\/b>
![\"\"](\"https:\/\/uplatz.com\/blog\/wp-content\/uploads\/2025\/11\/Token-Efficient-Inference-A-Comparative-Systems-Analysis-of-vLLM-and-NVIDIA-Triton-Serving-Architectures-1024x576.jpg\")
<\/p>\n