https:\/\/uplatz.com\/course-details\/bundle-combo-sap-ewm-ecc-and-s4hana\/316<\/a><\/p>\n1.1 The Central Conflict: Throughput vs. Latency<\/b><\/h3>\n
For cloud vendors and AI service providers, the primary objective is <\/span>high throughput<\/b>.<\/span>2<\/span> This is measured in metrics like tokens per second or requests per second, and it is the key to maximizing the utilization of expensive GPU hardware.<\/span>1<\/span> High throughput lowers the cost per token, enabling a scalable and profitable business.<\/span>2<\/span><\/p>\nFor the end-user of an interactive application, the primary concern is <\/span>low latency<\/b>.<\/span>2<\/span> Whether in a chatbot or a code assistant, the user demands a high Quality-of-Service (QoS), which is defined by a feeling of responsiveness and “real-time” interaction.<\/span>1<\/span><\/p>\nThis economic and engineering trade-off is the central challenge of LLM serving.<\/span>3<\/span> Every architectural decision, from batching algorithms to model sharding, is an attempt to navigate this core conflict.<\/span>4<\/span><\/p>\n1.2 The Two-Phase Problem: The Prefill vs. Decode Dichotomy<\/b><\/h3>\n
The technical root of the throughput-latency conflict lies in the unique, two-phase nature of LLM inference. Every user request forces the system to execute two distinct workloads with diametrically opposed performance characteristics.<\/span>5<\/span><\/p>\nPhase 1: Prefill (Compute-Bound)<\/span><\/p>\nThe prefill stage involves processing all tokens in the user’s input prompt in parallel.6 In this phase, the model generates the Key-Value (KV) cache, a data structure that stores the attention state of the prompt.9 Because this involves large, parallel matrix multiplications across all input tokens, the prefill stage is compute-bound.5 It can effectively saturate the GPU’s computational units, and its duration is proportional to the length of the input prompt.8<\/span><\/p>\nPhase 2: Decode (Memory-Bound)<\/span><\/p>\nThe decode stage is the auto-regressive generation of the response, one token at a time.7 Each newly generated token depends on all the tokens that came before it.6 This process is memory-bound.1<\/span><\/p>\nThe bottleneck is not the mathematical computation, which is small for a single token.<\/span>8<\/span> Instead, the bottleneck is the <\/span>memory bandwidth<\/b>.<\/span>1<\/span> For every single token generated, the GPU must load the <\/span>entire<\/span><\/i> multi-billion-parameter model and the <\/span>entire<\/span><\/i> (and growing) KV cache from high-bandwidth memory (HBM). This operation starves the GPU’s powerful compute units, leaving them <\/span>severely underutilized<\/span><\/i>.<\/span>2<\/span><\/p>\n <\/p>\n
1.3 The Interleaving Bottleneck<\/b><\/h3>\n
<\/p>\n
The fundamental challenge for any LLM serving system is that every single request <\/span>interleaves<\/span><\/i> these two wildly different compute paradigms.<\/span>4<\/span> The system must continuously schedule and execute a mix of compute-heavy parallel tasks (prefills) and memory-bandwidth-heavy sequential tasks (decodes).<\/span><\/p>\nA naive scheduler is forced into a difficult choice <\/span>4<\/span>:<\/span><\/p>\n\n- Prioritize Prefill (Optimize for Throughput):<\/b> When a new, compute-heavy prefill request arrives, the system <\/span>stalls<\/span><\/i> all ongoing, low-latency decode requests to process it. This gets new work onto the GPU quickly, maximizing throughput. However, it <\/span>destroys<\/span><\/i> the user experience for everyone else, causing “generation stalls” and high perceived latency.<\/span>2<\/span><\/li>\n
- Prioritize Decode (Optimize for Latency):<\/b> The system finishes all current decode requests before starting any new prefill. This provides a smooth experience for existing users, but it <\/span>wastes<\/span><\/i> GPU compute cycles and <\/span>lowers<\/span><\/i> overall system throughput, as the GPU sits idle waiting to start new work.<\/span>4<\/span><\/li>\n<\/ol>\n
This prefill-decode dichotomy is the “Rosetta Stone” of LLM inference performance. Every optimization discussed in this report\u2014batching, sharding, quantization, and speculative decoding\u2014is a direct attempt to solve the architectural mismatch and scheduling problems created by these two phases. Some advanced architectures even propose using entirely different hardware for each phase, highlighting how distinct they are.<\/span>14<\/span><\/p>\n <\/p>\n
Section 2: Deconstructing Performance: Latency and the Perception of Speed<\/b><\/h2>\n
<\/p>\n
To optimize the user experience, “latency” must be broken down into specific, user-facing metrics that quantify the “feel” of an AI application. The two most important metrics are Time to First Token (TTFT) and Time Per Output Token (TPOT).<\/span>16<\/span><\/p>\n <\/p>\n
2.1 Metrics That Define the User Experience<\/b><\/h3>\n
<\/p>\n
\n- Time to First Token (TTFT):<\/b> The duration from when a user submits a request to when the <\/span>first token<\/span><\/i> of the response appears on their screen.<\/span>16<\/span><\/li>\n
- Time Per Output Token (TPOT):<\/b> The average time <\/span>between<\/span><\/i> subsequent output tokens after the first one. This measures the “speed” of text generation.<\/span>16<\/span><\/li>\n
- End-to-End Latency:<\/b> The total time from request submission to the <\/span>final token<\/span><\/i> of the response.<\/span>17<\/span> This can be calculated using the formula: $Latency = TTFT + (TPOT \\times (Total\\_Output\\_Tokens – 1))$.<\/span>7<\/span><\/li>\n<\/ul>\n
<\/p>\n
2.2 Mapping Architectural Phases to User-Facing Metrics<\/b><\/h3>\n
<\/p>\n
These user-facing metrics map directly to the two-phase technical problem identified in Section 1:<\/span><\/p>\n\n- TTFT is the “Prefill Cost”:<\/b> TTFT is <\/span>dominated<\/span><\/i> by the <\/span>prefill stage<\/b>.<\/span>12<\/span> It is the direct, user-felt time cost of processing the entire input prompt (plus any network and queuing delays) and generating the very first token.<\/span>7<\/span><\/li>\n
- TPOT is the “Decode Cost”:<\/b> TPOT is the direct, user-felt cost of the <\/span>decode stage<\/b>.<\/span>19<\/span> It is a pure measure of the system’s performance in the memory-bandwidth-bound, auto-regressive generation loop.<\/span>7<\/span><\/li>\n<\/ul>\n
<\/p>\n
2.3 How TTFT and TPOT Shape Application “Feel”<\/b><\/h3>\n
<\/p>\n
Optimizing for TTFT vs. TPOT has radically different impacts on user perception.<\/span><\/p>\nTTFT: The “Silent Killer” of Responsiveness<\/span><\/p>\nA high TTFT is the initial “pause” or “lag” 19 that makes an application feel “dead” or “broken.” Even if the subsequent text generation (TPOT) is instantaneous, a long initial wait shatters the illusion of interactivity and conversation.16 This metric is therefore the primary driver of perceived responsiveness.<\/span><\/p>\n\n- Use Case (Chatbot):<\/b> A conversational AI <\/span>must<\/span><\/i> have a low TTFT to feel responsive. A common target is under 500 milliseconds.<\/span>16<\/span><\/li>\n
- Use Case (Code Completion):<\/b> The requirement is even more extreme. A code assistant must integrate into a developer’s “flow state” and feel “instant,” demanding a TTFT well below 100 milliseconds.<\/span>16<\/span><\/li>\n<\/ul>\n
TPOT: The “Flow State” of Generation<\/span><\/p>\nTPOT determines the “smoothness” and “speed” of the streaming response.7 A low, stable TPOT feels fluid and natural. A high or, just as importantly, a variable TPOT makes the text appear in “bursts,” which can be just as disruptive to the user experience as a high TTFT.16<\/span><\/p>\nHowever, TPOT has a “perceptual floor” defined by human reading speed. A user in a chatbot is <\/span>reading<\/span><\/i> as the text is generated.<\/span>21<\/span> One analysis notes that a TPOT of 100 milliseconds (10 tokens\/second) is equivalent to approximately 450 words per minute, which is <\/span>faster than a typical person can read<\/span><\/i>.<\/span>7<\/span><\/p>\n <\/p>\n
2.4 The Optimization Trap (TTFT vs. TPOT)<\/b><\/h3>\n
<\/p>\n
This “perceptual floor” for TPOT reveals a critical optimization trap. Optimizing TPOT <\/span>beyond<\/span><\/i> the point of human perception (e.g., from 80ms to 40ms) provides no discernible user benefit <\/span>21<\/span> and is a wasted engineering effort. That effort could have been redirected to the <\/span>far more critical<\/span><\/i> TTFT.<\/span><\/p>\nFurthermore, optimizing for overall system throughput (e.g., by using large batches) <\/span>actively degrades<\/span><\/i> both metrics. A large batch of 16 requests will have a higher TTFT (as 16 prefills must be processed) and a higher TPOT for each user (as the GPU’s memory bandwidth is split 16 ways).<\/span>7<\/span> This makes an architecture optimized for <\/span>maximum throughput<\/span><\/i> completely unusable for <\/span>interactive applications<\/span><\/i>.<\/span><\/p>\nThe optimal strategy for interactive apps is therefore bifurcated:<\/span><\/p>\n\n- Dedicate all available resources to minimizing <\/span>TTFT<\/b> to the lowest possible value (the “responsiveness” bottleneck).<\/span><\/li>\n
- Optimize <\/span>TPOT<\/b> only<\/span><\/i> to the point of “perceptual smoothness” (e.g., 50-150ms per token).<\/span>7<\/span><\/li>\n<\/ol>\n
<\/p>\n
\n\n\n| Metric<\/b><\/td>\n | Technical Driver<\/b><\/td>\n | System Bottleneck<\/b><\/td>\n | User Perception<\/b><\/td>\n | Critical For<\/b><\/td>\n<\/tr>\n\n| TTFT<\/b><\/td>\n | Prefill Stage <\/span>12<\/span><\/td>\n | Compute-bound <\/span>8<\/span><\/td>\n | “Responsiveness,” “Wait,” “Lag” <\/span>19<\/span><\/td>\n | Chatbots, Code Assistants <\/span>16<\/span><\/td>\n<\/tr>\n\n| TPOT<\/b><\/td>\n | Decode Stage <\/span>19<\/span><\/td>\n | Memory-bandwidth-bound [1, 5]<\/span><\/td>\n | “Speed,” “Flow,” “Smoothness” <\/span>7<\/span><\/td>\n | Streaming Chat, Long-form Generation <\/span>16<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n <\/p>\n Section 3: The Throughput Engine: Batching Strategies from Static to Continuous<\/b><\/h2>\n <\/p>\n 3.1 Why Batching is Non-Negotiable<\/b><\/h3>\n <\/p>\n As established in Section 1, the decode phase is memory-bound and leaves GPU compute units <\/span>dramatically<\/span><\/i> underutilized.<\/span>2<\/span> Running a single request (a batch size of 1) is profoundly inefficient and cost-prohibitive.<\/span><\/p>\nBatching<\/b>\u2014processing multiple requests in parallel\u2014is the <\/span>primary<\/span><\/i> technique for increasing compute utilization. By feeding the GPU enough parallel work, the system can better hide the memory-bound nature of individual decodes, significantly increasing overall system throughput <\/span>7<\/span> and reducing the operational cost per request.<\/span>23<\/span><\/p>\n <\/p>\n 3.2 The Evolution of Batching Algorithms<\/b><\/h3>\n <\/p>\n The strategy used to group requests for batching has evolved significantly, with each step attempting to solve the inefficiencies of the last.<\/span><\/p>\n\n- Static Batching:<\/b> This is the most basic approach. The server waits to collect a <\/span>full, fixed-size<\/span><\/i> batch (e.g., 16 requests), processes all of them simultaneously, and only returns the results when all requests in the batch are complete.<\/span>24<\/span><\/li>\n<\/ul>\n
\n- Failure Mode:<\/b> This method suffers from catastrophic <\/span>“Head-of-Line Blocking”<\/b>.<\/span>
| | |
![\"\"](\"https:\/\/uplatz.com\/blog\/wp-content\/uploads\/2025\/11\/LLM-Serving-Architecture-1024x576.jpg\")
<\/p>\n