bundle-combo-sap-mm-ecc-and-s4hana By Uplatz<\/a><\/h3>\n2. Foundational Pillars: Deconstructing Tokens and the Context Window<\/b><\/h2>\n
To analyze the architectural constraints of LLMs, it is first necessary to establish a precise technical vocabulary for the “units” of measurement (tokens) and the “boundary” of operation (the context window).<\/span><\/p>\n2.1. From Text to Tensors: The Tokenization Process<\/b><\/h3>\n
An LLM does not process raw text. It does not see words, characters, or sentences in the way a human does. Instead, it operates on <\/span>tokens<\/b>.<\/span>5<\/span> Tokenization is the foundational preprocessing step that bridges the gap between human language and the mathematical-vector space of the model.<\/span>15<\/span> The process involves breaking down a string of text into a sequence of these tokens, which are then converted into numerical IDs and, subsequently, into high-dimensional vectors known as embeddings.<\/span>15<\/span><\/p>\nThis token-based approach is a carefully engineered compromise. Early strategies presented a false dichotomy:<\/span><\/p>\n\n- Character-level tokenization:<\/b> This splits text into individual characters (e.g., ‘h’, ‘e’, ‘l’, ‘l’, ‘o’). While this creates a very small, fixed vocabulary (e.g., ~256 for ASCII), it results in extremely long token sequences, which massively increases the computational load.<\/span>18<\/span><\/li>\n
- Word-level tokenization:<\/b> This splits text by spaces (e.g., ‘hello’). This creates shorter sequences but requires a massive vocabulary (e.g., >1,000,000 words), which increases model size and memory usage. It also fails to handle typos, novel words, or complex syntax\u2014the “out-of-vocabulary” (OOV) problem.<\/span>18<\/span><\/li>\n<\/ul>\n
The industry solved this with <\/span>subword tokenization algorithms<\/b> like Byte-Pair Encoding (BPE), WordPiece, and SentencePiece.<\/span>15<\/span> These algorithms learn to break text into statistically common fragments. A common word like “hello” might be a single token, while a rarer word like “tokenization” might be split into “token” and “##ization”.<\/span>20<\/span> This approach provides a “best-of-both-worlds” solution: a manageable vocabulary size (e.g., 30k-100k) and the ability to represent any arbitrary text string by falling back to its constituent parts.<\/span><\/p>\nA common “rule of thumb,” particularly for English, is that 1 token is approximately 4 characters or \u00be of a word.<\/span>13<\/span> Therefore, 100 tokens equates to roughly 75 words.<\/span>13<\/span> However, this is a fragile approximation. For example, the quote “You miss 100% of the shots you don’t take” is 11 tokens.<\/span>13<\/span> More importantly, this ratio varies dramatically by language. The Spanish phrase “C\u00f3mo est\u00e1s” (10 characters) is 5 tokens.<\/span>13<\/span><\/p>\nThis variability in tokenization is not a neutral technical detail; it has profound second-order consequences for cost and equity. API billing for models like Google’s Gemini <\/span>22<\/span> and OpenAI’s GPT series <\/span>13<\/span> is calculated <\/span>per-token<\/span><\/i>. Likewise, the context window limit is a <\/span>token<\/span><\/i> limit.<\/span>13<\/span> The fact that non-English text consistently produces a higher token-to-character ratio <\/span>13<\/span> means that it is <\/span>both<\/span><\/i> more expensive to process and that <\/span>less<\/span><\/i> information (in terms of human-readable text) can fit into the same-sized context window. This creates a direct financial and performance penalty for using LLMs in languages other than English, a critical consideration for global enterprise deployments.<\/span><\/p>\n <\/p>\n
2.2. The Context Window: An LLM’s Finite Working Memory<\/b><\/h3>\n
<\/p>\n
The context window, also referred to as context length, is the maximum amount of information\u2014measured in tokens\u2014that an LLM can process in a single, discrete operation.<\/span>2<\/span> This limit dictates the total number of tokens, combining both the user’s input and the model’s generated output, that the model can “see” or “remember” during one inference step.<\/span>24<\/span><\/p>\nThis boundary is often analogized to a human’s “short-term memory” <\/span>4<\/span> or a “notepad” that the model uses during a conversation.<\/span>2<\/span> If a prompt, document, or conversation exceeds this limit, the model must truncate or summarize the input, and older information is discarded or “forgotten”.<\/span>2<\/span> This hard boundary is a defining design constraint of the Transformer architecture.<\/span>2<\/span><\/p>\nIt is crucial to distinguish between the <\/span>total context window<\/span><\/i> and the <\/span>maximum output token limit<\/span><\/i>. For example, the GPT-4o model has a 128,000-token total context window <\/span>27<\/span>, but its counterpart, GPT-4o-mini, has a maximum output limit of 16,384 tokens.<\/span>28<\/span> This means the model can <\/span>process<\/span><\/i> a large amount of input (e.g., 112,000 tokens) but can only <\/span>generate<\/span><\/i> a response up to its output limit (16,384 tokens).<\/span>29<\/span> The total context window is the “total workspace,” while the max output limit is the “maximum response size.”<\/span><\/p>\nThe “short-term memory” analogy, while useful, is also dangerously misleading. In truth, LLMs do not “remember” past interactions at all.<\/span>9<\/span> The perceived “conversational memory” of a chatbot is an expensive illusion. As <\/span>31<\/span> explains, “The entire conversational history is forwarded to the LLM on every query, until you exceed the context window size.”<\/span><\/p>\nThis reveals a profound operational reality: LLMs are fundamentally <\/span>stateless<\/span><\/i>. A chatbot’s “memory” is not a persistent, learned state that is efficiently updated. It is a brute-force, computationally expensive operation where the <\/span>entire<\/span><\/i> chat history is re-tokenized, re-embedded, and re-processed <\/span>from scratch<\/span><\/i> for <\/span>every single user response<\/span><\/i>. The “forgetting” that occurs when the context is truncated is not a human-like memory lapse; it is a hard architectural boundary being met. This stateless reprocessing is a primary driver of high inference costs and latency in any multi-turn dialogue application.<\/span><\/p>\n <\/p>\n
3. The Engine of Cognition: The “Attention Span” as Self-Attention<\/b><\/h2>\n
<\/p>\n
The “attention span” of an LLM is not a measure of time but a reference to its core processing mechanism: <\/span>self-attention<\/b>. This is the “engine” that operates <\/span>within<\/span><\/i> the context window, consuming tokens to produce dynamically computed, contextualized understanding.<\/span><\/p>\n <\/p>\n
3.1. A Foundational Shift: “Attention Is All You Need” (Vaswani et al., 2017)<\/b><\/h3>\n
<\/p>\n
The self-attention mechanism was the central innovation of the 2017 landmark paper, “Attention Is All You Need”.<\/span>32<\/span> This paper introduced the <\/span>Transformer<\/b> architecture, which proposed a radical new model for sequence modeling. It <\/span>dispensed entirely<\/span><\/i> with the recurrence (RNNs) and convolutions that had previously dominated the field.<\/span>33<\/span><\/p>\nPrevious models like LSTMs and RNNs were sequential\u2014they had to process token 1 to process token 2, making them difficult to parallelize on modern GPU hardware.<\/span>35<\/span> The Transformer’s sole reliance on self-attention allowed it to process all tokens in a sequence simultaneously, enabling massive parallelization during training.<\/span>34<\/span> This mechanism also proved far more effective at capturing long-range dependencies between tokens (e.g., how the first word of a paragraph relates to the last) than its recurrent predecessors.<\/span>35<\/span><\/p>\n <\/p>\n
3.2. The Query-Key-Value (QKV) Mechanism Explained<\/b><\/h3>\n
<\/p>\n
Self-attention (or intra-attention) is an attention mechanism that relates “different positions of a single sequence in order to compute a representation of the sequence”.<\/span>35<\/span> It functions by generating three specific vectors for <\/span>every token<\/span><\/i> in the context window.<\/span>38<\/span> These vectors\u2014Query, Key, and Value\u2014are created by multiplying the token’s embedding by three separate, learned weight matrices (a process called linear transformation).<\/span>38<\/span><\/p>\nThese vectors can be understood through a database or retrieval analogy:<\/span><\/p>\n\n- Query (Q):<\/b> This vector represents a token’s “search request.” It is a question that the token “asks” of all other tokens, representing the information it is <\/span>seeking<\/span><\/i> to better understand its own role in the sequence.<\/span>40<\/span><\/li>\n
- Key (K):<\/b> This vector acts as a token’s “label” or “advertisement.” It represents the information the token <\/span>contains<\/span><\/i> and is used to be “found” by other tokens’ queries.<\/span>38<\/span><\/li>\n
- Value (V):<\/b> This vector is the token’s “payload” or “content.” It is the actual information the token will <\/span>share<\/span><\/i> with other tokens if its Key is matched by a Query.<\/span>38<\/span><\/li>\n<\/ul>\n
This QKV mechanism is how context is <\/span>dynamically computed<\/span><\/i>. Unlike older models with static embeddings, self-attention creates contextual embeddings. As <\/span>43<\/span> and <\/span>43<\/span> explain, the final vector for the token “light” in the phrase “light as a feather” will be different from its vector in “turn on the light.” This is because its Q vector will interact differently with the K vectors of “feather” versus “turn,” resulting in a different weighted sum of V vectors.<\/span><\/p>\n <\/p>\n
3.3. Calculating Context: Scaled Dot-Product Attention<\/b><\/h3>\n
<\/p>\n
The Transformer computes attention using a specific formula: Scaled Dot-Product Attention. The equation is:<\/span><\/p>\n$$\\text{Attention}(Q, K, V) = \\text{softmax}\\left(\\frac{QK^T}{\\sqrt{d_k}}\\right)V$$<\/span><\/p>\n44<\/span><\/p>\nThis calculation proceeds in four distinct steps:<\/span><\/p>\n\n- Step 1: Compute Attention Scores ($QK^T$).<\/b> The model multiplies the Query matrix ($Q$, with shape $n \\times d_k$, where $n$ is sequence length and $d_k$ is key dimension) by the transpose of the Key matrix ($K^T$, with shape $d_k \\times n$).<\/span>46<\/span> This operation is the <\/span>source of the quadratic bottleneck<\/span><\/i>. It produces a massive $n \\times n$ “attention score matrix”.<\/span>46<\/span> Each entry $(i, j)$ in this matrix is the dot product of Query <\/span>i<\/span><\/i> and Key <\/span>j<\/span><\/i>, representing their “relevance” or “compatibility”.<\/span>37<\/span><\/li>\n
- Step 2: Scale ( \/ $\\sqrt{d_k}$).<\/b> The entire $n \\times n$ matrix is then scaled by dividing every entry by $\\sqrt{d_k}$, the square root of the key dimension.<\/span>38<\/span> This is a crucial, non-obvious step. As <\/span>48<\/span>
![\"\"](\"https:\/\/uplatz.com\/blog\/wp-content\/uploads\/2025\/11\/Deconstructing-the-Transformers-Bottleneck-An-Analysis-of-Context-Attention-and-Tokens-1024x576.jpg\"){kind=spacer}
<\/p>\n