https:\/\/uplatz.com\/course-details\/bundle-combo-sap-core-hcm-hcm-and-successfactors-ec\/439<\/a><\/p>\n1.1 The Dominance and Deficiencies of Recurrent Models<\/b><\/h3>\n
The core operational constraint of RNNs is their “recurrence.” They are inherently sequential, processing sequences one token at a time, from left to right.<\/span>4<\/span> This operation, mathematically described as $h_t = f(h_{t-1}, x_t)$, creates a computational dependency where the calculation for the current time step $t$ cannot begin until the calculation for time step $t-1$ is complete. This sequential nature “precluded parallelization” within a training example, making them slow to train.<\/span>3<\/span><\/p>\nFurthermore, while architectures like LSTM were explicitly designed to mitigate the <\/span>vanishing gradient<\/span><\/i> problem of simple RNNs <\/span>2<\/span>, they still struggled to capture dependencies over “very long-range” sequences.<\/span>5<\/span> Information must be propagated sequentially through the network’s state, and even with gating mechanisms, context can be lost. This combination of non-parallelizability and computational complexity made training state-of-the-art models on massive datasets “computationally expensive” and time-consuming.<\/span>2<\/span><\/p>\nThis architectural limitation represented a fundamental mismatch with the available hardware. The deep learning field was, and is, reliant on Graphics Processing Units (GPUs), which excel at massive parallel computation.<\/span>6<\/span> The sequential nature of RNNs was a hardware\/software mismatch, creating a scalability bottleneck.<\/span><\/p>\n <\/p>\n
1.2 The “Attention Is All You Need” Intervention<\/b><\/h3>\n
<\/p>\n
In 2017, a landmark paper from Google researchers, “Attention Is All You Need,” introduced a “new simple network architecture” that proposed a radical solution.<\/span>3<\/span> This paper, now considered a “watershed moment” in deep learning <\/span>8<\/span>, proposed to solve the sequential bottleneck by “dispensing with recurrence and convolutions entirely”.<\/span>3<\/span><\/p>\nThe proposed architecture, named the “Transformer,” was “based <\/span>solely<\/span><\/i> on attention mechanisms”.<\/span>3<\/span> The primary advantage, as stated by the authors, was that it was “more parallelizable” and required “significantly less time to train”.<\/span>3<\/span> The model validated these claims immediately, achieving a new state-of-the-art (SOTA) BLEU score of 28.4 on the WMT 2014 English-to-German translation task, and 41.0 on the English-to-French task, in a “small fraction of the training costs” of the best models at the time.<\/span>3<\/span><\/p>\nBy removing recurrence, the authors solved the $O(n)$ <\/span>sequential<\/span><\/i> bottleneck. However, this design introduced a new, fundamental trade-off: the $O(n^2)$ <\/span>parallel<\/span><\/i> bottleneck.<\/span>1<\/span> The self-attention mechanism, which connects all tokens to all other tokens, has a computational and memory complexity that is quadratic with respect to the sequence length $n$.<\/span>11<\/span> At the time of publication, this was a brilliant and practical compromise. For tasks like machine translation, sequence lengths $n$ were often “smaller than the representation dimensionality d”.<\/span>1<\/span> In that specific regime, self-attention was computationally “faster than recurrent layers”.<\/span>1<\/span> This single trade-off\u2014swapping a sequential dependency for a quadratic parallel one\u2014defined the Transformer and would become the central challenge for the entire field in the years to come.<\/span><\/p>\n <\/p>\n
Table 1: Comparative Analysis: Recurrent vs. Transformer Architectures<\/b><\/h3>\n
<\/p>\n
\n\n\n| Feature<\/b><\/td>\n | Recurrent Neural Network (RNN)<\/b><\/td>\n | Long Short-Term Memory (LSTM)<\/b><\/td>\n | Transformer (Self-Attention)<\/b><\/td>\n<\/tr>\n\n| Core Operation<\/b><\/td>\n | Sequential\/Recurrent<\/span><\/td>\n | Sequential\/Recurrent (Gated)<\/span><\/td>\n | Parallel\/Self-Attention<\/span><\/td>\n<\/tr>\n\n| Parallelization (Intra-sequence)<\/b><\/td>\n | None <\/span>4<\/span><\/td>\n | None <\/span>4<\/span><\/td>\n | High [3, 5]<\/span><\/td>\n<\/tr>\n\n| Primary Computational Complexity<\/b><\/td>\n | $O(n \\cdot d^2)$<\/span><\/td>\n | $O(n \\cdot d^2)$<\/span><\/td>\n | $O(n^2 \\cdot d)$ <\/span>1<\/span><\/td>\n<\/tr>\n\n| Path Length for Long-Range Dependencies<\/b><\/td>\n | $O(n)$<\/span><\/td>\n | $O(n)$<\/span><\/td>\n | $O(1)$ <\/span>1<\/span><\/td>\n<\/tr>\n\n| Primary Limitation<\/b><\/td>\n | Sequential bottleneck; Vanishing gradients [4]<\/span><\/td>\n | Sequential bottleneck; Computationally expensive [2, 7]<\/span><\/td>\n | Quadratic $O(n^2)$ memory and compute bottleneck <\/span>11<\/span><\/td>\n<\/tr>\n\n| (n = sequence length, d = representation dimension)<\/span><\/i><\/td>\n | <\/td>\n | <\/td>\n | <\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n <\/p>\n 2.0 Anatomy of the Canonical Transformer: The Encoder-Decoder Framework<\/b><\/h2>\n <\/p>\n In its original form, the Transformer is an <\/span>Encoder-Decoder architecture<\/b>.<\/span>12<\/span> It was presented as a sequence-to-sequence model for machine translation, taking a sentence in one language and outputting its translation in another.<\/span>12<\/span><\/p>\n <\/p>\n 2.1 High-Level Architecture<\/b><\/h3>\n <\/p>\n The architecture consists of two primary components: an <\/span>Encoder Stack<\/b> and a <\/span>Decoder Stack<\/b>, with explicit connections between them.<\/span>12<\/span> The original paper used a stack of <\/span>N=6 identical layers<\/b> for both the encoder and decoder components.<\/span>12<\/span> This design creates a clear separation of labor: the Encoder’s responsibility is to <\/span>understand<\/span><\/i> the input sequence, while the Decoder’s responsibility is to <\/span>generate<\/span><\/i> the output sequence, guided by the Encoder’s understanding.<\/span>1<\/span><\/p>\n <\/p>\n 2.2 The Encoder Stack<\/b><\/h3>\n <\/p>\n The function of the encoder stack is to ingest the source sequence and produce a rich, contextualized vector representation for each token. The input sequence (e.g., text tokens) is first passed through an embedding layer and combined with positional encodings to inject information about word order.<\/span>13<\/span><\/p>\nEach of the N=6 layers in the stack is identical and composed of two sequential sub-layers <\/span>13<\/span>:<\/span><\/p>\n\n- Sub-layer 1: Multi-Head Self-Attention:<\/b> This mechanism allows every token in the input sequence to look at and incorporate information from every <\/span>other<\/span><\/i> token in the <\/span>input<\/span><\/i> sequence (all-to-all attention).<\/span>1<\/span><\/li>\n
- Sub-layer 2: Position-wise Feed-Forward Network (FFN):<\/b> A simple, fully connected neural network that processes each token’s representation independently and identically.<\/span>13<\/span><\/li>\n<\/ol>\n
To enable the training of such a deep network, a <\/span>residual connection<\/b> (“Add”) followed by <\/span>layer normalization<\/b> (“Norm”) is applied around each of the two sub-layers.<\/span>13<\/span> The output of a sub-layer is thus: $LayerNorm(x + Sublayer(x))$.<\/span><\/p>\n <\/p>\n 2.3 The Decoder Stack<\/b><\/h3>\n <\/p>\n The function of the decoder stack is to generate the target sequence token by token, in an <\/span>autoregressive<\/b> manner.<\/span>14<\/span> At each step, it takes the target tokens generated <\/span>so far<\/span><\/i> as input (which are also embedded and combined with positional encodings).<\/span>13<\/span><\/p>\nEach of the N=6 decoder layers is composed of <\/span>three<\/span><\/i> sub-layers <\/span>13<\/span>:<\/span><\/p>\n\n- Sub-layer 1: Masked Multi-Head Self-Attention:<\/b> This allows each position in the <\/span>decoder<\/span><\/i> to attend to all positions in the decoder <\/span>up to and including that position<\/span><\/i>.<\/span>1<\/span> This “causal masking” is critical; it prevents the decoder from “cheating” by looking at future tokens (e.g., the word it is about to predict), thereby preserving the autoregressive, generative property.<\/span>14<\/span><\/li>\n
- Sub-layer 2: Encoder-Decoder Cross-Attention:<\/b> This is the critical connection point between the two stacks. In this layer, the <\/span>Queries (Q)<\/b> come from the previous decoder layer, while the <\/span>Keys (K) and Values (V) come from the output of the <\/b>entire<\/i><\/b> encoder stack<\/b>.<\/span>1<\/span> This mechanism “allows every position in the decoder to attend over all positions in the input sequence”.<\/span>1<\/span><\/li>\n
- Sub-layer 3: Position-wise Feed-Forward Network (FFN):<\/b> This is identical in structure to the FFN in the encoder layers.<\/span>13<\/span><\/li>\n<\/ol>\n
As in the encoder, residual connections and layer normalization are applied around each of these <\/span>three<\/span><\/i> sub-layers.<\/span>13<\/span><\/p>\nThis cross-attention mechanism represents a massive leap over previous seq2seq models. Older RNN-based models (pre-attention) suffered from a notorious “bottleneck problem” <\/span>6<\/span>, where they had to compress the <\/span>entire<\/span><\/i> meaning of the input sentence into a single, fixed-size context vector. The Transformer’s cross-attention, by contrast, provides the decoder with a <\/span>direct-access lookup<\/span><\/i> into the <\/span>entire<\/span><\/i> encoded input sequence\u2014with all its per-token representations\u2014at <\/span>every single<\/span><\/i> step of generation. This completely solves the fixed-size context bottleneck.<\/span>1<\/span><\/p>\n <\/p>\n 2.4 The Final Output Layer<\/b><\/h3>\n <\/p>\n The output from the top decoder layer, a sequence of vectors, is passed through a final <\/span>Linear layer<\/b> and a <\/span>Softmax function<\/b>.<\/span>16<\/span> This final stage functions as a classifier, generating a probability distribution over the entire target vocabulary. The token with the highest probability is typically selected as the next word in the generated sequence.<\/span>17<\/span><\/p>\n <\/p>\n 3.0 Core Mechanisms: A Component-Level Deconstruction<\/b><\/h2>\n <\/p>\n The Transformer’s functionality is enabled by several key components and mathematical operations.<\/span><\/p>\n <\/p>\n 3.1 The Necessity of Positional Encoding<\/b><\/h3>\n <\/p>\n The self-attention mechanism, which computes a weighted sum, is “permutation invariant”\u2014it treats the input as an unordered “bag” of vectors. By default, the model has no concept of word order.<\/span>19<\/span> This is a critical flaw, as language is order-dependent; the sentences “Allen walks dog” and “dog walks Allen” use identical tokens but have opposite meanings.<\/span>20<\/span><\/p>\nThe solution is <\/span>Positional Encodings<\/b> (PE). These are vectors, of the same dimension as the embeddings ($d_{model}$), that contain information about a token’s <\/span>position<\/span><\/i> in the sequence.<\/span>21<\/span> These PE vectors are <\/span>added<\/span><\/i> to the input (word embedding) vectors at the bottom of both the encoder and decoder stacks.<\/span>13<\/span><\/p>\nTwo primary methods exist for this:<\/span><\/p>\n\n- Sinusoidal (Original Paper):<\/b> Vaswani et al. proposed using fixed, non-learned sinusoidal functions.<\/span>20<\/span> The formulae are: $PE_{(pos, 2i)} = \\sin(pos \/ 10000^{2i\/d_{model}})$ and $PE_{(pos, 2i+1)} = \\cos(pos \/ 10000^{2i\/d_{model}})$.<\/span>20<\/span> The rationale was that the periodic nature of these waves might allow the model to “extrapolate to sequence lengths longer than the ones encountered during training”.<\/span>6<\/span><\/li>\n
| | | | | | |
![\"\"](\"https:\/\/uplatz.com\/blog\/wp-content\/uploads\/2025\/11\/Transformer-Architecture-Explained-1024x576.jpg\"){kind=spacer}
<\/p>\n