bundle-combo—sap-core-hcm-hcm-and-successfactors-ec By Uplatz<\/a><\/h3>\nFirst, the <\/span>memory constraint<\/b> is the most immediate barrier. A single accelerator device has a finite amount of high-bandwidth memory (VRAM), which must accommodate not only the model’s parameters but also the optimizer states, gradients, and intermediate activations generated during training.<\/span>4<\/span> For a model like GPT-3, the parameters alone, stored in 16-bit floating-point format, would require 350 GB of memory, far exceeding the capacity of any single commercially available GPU.<\/span>5<\/span> Widely used optimizers like ADAM further exacerbate this issue by storing additional momentum and variance terms for each parameter, effectively tripling the memory required for the model state.<\/span>9<\/span><\/p>\nSecond, even if a model could fit into a single device’s memory, the <\/span>compute constraint<\/b> would render its training impractical. The sheer volume of floating-point operations (FLOPs) required to train such models on a single device would lead to infeasibly long training times, potentially spanning years.<\/span>3<\/span> Distributing the computational workload is therefore not merely an optimization but a necessity for completing training within a reasonable timeframe.<\/span><\/p>\nThe progression of deep learning models has thus outpaced the growth of single-device memory and compute capabilities. This has necessitated a paradigm shift from single-device training to distributed training, where multiple accelerators work in concert. The limitations of one paradigm, such as the inability to handle models larger than a single device’s memory, have directly spurred the development of more sophisticated approaches. This evolution reflects a direct causal relationship: as model architectures have grown in response to the demand for higher performance, they have consistently pushed against hardware limits, driving the innovation of new parallelism strategies designed to overcome these specific bottlenecks.<\/span><\/p>\n <\/p>\n
1.2 A Taxonomy of Distributed Training Strategies<\/b><\/h3>\n
<\/p>\n
To address the challenges of scale, the deep learning community has developed a set of canonical parallelism strategies, often referred to collectively as “3D Parallelism”.<\/span>6<\/span> These strategies\u2014Data Parallelism, Pipeline Parallelism, and Tensor Parallelism\u2014offer distinct methods for partitioning the training workload across a cluster of devices. Understanding the mechanics, advantages, and limitations of each is crucial for contextualizing the unique role of tensor parallelism.<\/span><\/p>\n <\/p>\n
1.2.1 Data Parallelism (DP): Replicating the Model<\/b><\/h4>\n
<\/p>\n
Data Parallelism is the most common and conceptually simplest approach to distributed training.<\/span>3<\/span><\/p>\n\n- Mechanism:<\/b> In a data-parallel setup, the entire model is replicated on each participating accelerator device. The global data batch is then divided into smaller “micro-batches,” and each device processes one micro-batch in parallel.<\/span>3<\/span> During the forward pass, each model replica computes its output independently. During the backward pass, each replica computes gradients based on its local micro-batch.<\/span><\/li>\n
- Communication:<\/b> To ensure that the model replicas do not diverge, their parameters must be kept synchronized. This is achieved by performing a collective communication operation, typically an All-Reduce, on the gradients computed by each device at the end of the backward pass. This operation sums the gradients from all devices and distributes the result back to each one, ensuring that every model replica performs an identical weight update.<\/span>3<\/span><\/li>\n
- Limitation:<\/b> The primary and defining limitation of data parallelism is its memory requirement. Since a full copy of the model, its gradients, and its optimizer states must reside on each device, this strategy is only viable for models that can fit within the memory of a single accelerator.<\/span>6<\/span> When a model’s size exceeds this threshold, data parallelism is no longer a feasible option.<\/span><\/li>\n<\/ul>\n
<\/p>\n
1.2.2 Pipeline Parallelism (PP): Vertical Model Splitting (Inter-Layer)<\/b><\/h4>\n
<\/p>\n
Pipeline Parallelism is a form of model parallelism that addresses the memory limitations of data parallelism by partitioning the model itself. It is characterized as an <\/span>inter-layer<\/span><\/i> or “vertical” splitting strategy.<\/span>3<\/span><\/p>\n\n- Mechanism:<\/b> The model’s sequence of layers is divided into contiguous blocks, known as “stages.” Each stage is then assigned to a different device.<\/span>6<\/span> During training, data flows through these stages sequentially: the output activations of the layers on device 1 are passed as input to the layers on device 2, and so on, until the final output is produced.<\/span>3<\/span> The backward pass follows the reverse path, with gradients being passed from later stages to earlier ones.<\/span><\/li>\n
- Communication:<\/b> Communication in pipeline parallelism is typically limited to point-to-point transfers between adjacent devices in the pipeline. Each stage sends its output activations forward and receives input gradients from the subsequent stage.<\/span>12<\/span><\/li>\n
- Limitation:<\/b> The sequential nature of pipeline parallelism introduces a significant inefficiency known as the “pipeline bubble”.<\/span>6<\/span> At the beginning of processing a batch, only the first device is active. As the first micro-batch moves to the second stage, the first device can start on the next micro-batch, but the last device remains idle until the data propagates all the way through the pipeline. A similar “ramp-down” phase occurs at the end of the batch. This idle time reduces overall hardware utilization. Modern implementations mitigate this by processing many micro-batches concurrently to keep the pipeline as full as possible, but some degree of inefficiency is inherent to the approach.<\/span><\/li>\n<\/ul>\n
<\/p>\n
1.2.3 Tensor Parallelism (TP): Horizontal Model Splitting (Intra-Layer)<\/b><\/h4>\n
<\/p>\n
Tensor Parallelism is another form of model parallelism, but it operates at a much finer granularity than pipeline parallelism. It is an <\/span>intra-layer<\/span><\/i> or “horizontal” splitting strategy, meaning it partitions the computations <\/span>within<\/span><\/i> a single layer.<\/span>3<\/span><\/p>\n\n- Mechanism:<\/b> Instead of assigning whole layers to different devices, tensor parallelism splits the large tensors that constitute a layer\u2014primarily the weight matrices\u2014into shards. Each device in a tensor-parallel group holds only a fraction of the layer’s weights and performs computations on its respective shard.<\/span>5<\/span><\/li>\n
- Key Differentiator:<\/b> The fundamental difference from pipeline parallelism is its ability to parallelize a single, massive layer that is too large to fit on one device.<\/span>13<\/span> While pipeline parallelism can distribute a deep model, it cannot help if a single “wide” layer is the memory bottleneck. Tensor parallelism directly solves this problem by splitting the layer itself.<\/span><\/li>\n
- Initial Trade-off:<\/b> Tensor parallelism elegantly avoids the pipeline bubble problem, as all devices in the group work concurrently on the same data batch for a given operation.<\/span>13<\/span> However, this concurrency comes at the cost of requiring frequent and high-bandwidth communication <\/span>within<\/span><\/i> the forward and backward passes of each parallelized layer to synchronize the partial results.<\/span>10<\/span> This trade-off between utilization and communication overhead is a central theme in the design and application of tensor parallelism.<\/span><\/li>\n<\/ul>\n
<\/p>\n
Section 2: The Mathematical Foundations of Tensor Parallelism<\/b><\/h2>\n
<\/p>\n
At its core, tensor parallelism is an application of fundamental linear algebra principles to distribute computation. To understand how it works, one must first deconstruct the primary operation within modern neural networks\u2014matrix multiplication\u2014and then explore how this operation can be mathematically partitioned and subsequently reassembled using collective communication.<\/span><\/p>\n <\/p>\n
2.1 Deconstructing the Linear Layer: The Primacy of Matrix Multiplication<\/b><\/h3>\n
<\/p>\n
The vast majority of parameters and computations in large-scale models like Transformers are concentrated in their linear layers (also known as fully connected or dense layers).<\/span>8<\/span> A linear layer performs an affine transformation on its input, which can be expressed as the matrix equation:<\/span><\/p>\n$$Y = XA + b$$<\/span><\/p>\nHere, $X$ is the input activation tensor, $A$ is the weight matrix, $b$ is the bias vector, and $Y$ is the output tensor. The dominant computational cost of this operation is the matrix multiplication $XA$. Therefore, the problem of parallelizing an entire layer effectively reduces to the problem of parallelizing this matrix multiplication.<\/span>3<\/span><\/p>\n <\/p>\n
2.2 Sharding Strategies for Matrix Multiplication<\/b><\/h3>\n
<\/p>\n
The definition of matrix multiplication provides a natural basis for partitioning the computation. For a matrix multiplication $C = AB$, each element $C_{i,j}$ is the dot product of the $i$-th row of $A$ and the $j$-th column of $B$.<\/span>16<\/span> This property allows for two primary sharding strategies.<\/span><\/p>\n <\/p>\n
2.2.1 Column-Wise Parallelism (Splitting the Output Dimension)<\/b><\/h4>\n
<\/p>\n
In column-wise parallelism, the weight matrix is partitioned along its column dimension. Consider a weight matrix $A$ and an input matrix $X$. If we split $A$ into two column blocks, $A = [A_1 | A_2]$, the matrix multiplication can be written as:<\/span><\/p>\n$$Y = XA = X[A_1 | A_2] = [XA_1 | XA_2]$$<\/span><\/p>\nThis formulation reveals a path to parallelization. The input matrix $X$ can be broadcast to two separate devices. Device 1 computes the partial result $Y_1 = XA_1$, while Device 2 computes $Y_2 = XA_2$.<\/span>10<\/span> These computations can occur simultaneously. The final output $Y$ is then obtained by concatenating the partial results along the column dimension. In this scheme, the input tensor $X$ is replicated across devices, but the weight matrix $A$ and the output tensor $Y$ are sharded (or “split”).<\/span>13<\/span><\/p>\n <\/p>\n
2.2.2 Row-Wise Parallelism (Splitting the Input Dimension)<\/b><\/h4>\n
<\/p>\n
In row-wise parallelism, the weight matrix is partitioned along its row dimension. If we split $A$ into two row blocks, $A = \\begin{pmatrix} A_1 \\\\ A_2 \\end{pmatrix}$, the multiplication $XA$ requires a corresponding split of the input matrix $X$ along its column dimension, $X = [X_1 | X_2]$. The multiplication then becomes:<\/span><\/p>\n$$Y = XA = [X_1 | X_2] \\begin{pmatrix} A_1 \\\\ A_2 \\end{pmatrix} = X_1A_1 + X_2A_2$$<\/span><\/p>\nThis decomposition also lends itself to parallel execution. Device 1, holding the input shard $X_1$ and weight shard $A_1$, computes the partial result $Y_1 = X_1A_1$. Concurrently, Device 2, holding $X_2$ and $A_2$, computes $Y_2 = X_2A_2$.<\/span>10<\/span> The final output $Y$ is obtained by performing an element-wise sum of the partial results. In this case, the input $X$ and weight matrix $A$ are both sharded, while the final output $Y$ is a replicated (or “full”) tensor after the summation.<\/span>8<\/span><\/p>\n <\/p>\n
2.3 The Role of Collective Communication Primitives<\/b><\/h3>\n
<\/p>\n
Partitioning the computation is only one part of the process. To ensure the mathematical correctness of the final result and to make that result available for subsequent layers, devices must communicate and synchronize their partial results. This is accomplished using highly optimized collective communication primitives, which are fundamental operations in parallel computing libraries like MPI and NVIDIA’s NCCL.<\/span>10<\/span><\/p>\n <\/p>\n
2.3.1 All-Gather: Reconstructing Tensors from Shards<\/b><\/h4>\n
<\/p>\n
The All-Gather operation is used to collect tensor shards from a group of devices and make the complete, concatenated tensor available on every device in that group.<\/span>6<\/span><\/p>\n\n- Function:<\/b> If Device 1 holds tensor $T_1$ and Device 2 holds tensor $T_2$, an All-Gather operation results in both devices holding the concatenated tensor $$.<\/span><\/li>\n
- Use Case:<\/b> This primitive is the natural communication pattern for finalizing a column-wise parallel operation. After each device computes its partial output shard ($Y_1$ and $Y_2$), an All-Gather is performed to reconstruct the full output tensor $Y =$ on all participating devices, making it ready for the next layer.<\/span>10<\/span><\/li>\n<\/ul>\n
<\/p>\n
2.3.2 All-Reduce: Aggregating Partial Results<\/b><\/h4>\n
<\/p>\n
The All-Reduce operation collects input tensors from all devices, applies a specified reduction operation (most commonly, summation), and distributes the final, reduced result back to all devices.<\/span>7<\/span><\/p>\n\n- Function:<\/b> If Device 1 holds tensor $T_1$ and Device 2 holds tensor $T_2$, an All-Reduce with a sum operation results in both devices holding the tensor $T_1 + T_2$.<\/span><\/li>\n
- Use Case:<\/b> This primitive is essential for completing a row-wise parallel operation. After each device computes its partial output ($Y_1$ and $Y_2$), an All-Reduce sums these partial results to produce the final, correct output $Y = Y_1 + Y_2$ on all devices.<\/span>10<\/span><\/li>\n<\/ul>\n
The choice between these sharding strategies and their corresponding communication primitives is not arbitrary. It is a deliberate design decision driven by the need to manage the “sharding state” of the activation tensors as they flow through the network. A column-parallel layer transforms a replicated input tensor into a sharded output tensor. Conversely, a row-parallel layer is designed to accept a sharded input tensor and produce a replicated output tensor. This predictable transformation, Replicated -> ColumnParallel -> Sharded -> RowParallel -> Replicated, forms the cornerstone of efficient tensor-parallel model design. It allows for the chaining of parallel layers in a way that minimizes communication by ensuring the output sharding of one layer perfectly matches the required input sharding of the next. This principle is what enables frameworks to abstract away the complexity, but understanding it is vital for performance optimization.<\/span><\/p>\n <\/p>\n
2.4 The Tensor Parallel Forward and Backward Pass: A Step-by-Step Walkthrough<\/b><\/h3>\n
<\/p>\n
To solidify these concepts, consider a simple two-layer Multi-Layer Perceptron (MLP) with an activation function $f$: $Z = f(XA)B$. This model can be parallelized across two devices using a combination of column-wise and row-wise parallelism.<\/span>8<\/span><\/p>\n <\/p>\n
Forward Pass<\/b><\/h4>\n
<\/p>\n
\n- Partitioning:<\/b> The first weight matrix, $A$, is split <\/span>column-wise<\/b> into $A = [A_1 | A_2]$. The second weight matrix, $B$, is split <\/span>row-wise<\/b> into $B = \\begin{pmatrix} B_1 \\\\ B_2 \\end{pmatrix}$.<\/span><\/li>\n
- Device 1 Computation:<\/b><\/li>\n<\/ol>\n
\n- Receives the full input $X$.<\/span><\/li>\n
- Computes the first partial activation: $Y_1 = f(XA_1)$.<\/span><\/li>\n
- Computes the first partial output: $Z_1 = Y_1B_1$.<\/span><\/li>\n<\/ul>\n
\n- Device 2 Computation:<\/b><\/li>\n<\/ol>\n
\n- Receives the full input $X$.<\/span><\/li>\n
- Computes the second partial activation: $Y_2 = f(XA_2)$.<\/span><\/li>\n
- Computes the second partial output: $Z_2 = Y_2B_2$.<\/span><\/li>\n<\/ul>\n
\n- Communication:<\/b> An All-Reduce operation is performed to sum the partial outputs. Both devices now hold the final, correct output: $Z = Z_1 + Z_2$.<\/span><\/li>\n<\/ol>\n
This sequence can be viewed as two functions, $g$ and $h$, applied to the input $X$. The forward pass for $g(X) = X$ is an identity operation (broadcasting $X$ to all devices). The forward pass for $h(Z) = \\text{AllReduce}(Z)$ aggregates the final result.<\/span><\/p>\n <\/p>\n
Backward Pass<\/b><\/h4>\n
<\/p>\n
The backward pass involves computing the gradients of a loss function $L$ with respect to the parameters $A$ and $B$, and with respect to the input $X$ (to be passed to the previous layer).<\/span><\/p>\n\n- Gradient w.r.t. Z:<\/b> The gradient $\\frac{\\partial L}{\\partial Z}$ is computed. Since $Z = Z_1 + Z_2$, the chain rule implies that the gradients with respect to the partial outputs are equal to the global gradient: $\\frac{\\partial L}{\\partial Z_1} = \\frac{\\partial L}{\\partial Z_2} = \\frac{\\partial L}{\\partial Z}$. This global gradient is available on all devices after the forward pass’s All-Reduce and is used to start the backward pass in parallel.<\/span>8<\/span> This corresponds to an identity operation for the backward pass of function $h$.<\/span><\/li>\n
- Gradients w.r.t. B (Parallel):<\/b><\/li>\n<\/ol>\n
\n- Device 1 computes $\\frac{\\partial L}{\\partial B_1} = Y_1^T \\frac{\\partial L}{\\partial Z_1}$.<\/span><\/li>\n
- Device 2 computes $\\frac{\\partial L}{\\partial B_2} = Y_2^T \\frac{\\partial L}{\\partial Z_2}$.<\/span><\/li>\n<\/ul>\n
\n- Gradients w.r.t. Y (Parallel):<\/b><\/li>\n<\/ol>\n
\n- Device 1 computes $\\frac{\\partial L}{\\partial Y_1} = \\frac{\\partial L}{\\partial Z_1} B_1^T$.<\/span><\/li>\n
- Device 2 computes $\\frac{\\partial L}{\\partial Y_2} = \\frac{\\partial L}{\\partial Z_2} B_2^T$.<\/span><\/li>\n<\/ul>\n
\n- Gradients w.r.t. A (Parallel):<\/b> The gradients are propagated through the activation function $f’$.<\/span><\/li>\n<\/ol>\n
\n- Device 1 computes $\\frac{\\partial L}{\\partial A_1} = X^T ( \\frac{\\partial L}{\\partial Y_1} \\circ f'(XA_1) )$.<\/span><\/li>\n
- Device 2 computes $\\frac{\\partial L}{\\partial A_2} = X^T ( \\frac{\\partial L}{\\partial Y_2} \\circ f'(XA_2) )$.<\/span><\/li>\n<\/ul>\n
\n- Gradient w.r.t. X (Communication):<\/b> To compute the gradient with respect to the original input $X$, the partial gradients must be summed.<\/span><\/li>\n<\/ol>\n
\n- Device 1 computes its partial gradient: $(\\frac{\\partial L}{\\partial X})_1 = (\\frac{\\partial L}{\\partial Y_1} \\circ f'(XA_1)) A_1^T$.<\/span><\/li>\n
- Device 2 computes its partial gradient: $(\\frac{\\partial L}{\\partial X})_2 = (\\frac{\\partial L}{\\partial Y_2} \\circ f'(XA_2)) A_2^T$.<\/span><\/li>\n
- An All-Reduce operation is performed to get the final gradient: $\\frac{\\partial L}{\\partial X} = (\\frac{\\partial L}{\\partial X})_1 + (\\frac{\\partial L}{\\partial X})_2$. This corresponds to the backward pass of function $g$.<\/span><\/li>\n<\/ul>\n
In summary, for this two-layer block, the forward pass requires one All-Reduce at the end, and the backward pass requires one All-Reduce at the beginning (for the gradient w.r.t. the input). This efficient communication pattern is key to the performance of tensor parallelism.<\/span><\/p>\n <\/p>\n
Section 3: Applying Tensor Parallelism to Transformer Architectures<\/b><\/h2>\n
<\/p>\n
The theoretical principles of sharding matrix multiplications are not merely academic; they form the practical basis for distributing the most computationally intensive components of the Transformer architecture. The design of tensor-parallel Transformers, pioneered by frameworks like Megatron-LM, reveals an elegant and recurring pattern that efficiently parallelizes both the Feed-Forward Network and the Multi-Head Attention mechanism.<\/span><\/p>\n <\/p>\n
3.1 Anatomy of a Transformer Block<\/b><\/h3>\n
<\/p>\n
A standard Transformer block is composed of two main sub-layers: a Multi-Head Self-Attention (MHSA) module and a position-wise Feed-Forward Network (FFN). Both sub-layers are followed by a residual connection and a layer normalization step.<\/span>15<\/span> The FFN and MHSA modules are where the vast majority of the model’s parameters and computations reside, making them the primary targets for tensor parallelism.<\/span><\/p>\n <\/p>\n
3.2 Parallelizing the Feed-Forward Network (FFN)<\/b><\/h3>\n
<\/p>\n
3.2.1 The Megatron-LM Approach: A Column-Parallel and Row-Parallel Pair<\/b><\/h4>\n
<\/p>\n
The FFN in a Transformer typically consists of two linear layers with a non-linear activation function, such as GeLU, in between.<\/span>16<\/span> The transformation can be expressed as:<\/span><\/p>\n$$Y_{FFN} = \\text{GeLU}(X A) B$$<\/span><\/p>\n
![\"\"](\"https:\/\/uplatz.com\/blog\/wp-content\/uploads\/2025\/10\/The-Mechanics-of-Tensor-Parallelism-A-Deep-Dive-into-Intra-Layer-Model-Distribution-1024x576.jpg\")
<\/p>\n