![\"\"](\"https:\/\/uplatz.com\/blog\/wp-content\/uploads\/2025\/10\/The-Zero-Redundancy-Optimizer-ZeRO-A-Definitive-Technical-Report-on-Memory-Efficient-Large-Scale-Distributed-Training-1024x576.jpg\")
<\/p>\n
bundle-combo—sap-ewm-ecc-and-s4hana By Uplatz<\/a><\/h3>\nBeyond its core partitioning strategy, the ZeRO ecosystem has evolved to incorporate advanced extensions that address subsequent bottlenecks. ZeRO-Offload and ZeRO-Infinity leverage heterogeneous memory systems, offloading model states to CPU RAM and Non-Volatile Memory Express (NVMe) drives to break through the physical GPU memory wall and train models with tens of trillions of parameters.<\/span>7<\/span> Concurrently, ZeRO++ introduces sophisticated communication optimizations, such as quantization and hierarchical partitioning, to reduce data transfer volume by up to 4x, mitigating the communication overhead that becomes dominant at extreme scales.<\/span>9<\/span><\/p>\nThe quantifiable impact of ZeRO is substantial. It has enabled the training of models with over 100 billion parameters, achieving up to a 10x increase in performance over previous state-of-the-art systems.<\/span>1<\/span> Its integration into the DeepSpeed library and subsequent adoption by high-level frameworks like Hugging Face Accelerate and PyTorch Lightning have democratized access to large-scale training, making it feasible for a broader community of researchers and developers.<\/span>7<\/span> Ultimately, ZeRO is not merely a technical tool but a foundational enabler of the current era of large language models (LLMs), providing the systems-level breakthrough necessary to translate theoretical model designs into tangible, state-of-the-art artifacts.<\/span><\/p>\n <\/p>\n
Section 2: The Challenge of Memory Redundancy in Distributed Training<\/b><\/h3>\n
<\/p>\n
The rapid escalation in the size and complexity of deep learning models has consistently outpaced the growth of hardware memory capacity. This disparity created a significant challenge for distributed training, where traditional methods proved incapable of efficiently utilizing the aggregate resources of a compute cluster. The core of this problem lay in the inherent memory redundancy of prevailing parallelization strategies, which established a “memory wall” that limited model scale based on the constraints of a single accelerator.<\/span><\/p>\n <\/p>\n
2.1 The Inefficiency of Traditional Data Parallelism (DP)<\/b><\/h4>\n
<\/p>\n
Data Parallelism (DP) is a foundational and widely adopted strategy for distributed training. Its conceptual simplicity is appealing: a model is replicated in its entirety across multiple GPU workers, and each worker processes a different subset of the training data. After a forward and backward pass, the gradients computed on each worker are synchronized and averaged across all workers, typically via an All-Reduce communication collective, to ensure that the model weights remain consistent.<\/span>12<\/span><\/p>\nWhile this approach effectively parallelizes computation and can significantly accelerate training time for large datasets, it is fundamentally inefficient from a memory perspective. The total memory required during a training step comprises three primary components: the model parameters ($P$), the gradients ($G$), and the optimizer states ($OS$).<\/span>12<\/span> For modern optimizers like Adam, which store first and second-order moments (momentum and variance), the optimizer states alone can consume two to three times the memory of the model parameters.<\/span>6<\/span><\/p>\nIn a standard DP setup with $N$ GPUs, each of these components is replicated on every worker. Consequently, the total memory consumed for these model states scales linearly with the number of workers: $N \\times (P + G + OS)$.<\/span>12<\/span> This replication creates a systemic bottleneck. The maximum size of a model that can be trained is not determined by the total, aggregate memory of the cluster, but by the memory capacity of a <\/span>single<\/span><\/i> GPU. This hard constraint, often referred to as the “GPU Memory Wall,” means that adding more GPUs to a cluster does not enable the training of a larger model; it only allows for a larger global batch size.<\/span>6<\/span> This inefficiency represents a software paradigm that fails to leverage the full potential of the available hardware.<\/span><\/p>\n <\/p>\n
2.2 Limitations of Model Parallelism (MP) as a Naive Solution<\/b><\/h4>\n
<\/p>\n
As an alternative to DP, Model Parallelism (MP) partitions the model itself across multiple GPUs. In its most common form, different layers of a neural network are placed on different devices.<\/span>13<\/span> This approach directly addresses the single-GPU memory limitation of DP, as no single device needs to hold the entire model. However, this solution introduces a new set of significant challenges.<\/span><\/p>\nThe primary drawback of this layer-wise partitioning is severe hardware underutilization. Because the computation is sequential\u2014the output of layer $i$ on GPU A is the input to layer $i+1$ on GPU B\u2014only one GPU is actively computing at any given moment during the forward or backward pass. The remaining GPUs are idle, waiting for data. This sequential dependency creates a “bubble” of inactivity that travels through the pipeline, drastically reducing computational efficiency.<\/span>19<\/span> While techniques like pipeline parallelism can mitigate this by splitting batches into micro-batches, they add complexity and do not entirely eliminate the bubble effect.<\/span>20<\/span><\/p>\nFurthermore, implementing MP is notoriously complex. It often requires intrusive and model-specific code refactoring to slice the model architecture and manage the data flow between devices. This high barrier to entry makes it a less generalizable and more difficult solution for many researchers and practitioners, hindering rapid experimentation and development.<\/span>1<\/span><\/p>\n <\/p>\n
2.3 The Motivation for ZeRO: A New Paradigm<\/b><\/h4>\n
<\/p>\n
The limitations of both DP and MP highlighted the need for a new distributed training paradigm. The ideal solution would synthesize the strengths of both approaches: the computational efficiency and ease of use of Data Parallelism combined with the memory scalability of Model Parallelism, while avoiding their respective weaknesses.<\/span>12<\/span><\/p>\nThis is the context in which Microsoft Research introduced the Zero Redundancy Optimizer. ZeRO was conceived as a novel solution that directly attacks the root cause of DP’s inefficiency\u2014memory redundancy. The conceptual leap was to re-architect the software paradigm to fundamentally alter the relationship between aggregate cluster memory and trainable model size. Instead of treating each GPU as an isolated memory island that must contain the entire model state, ZeRO treats the aggregate memory of the cluster as a single, unified pool.<\/span>3<\/span> By partitioning the model states across all data-parallel workers, ZeRO eliminates redundancy while retaining high computational granularity and manageable communication volume. This allows the maximum trainable model size to become a function of the <\/span>total cluster memory<\/span><\/i>, not single-GPU memory, representing a fundamental shift in the scaling equation from a constant constraint to a linearly scaling capability.<\/span>1<\/span><\/p>\n <\/p>\n
Section 3: The ZeRO Architecture: Progressive Partitioning for Memory Efficiency<\/b><\/h3>\n
<\/p>\n
The Zero Redundancy Optimizer achieves its remarkable memory efficiency through a simple yet powerful principle: it partitions the three primary model states\u2014optimizer states, gradients, and parameters\u2014across data-parallel processes instead of replicating them.<\/span>3<\/span> This strategy is implemented in a series of progressive stages, each offering a greater degree of memory savings at the cost of increased communication. This staged design provides a crucial “knob” for developers, allowing them to navigate the fundamental trade-off between memory footprint and communication overhead and tailor the system to their specific hardware, model architecture, and performance goals.<\/span><\/p>\n <\/p>\n
3.1 Stage 1: Optimizer State Partitioning (<\/b>$P_{os}$<\/b>)<\/b><\/h4>\n
<\/p>\n
The first and most foundational stage of ZeRO targets the optimizer states, which are often the largest consumer of memory in mixed-precision training.<\/span><\/p>\n\n- Mechanism:<\/b> In Stage 1, only the optimizer states (e.g., the 32-bit momentum and variance buffers for the Adam optimizer) are partitioned across the $N$ data-parallel workers. Each GPU continues to hold a full replica of the model parameters (in 16-bit) and gradients. During the optimizer step, each GPU is responsible for updating only its assigned 1\/Nth partition of the optimizer state and the corresponding model parameters. After the local update, an All-Gather operation is performed to ensure all GPUs receive the updated full set of parameters.<\/span>4<\/span><\/li>\n
- Memory Savings:<\/b> Because optimizer states can account for a large portion of the total memory (e.g., 12 bytes per parameter for Adam in mixed precision, compared to 2 bytes for the FP16 parameters), partitioning them yields substantial savings. This stage can provide up to a 4x reduction in memory compared to standard data parallelism, enabling the training of significantly larger models.<\/span>16<\/span><\/li>\n
- Communication:<\/b> The communication pattern is only minimally altered from standard DP. The primary All-Reduce on gradients remains, with an additional All-Gather for the updated parameters. This makes Stage 1 a low-risk, high-reward optimization that is easy to adopt. It was this stage that was first implemented in DeepSpeed and used to train the pioneering 17-billion-parameter Turing-NLG model.<\/span>16<\/span><\/li>\n<\/ul>\n
<\/p>\n
3.2 Stage 2: Gradient Partitioning (<\/b>$P_{os+g}$<\/b>)<\/b><\/h4>\n
<\/p>\n
ZeRO Stage 2 builds directly upon Stage 1, extending the partitioning strategy to include the gradients computed during the backward pass.<\/span><\/p>\n\n- Mechanism:<\/b> In addition to partitioning the optimizer states, Stage 2 also partitions the 16-bit gradients. After the backward pass, each GPU no longer holds the full gradient tensor. Instead, it only retains the gradients corresponding to its partition of the optimizer states.<\/span>5<\/span><\/li>\n
- Memory Savings:<\/b> By eliminating the redundant storage of gradients, this stage nearly doubles the memory savings of Stage 1. It can achieve up to an 8x reduction in memory for model states compared to standard DP.<\/span>16<\/span> This increase in efficiency allows for the training of models up to approximately 13 billion parameters without resorting to the complexities of model parallelism.<\/span>16<\/span><\/li>\n
- Communication:<\/b> The communication pattern is modified more significantly than in Stage 1. The standard All-Reduce collective, which both reduces (sums) and broadcasts the gradients, is replaced by a Reduce-Scatter operation. In this operation, gradients are summed and immediately scattered, so each GPU receives only its corresponding partition of the averaged gradients. This is followed by an All-Gather of the updated parameters after the optimizer step. The total communication volume remains similar to standard DP, but the pattern is different.<\/span>15<\/span><\/li>\n<\/ul>\n
<\/p>\n
3.3 Stage 3: Full Parameter Partitioning (<\/b>$P_{os+g+p}$<\/b>)<\/b><\/h4>\n
<\/p>\n
Stage 3 is the most aggressive and memory-efficient level of ZeRO optimization, partitioning all three model states and enabling model size to scale linearly with the number of devices.<\/span><\/p>\n\n- Mechanism:<\/b> This stage partitions the optimizer states, gradients, and the 16-bit model parameters themselves.<\/span>3<\/span> As a result, at no point during training does any single GPU hold the complete model in its memory.<\/span>16<\/span><\/li>\n
- Dynamic Materialization:<\/b> To perform computation, the full parameters for a given layer must be momentarily reconstructed on each GPU. This is achieved through a process of dynamic materialization. Just before a layer’s forward or backward pass is executed, an All-Gather communication collective is issued to gather the necessary parameter partitions from all other GPUs. Once the computation for that layer is complete, the now-stale full parameter tensor is discarded, freeing up the memory.<\/span>3<\/span><\/li>\n
- Memory Savings:<\/b> Stage 3 provides the maximum possible memory efficiency for a data-parallel approach. It makes the aggregate memory of the entire cluster available for storing the model, which is essential for training models with more than 13 billion parameters and is the foundational technology for pursuing trillion-parameter models.<\/span>1<\/span><\/li>\n
- Communication:<\/b> This efficiency comes at the cost of the highest communication overhead. The frequent All-Gather operations\u2014one for each layer during the forward pass and another during the backward pass\u2014are in addition to the gradient reduction communication. This increased communication volume makes the performance of Stage 3 highly sensitive to the underlying network interconnect bandwidth and the training batch size. Larger batch sizes can help amortize the communication cost by increasing the computation-to-communication ratio.<\/span>24<\/span><\/li>\n<\/ul>\n
The choice of ZeRO stage is therefore an optimization problem in itself. A user with a model that nearly fits in memory on a high-bandwidth cluster might choose ZeRO-2 for its balance of significant memory savings and high throughput. In contrast, a user aiming to train a model far too large for any single device must use ZeRO-3, accepting the communication penalty as the necessary cost of feasibility. This configurability is a key practical advantage of the ZeRO architecture.<\/span><\/p>\nTable 1: Comparison of ZeRO Optimization Stages<\/b><\/p>\n\n\n\nFeature<\/b><\/td>\n Standard DP (Baseline)<\/b><\/td>\n ZeRO Stage 1<\/b><\/td>\n ZeRO Stage 2<\/b><\/td>\n ZeRO Stage 3<\/b><\/td>\n<\/tr>\n\nPartitioned States<\/b><\/td>\n None<\/span><\/td>\n Optimizer States<\/span><\/td>\n Optimizer States, Gradients<\/span><\/td>\n Optimizer States, Gradients, Parameters<\/span><\/td>\n<\/tr>\n\nMemory Savings<\/b><\/td>\n 1x (Baseline)<\/span><\/td>\n Up to 4x<\/span><\/td>\n Up to 8x<\/span><\/td>\n Linear with N devices<\/span><\/td>\n<\/tr>\n\nKey Communication<\/b><\/td>\n All-Reduce (Gradients)<\/span><\/td>\n All-Reduce (Gradients), All-Gather (Parameters)<\/span><\/td>\n Reduce-Scatter (Gradients), All-Gather (Parameters)<\/span><\/td>\n All-Gather (Parameters, Fwd\/Bwd), Reduce-Scatter (Gradients)<\/span><\/td>\n<\/tr>\n\nCommunication Volume<\/b><\/td>\n 1.5$P$<\/span><\/td>\n 1.5$P$<\/span><\/td>\n 1.5$P$<\/span><\/td>\n 2.5$P$ + Forward\/Backward All-Gather<\/span><\/td>\n<\/tr>\n\nIdeal Use Case<\/b><\/td>\n Small models (<1.4B)<\/span><\/td>\n Models up to ~6B<\/span><\/td>\n Models up to ~13B<\/span><\/td>\n Models >13B; Trillion-scale<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\nNote: Memory savings and communication volumes are approximate and depend on factors like the optimizer used and mixed-precision settings. $P$ denotes the number of model parameters.<\/span><\/i><\/p>\n <\/p>\n
Section 4: Scaling Beyond GPU Memory: ZeRO-Offload and ZeRO-Infinity<\/b><\/h3>\n
<\/p>\n
While the core ZeRO stages dramatically improve the utilization of aggregate GPU memory, the total VRAM in a cluster remains a finite resource. To push the boundaries of model scale even further, the DeepSpeed team developed extensions that integrate a hierarchy of slower but more abundant memory tiers\u2014namely CPU RAM and NVMe storage\u2014into the training process. This evolution represents a paradigm shift from a purely GPU-centric view of training to a holistic, system-level approach that orchestrates all available memory and compute resources.<\/span><\/p>\n <\/p>\n
4.1 ZeRO-Offload: Democratizing Billion-Scale Training<\/b><\/h4>\n
<\/p>\n
ZeRO-Offload was the first major step in this direction, designed to make training billion-parameter models accessible even on systems with limited GPU resources.<\/span><\/p>\n\n- Concept:<\/b> ZeRO-Offload builds upon the foundation of ZeRO Stage 2. It offloads the partitioned optimizer states and gradients from the GPU’s high-bandwidth memory (HBM) to the host system’s main CPU memory (DRAM). Crucially, it also offloads the optimizer computation itself\u2014the optimizer.step() call\u2014to be executed on the CPU.<\/span>7<\/span><\/li>\n
- Impact:<\/b> This strategy frees up a massive amount of GPU VRAM, which can then be used to fit larger models or increase batch sizes. The impact is transformative: ZeRO-Offload enables the training of models with over 13 billion parameters on a <\/span>single GPU<\/span><\/i>, a tenfold increase compared to what is possible with standard frameworks like PyTorch.<\/span>7<\/span> This effectively “democratizes” large model training, allowing researchers and developers without access to large multi-GPU clusters to work with state-of-the-art models.<\/span>27<\/span><\/li>\n
- Efficiency:<\/b> A naive implementation would be crippled by the slow PCIe bus connecting the CPU and GPU. ZeRO-Offload is designed to be optimal by minimizing this data movement. It carefully schedules the transfer of gradients to the CPU and updated weights back to the GPU to overlap with computation, ensuring that the offloaded CPU work does not become a performance bottleneck. This allows it to achieve high computational throughput (e.g., 40 TFlops\/GPU on an NVIDIA V100) even while leveraging CPU resources.<\/span>7<\/span><\/li>\n<\/ul>\n
<\/p>\n
4.2 ZeRO-Infinity: Breaking the GPU Memory Wall with Heterogeneous Memory<\/b><\/h4>\n
<\/p>\n
ZeRO-Infinity represents the next generation of offloading technology, extending the principles of ZeRO-Offload to their logical conclusion by integrating the entire memory hierarchy of a modern compute node.<\/span><\/p>\n\n- Concept:<\/b> ZeRO-Infinity is built on top of the full partitioning of ZeRO Stage 3. It is capable of offloading <\/span>all<\/span><\/i> partitioned model states\u2014parameters, gradients, and optimizer states\u2014to a hierarchy of heterogeneous memory. This includes not only the CPU’s main memory but also high-speed Non-Volatile Memory Express (NVMe) solid-state drives, which offer terabytes of storage at a lower cost than DRAM.<\/span>3<\/span><\/li>\n
- Unprecedented Scale:<\/b> By leveraging the full memory capacity of the entire system, ZeRO-Infinity effectively breaks through the memory wall of the GPU cluster itself. It provides a clear path to training models with tens or even hundreds of trillions of parameters on current-generation hardware.<\/span>3<\/span> For instance, it can be used to fine-tune a trillion-parameter model on a single DGX-2 node or train a 30-trillion-parameter model on 512 GPUs.<\/span>3<\/span><\/li>\n
- Ease of Use:<\/b> A significant advantage of ZeRO-Infinity is that it achieves this massive scale without requiring the user to implement complex hybrid parallelism strategies (like 3D parallelism) or perform manual, intrusive model refactoring. The system automates the necessary communication and data movement, simplifying the process of training at an extreme scale.<\/span>3<\/span><\/li>\n<\/ul>\n
<\/p>\n
4.3 Overcoming Bandwidth Limitations of Offloading<\/b><\/h4>\n
<\/p>\n
The primary challenge of offloading is the significant bandwidth disparity between GPU HBM (TB\/s), CPU DRAM (GB\/s), and NVMe storage (GB\/s). ZeRO-Infinity employs several sophisticated, system-level innovations to manage this data pipeline and hide the latency of slower memory tiers.<\/span><\/p>\n\n- Bandwidth-Centric Partitioning:<\/b> Traditional ZeRO-3 assigns each parameter partition to a single GPU, which then broadcasts it when needed. ZeRO-Infinity alters this by partitioning each individual parameter across <\/span>all<\/span><\/i> data-parallel GPUs. When the full parameter is needed, an All-Gather collective is used. This is advantageous because on a multi-node cluster, the aggregate interconnect bandwidth (e.g., InfiniBand) is far greater than the PCIe bandwidth of a single node. This strategy effectively uses the high-speed network to compensate for the slow local CPU-GPU link.<\/span>8<\/span><\/li>\n
- Overlap-Centric Design:<\/b> The system features a dynamic prefetching engine that intelligently schedules the multi-stage data movement. It can overlap the NVMe-to-CPU transfer for a future layer’s parameters with the CPU-to-GPU transfer of the next layer’s parameters, all while the GPU is computing the current layer. This sophisticated scheduling creates a deep pipeline that effectively hides the latency of the slower memory transfers.<\/span>3<\/span><\/li>\n
- DeepNVMe Engine:<\/b> To maximize the performance of NVMe offloading, ZeRO-Infinity includes a high-performance C++ library called DeepNVMe. This engine supports asynchronous bulk read\/write requests, allowing the overlap engine to manage I\/O operations in parallel with computation and communication, and is capable of achieving near-peak sequential bandwidth from the underlying NVMe hardware.<\/span>8<\/span><\/li>\n<\/ul>\n
Through these innovations, ZeRO-Infinity transitions from a GPU-centric training model to a holistic systems approach. It treats the entire compute node\u2014with its hierarchy of GPU, CPU, and NVMe resources\u2014as a single, powerful, and intelligently orchestrated unit, paving the way for the next generation of extreme-scale AI.<\/span><\/p>\n <\/p>\n
Section 5: Optimizing Communication: The ZeRO++ Enhancements<\/b><\/h3>\n
<\/p>\n
As ZeRO-3 and ZeRO-Infinity successfully addressed the memory capacity bottleneck, enabling the construction of massive models, the performance bottleneck naturally shifted to the next limiting factor: communication overhead. The sheer volume of data that needs to be moved between devices during each training step can saturate network links and limit overall throughput. This is particularly acute in two common scenarios: 1) training on clusters with lower-bandwidth interconnects (e.g., Ethernet instead of InfiniBand), and 2) training at very large scales, where the global batch size is fixed for convergence reasons, leading to a very small per-GPU batch size and thus a low computation-to-communication ratio.<\/span>9<\/span><\/p>\nThe total communication volume of a standard ZeRO-3 implementation is approximately $3M$ for a model of size $M$, composed of an $M$-sized All-Gather for weights in the forward pass, another $M$-sized All-Gather for weights in the backward pass, and an $M$-sized Reduce-Scatter for gradients.<\/span>9<\/span> ZeRO++ was introduced as a suite of powerful techniques built on top of ZeRO-3 to drastically reduce this volume, shifting the optimization focus from the communication <\/span>pipe<\/span><\/i> to the <\/span>data<\/span><\/i> being sent through it.<\/span><\/p>\n <\/p>\n
5.1 Key Techniques of ZeRO++<\/b><\/h4>\n
<\/p>\n
ZeRO++ is not a monolithic change but a collection of three distinct, independently-configurable optimizations that target each of the major communication collectives in ZeRO-3.<\/span>10<\/span> This design reflects a cross-pollination of ideas, applying concepts like quantization, typically used for inference optimization, to the distributed training communication process itself.<\/span><\/p>\n\n- Quantized Weight Communication (qwZ):<\/b> This technique targets the $M$-sized All-Gather of weights during the forward pass. Instead of communicating the parameters in their standard 16-bit floating-point (FP16) format, qwZ applies block-based quantization to shrink each parameter to a lower-precision format, such as 8-bit integer (INT8), before communication. After the quantized data is received, it is dequantized back to FP16 for the computation. This simple change immediately reduces the communication volume for the forward pass by half, from $M$ to $0.5M$.<\/span>9<\/span><\/li>\n
- Hierarchical Partitioning ZeRO (hpZ):<\/b> This technique is designed to eliminate the expensive cross-node All-Gather of weights during the backward pass entirely. It achieves this by making a strategic trade-off between memory and communication. Instead of partitioning the model weights across all GPUs in the entire cluster, hpZ maintains a full copy of the model parameters <\/span>within each compute node<\/span><\/i>, while still partitioning them across the GPUs inside that node. This increases the memory footprint on each GPU, but it means that the All-Gather operation required for the backward pass can now be performed over the extremely high-bandwidth, low-latency intra-node interconnect (e.g., NVLink), rather than the slower inter-node network. This effectively reduces the cross-node communication volume for the backward pass from $M$ to zero.<\/span>9<\/span><\/li>\n
- Quantized Gradient Averaging (qgZ):<\/b> This is the most novel component, targeting the $M$-sized Reduce-Scatter of gradients. A naive approach of quantizing gradients within a standard ring-based Reduce-Scatter would introduce cumulative quantization errors and high latency. Instead, qgZ replaces the collective entirely with a new paradigm based on a 1-hop All-to-All communication pattern. In this approach, each GPU first quantizes its local gradient partition. Then, a single All-to-All operation exchanges these quantized partitions among all GPUs. Finally, each GPU dequantizes the received gradient chunks back to full precision <\/span>before<\/span><\/i> performing the reduction (summation). By performing the reduction on high-precision values, this method preserves numerical accuracy while communicating only low-precision data, reducing the gradient communication volume by up to 4x (e.g., from FP16 to INT4).<\/span>9<\/span><\/li>\n<\/ul>\n
<\/p>\n
5.2 Performance Impact<\/b><\/h4>\n
<\/p>\n
The collective impact of these three optimizations is a dramatic reduction in communication overhead and a corresponding increase in training throughput.<\/span><\/p>\n\n- Communication Volume Reduction:<\/b> Together, qwZ, hpZ, and qgZ reduce the total cross-node communication volume of ZeRO by 4x, from the original $3M$ to less than $0.75M$ ($0.5M$ for forward all-gather, $0$ for backward all-gather, and $\\approx 0.25M$ for gradient all-to-all).<\/span>9<\/span><\/li>\n
The quantifiable impact of ZeRO is substantial. It has enabled the training of models with over 100 billion parameters, achieving up to a 10x increase in performance over previous state-of-the-art systems.<\/span>1<\/span> Its integration into the DeepSpeed library and subsequent adoption by high-level frameworks like Hugging Face Accelerate and PyTorch Lightning have democratized access to large-scale training, making it feasible for a broader community of researchers and developers.<\/span>7<\/span> Ultimately, ZeRO is not merely a technical tool but a foundational enabler of the current era of large language models (LLMs), providing the systems-level breakthrough necessary to translate theoretical model designs into tangible, state-of-the-art artifacts.<\/span><\/p>\n <\/p>\n <\/p>\n The rapid escalation in the size and complexity of deep learning models has consistently outpaced the growth of hardware memory capacity. This disparity created a significant challenge for distributed training, where traditional methods proved incapable of efficiently utilizing the aggregate resources of a compute cluster. The core of this problem lay in the inherent memory redundancy of prevailing parallelization strategies, which established a “memory wall” that limited model scale based on the constraints of a single accelerator.<\/span><\/p>\n <\/p>\n <\/p>\n Data Parallelism (DP) is a foundational and widely adopted strategy for distributed training. Its conceptual simplicity is appealing: a model is replicated in its entirety across multiple GPU workers, and each worker processes a different subset of the training data. After a forward and backward pass, the gradients computed on each worker are synchronized and averaged across all workers, typically via an All-Reduce communication collective, to ensure that the model weights remain consistent.<\/span>12<\/span><\/p>\n While this approach effectively parallelizes computation and can significantly accelerate training time for large datasets, it is fundamentally inefficient from a memory perspective. The total memory required during a training step comprises three primary components: the model parameters ($P$), the gradients ($G$), and the optimizer states ($OS$).<\/span>12<\/span> For modern optimizers like Adam, which store first and second-order moments (momentum and variance), the optimizer states alone can consume two to three times the memory of the model parameters.<\/span>6<\/span><\/p>\n In a standard DP setup with $N$ GPUs, each of these components is replicated on every worker. Consequently, the total memory consumed for these model states scales linearly with the number of workers: $N \\times (P + G + OS)$.<\/span>12<\/span> This replication creates a systemic bottleneck. The maximum size of a model that can be trained is not determined by the total, aggregate memory of the cluster, but by the memory capacity of a <\/span>single<\/span><\/i> GPU. This hard constraint, often referred to as the “GPU Memory Wall,” means that adding more GPUs to a cluster does not enable the training of a larger model; it only allows for a larger global batch size.<\/span>6<\/span> This inefficiency represents a software paradigm that fails to leverage the full potential of the available hardware.<\/span><\/p>\n <\/p>\n <\/p>\n As an alternative to DP, Model Parallelism (MP) partitions the model itself across multiple GPUs. In its most common form, different layers of a neural network are placed on different devices.<\/span>13<\/span> This approach directly addresses the single-GPU memory limitation of DP, as no single device needs to hold the entire model. However, this solution introduces a new set of significant challenges.<\/span><\/p>\n The primary drawback of this layer-wise partitioning is severe hardware underutilization. Because the computation is sequential\u2014the output of layer $i$ on GPU A is the input to layer $i+1$ on GPU B\u2014only one GPU is actively computing at any given moment during the forward or backward pass. The remaining GPUs are idle, waiting for data. This sequential dependency creates a “bubble” of inactivity that travels through the pipeline, drastically reducing computational efficiency.<\/span>19<\/span> While techniques like pipeline parallelism can mitigate this by splitting batches into micro-batches, they add complexity and do not entirely eliminate the bubble effect.<\/span>20<\/span><\/p>\n Furthermore, implementing MP is notoriously complex. It often requires intrusive and model-specific code refactoring to slice the model architecture and manage the data flow between devices. This high barrier to entry makes it a less generalizable and more difficult solution for many researchers and practitioners, hindering rapid experimentation and development.<\/span>1<\/span><\/p>\n <\/p>\n <\/p>\n The limitations of both DP and MP highlighted the need for a new distributed training paradigm. The ideal solution would synthesize the strengths of both approaches: the computational efficiency and ease of use of Data Parallelism combined with the memory scalability of Model Parallelism, while avoiding their respective weaknesses.<\/span>12<\/span><\/p>\n This is the context in which Microsoft Research introduced the Zero Redundancy Optimizer. ZeRO was conceived as a novel solution that directly attacks the root cause of DP’s inefficiency\u2014memory redundancy. The conceptual leap was to re-architect the software paradigm to fundamentally alter the relationship between aggregate cluster memory and trainable model size. Instead of treating each GPU as an isolated memory island that must contain the entire model state, ZeRO treats the aggregate memory of the cluster as a single, unified pool.<\/span>3<\/span> By partitioning the model states across all data-parallel workers, ZeRO eliminates redundancy while retaining high computational granularity and manageable communication volume. This allows the maximum trainable model size to become a function of the <\/span>total cluster memory<\/span><\/i>, not single-GPU memory, representing a fundamental shift in the scaling equation from a constant constraint to a linearly scaling capability.<\/span>1<\/span><\/p>\n <\/p>\n <\/p>\n The Zero Redundancy Optimizer achieves its remarkable memory efficiency through a simple yet powerful principle: it partitions the three primary model states\u2014optimizer states, gradients, and parameters\u2014across data-parallel processes instead of replicating them.<\/span>3<\/span> This strategy is implemented in a series of progressive stages, each offering a greater degree of memory savings at the cost of increased communication. This staged design provides a crucial “knob” for developers, allowing them to navigate the fundamental trade-off between memory footprint and communication overhead and tailor the system to their specific hardware, model architecture, and performance goals.<\/span><\/p>\n <\/p>\n <\/p>\n The first and most foundational stage of ZeRO targets the optimizer states, which are often the largest consumer of memory in mixed-precision training.<\/span><\/p>\n <\/p>\n <\/p>\n ZeRO Stage 2 builds directly upon Stage 1, extending the partitioning strategy to include the gradients computed during the backward pass.<\/span><\/p>\n <\/p>\n <\/p>\n Stage 3 is the most aggressive and memory-efficient level of ZeRO optimization, partitioning all three model states and enabling model size to scale linearly with the number of devices.<\/span><\/p>\n The choice of ZeRO stage is therefore an optimization problem in itself. A user with a model that nearly fits in memory on a high-bandwidth cluster might choose ZeRO-2 for its balance of significant memory savings and high throughput. In contrast, a user aiming to train a model far too large for any single device must use ZeRO-3, accepting the communication penalty as the necessary cost of feasibility. This configurability is a key practical advantage of the ZeRO architecture.<\/span><\/p>\n Table 1: Comparison of ZeRO Optimization Stages<\/b><\/p>\n Note: Memory savings and communication volumes are approximate and depend on factors like the optimizer used and mixed-precision settings. $P$ denotes the number of model parameters.<\/span><\/i><\/p>\n <\/p>\n <\/p>\n While the core ZeRO stages dramatically improve the utilization of aggregate GPU memory, the total VRAM in a cluster remains a finite resource. To push the boundaries of model scale even further, the DeepSpeed team developed extensions that integrate a hierarchy of slower but more abundant memory tiers\u2014namely CPU RAM and NVMe storage\u2014into the training process. This evolution represents a paradigm shift from a purely GPU-centric view of training to a holistic, system-level approach that orchestrates all available memory and compute resources.<\/span><\/p>\n <\/p>\n <\/p>\n ZeRO-Offload was the first major step in this direction, designed to make training billion-parameter models accessible even on systems with limited GPU resources.<\/span><\/p>\n <\/p>\n <\/p>\n ZeRO-Infinity represents the next generation of offloading technology, extending the principles of ZeRO-Offload to their logical conclusion by integrating the entire memory hierarchy of a modern compute node.<\/span><\/p>\n <\/p>\n <\/p>\n The primary challenge of offloading is the significant bandwidth disparity between GPU HBM (TB\/s), CPU DRAM (GB\/s), and NVMe storage (GB\/s). ZeRO-Infinity employs several sophisticated, system-level innovations to manage this data pipeline and hide the latency of slower memory tiers.<\/span><\/p>\n Through these innovations, ZeRO-Infinity transitions from a GPU-centric training model to a holistic systems approach. It treats the entire compute node\u2014with its hierarchy of GPU, CPU, and NVMe resources\u2014as a single, powerful, and intelligently orchestrated unit, paving the way for the next generation of extreme-scale AI.<\/span><\/p>\n <\/p>\n <\/p>\n As ZeRO-3 and ZeRO-Infinity successfully addressed the memory capacity bottleneck, enabling the construction of massive models, the performance bottleneck naturally shifted to the next limiting factor: communication overhead. The sheer volume of data that needs to be moved between devices during each training step can saturate network links and limit overall throughput. This is particularly acute in two common scenarios: 1) training on clusters with lower-bandwidth interconnects (e.g., Ethernet instead of InfiniBand), and 2) training at very large scales, where the global batch size is fixed for convergence reasons, leading to a very small per-GPU batch size and thus a low computation-to-communication ratio.<\/span>9<\/span><\/p>\n The total communication volume of a standard ZeRO-3 implementation is approximately $3M$ for a model of size $M$, composed of an $M$-sized All-Gather for weights in the forward pass, another $M$-sized All-Gather for weights in the backward pass, and an $M$-sized Reduce-Scatter for gradients.<\/span>9<\/span> ZeRO++ was introduced as a suite of powerful techniques built on top of ZeRO-3 to drastically reduce this volume, shifting the optimization focus from the communication <\/span>pipe<\/span><\/i> to the <\/span>data<\/span><\/i> being sent through it.<\/span><\/p>\n <\/p>\n <\/p>\n ZeRO++ is not a monolithic change but a collection of three distinct, independently-configurable optimizations that target each of the major communication collectives in ZeRO-3.<\/span>10<\/span> This design reflects a cross-pollination of ideas, applying concepts like quantization, typically used for inference optimization, to the distributed training communication process itself.<\/span><\/p>\n <\/p>\n <\/p>\n The collective impact of these three optimizations is a dramatic reduction in communication overhead and a corresponding increase in training throughput.<\/span><\/p>\nSection 2: The Challenge of Memory Redundancy in Distributed Training<\/b><\/h3>\n
2.1 The Inefficiency of Traditional Data Parallelism (DP)<\/b><\/h4>\n
2.2 Limitations of Model Parallelism (MP) as a Naive Solution<\/b><\/h4>\n
2.3 The Motivation for ZeRO: A New Paradigm<\/b><\/h4>\n
Section 3: The ZeRO Architecture: Progressive Partitioning for Memory Efficiency<\/b><\/h3>\n
3.1 Stage 1: Optimizer State Partitioning (<\/b>$P_{os}$<\/b>)<\/b><\/h4>\n
\n
3.2 Stage 2: Gradient Partitioning (<\/b>$P_{os+g}$<\/b>)<\/b><\/h4>\n
\n
3.3 Stage 3: Full Parameter Partitioning (<\/b>$P_{os+g+p}$<\/b>)<\/b><\/h4>\n
\n
\n\n
\n Feature<\/b><\/td>\n Standard DP (Baseline)<\/b><\/td>\n ZeRO Stage 1<\/b><\/td>\n ZeRO Stage 2<\/b><\/td>\n ZeRO Stage 3<\/b><\/td>\n<\/tr>\n \n Partitioned States<\/b><\/td>\n None<\/span><\/td>\n Optimizer States<\/span><\/td>\n Optimizer States, Gradients<\/span><\/td>\n Optimizer States, Gradients, Parameters<\/span><\/td>\n<\/tr>\n \n Memory Savings<\/b><\/td>\n 1x (Baseline)<\/span><\/td>\n Up to 4x<\/span><\/td>\n Up to 8x<\/span><\/td>\n Linear with N devices<\/span><\/td>\n<\/tr>\n \n Key Communication<\/b><\/td>\n All-Reduce (Gradients)<\/span><\/td>\n All-Reduce (Gradients), All-Gather (Parameters)<\/span><\/td>\n Reduce-Scatter (Gradients), All-Gather (Parameters)<\/span><\/td>\n All-Gather (Parameters, Fwd\/Bwd), Reduce-Scatter (Gradients)<\/span><\/td>\n<\/tr>\n \n Communication Volume<\/b><\/td>\n 1.5$P$<\/span><\/td>\n 1.5$P$<\/span><\/td>\n 1.5$P$<\/span><\/td>\n 2.5$P$ + Forward\/Backward All-Gather<\/span><\/td>\n<\/tr>\n \n Ideal Use Case<\/b><\/td>\n Small models (<1.4B)<\/span><\/td>\n Models up to ~6B<\/span><\/td>\n Models up to ~13B<\/span><\/td>\n Models >13B; Trillion-scale<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n Section 4: Scaling Beyond GPU Memory: ZeRO-Offload and ZeRO-Infinity<\/b><\/h3>\n
4.1 ZeRO-Offload: Democratizing Billion-Scale Training<\/b><\/h4>\n
\n
4.2 ZeRO-Infinity: Breaking the GPU Memory Wall with Heterogeneous Memory<\/b><\/h4>\n
\n
4.3 Overcoming Bandwidth Limitations of Offloading<\/b><\/h4>\n
\n
Section 5: Optimizing Communication: The ZeRO++ Enhancements<\/b><\/h3>\n
5.1 Key Techniques of ZeRO++<\/b><\/h4>\n
\n
5.2 Performance Impact<\/b><\/h4>\n
\n