Section 1: Introduction – The Evolution of Parallel Graphics Processing
1.1 The Foundational Premise of Multi-GPU Scaling
The principle of multi-GPU (Graphics Processing Unit) scaling is rooted in the fundamental concept of parallel processing: the distribution of a complex computational task across multiple processors to achieve a result faster than a single processor could alone.1 In the domain of computer graphics and, more recently, general-purpose computing, the GPU has become a powerhouse of parallel execution. The strategy of harnessing multiple GPUs in a single system is a direct attempt to multiply this power, aiming to overcome the performance limitations of a single chip for the most demanding workloads.1 Whether for rendering photorealistic 3D environments, training sophisticated artificial intelligence models, or running complex scientific simulations, the goal remains the same: to divide the workload, process the constituent parts simultaneously, and synthesize the results into a cohesive whole, thereby reducing execution time and enabling the processing of datasets and models that would be intractable for a single GPU.2
1.2 Historical Roots: 3dfx and Scan-Line Interleave (SLI)
The concept of combining multiple graphics processors for consumer applications is not a recent innovation. Its origins can be traced back to 1998 with the pioneering work of 3dfx Interactive on its Voodoo2 graphics card.3 3dfx developed a technology it called Scan-Line Interleave (SLI), which allowed two Voodoo2 cards to operate in parallel within a single system. The operational principle was straightforward: one GPU was responsible for rendering all the even-numbered horizontal scan lines of the display, while the second GPU rendered all the odd-numbered lines.3
The initial promise of this technology was compelling. By dividing the rendering work, SLI could theoretically double the fill rate, reduce the time required to draw a complete frame, and increase the total available frame buffer memory, which in turn would allow for higher screen resolutions.3 However, this early implementation immediately exposed the fundamental challenge that would plague consumer multi-GPU technologies for the next two decades: the law of diminishing returns. A critical limitation was that the texture memory was not pooled; instead, it had to be duplicated on each card, as both processors needed access to the same scene data to render their respective lines.3 This redundancy, combined with other system overheads, meant that the real-world performance improvement was far from the theoretical 100% increase. An investment of double the cost—a pair of Voodoo2 cards retailed for nearly $500 in 1998—yielded a performance gain of only 60% to 70%, depending on the application and the system’s CPU.3 This established a crucial precedent: the cost-benefit analysis for multi-GPU setups has been unfavorable from the very beginning, a problem that was inherited, but never truly solved, by its successors.
1.3 The Bifurcation of Purpose
The history of multi-GPU technology is a story of divergence. After NVIDIA acquired 3dfx’s intellectual property and reintroduced the SLI brand in 2004, a clear split began to emerge in the application and design philosophy of multi-GPU systems.3 This report will argue that this divergence represents a fundamental bifurcation of purpose, from a consumer-focused goal of enhancing gaming performance to a professional-centric paradigm focused on data-intensive accelerated computing.
The consumer application, epitomized by NVIDIA’s SLI and AMD’s CrossFire, was almost singularly focused on a simple metric: increasing the frame rate in video games.5 This objective, while seemingly straightforward, was fraught with technical challenges related to frame synchronization, software support, and the aforementioned diminishing returns.7 In stark contrast, the burgeoning fields of High-Performance Computing (HPC) and Artificial Intelligence (AI) presented a vastly different and more complex set of demands. These professional workloads are not concerned with frames per second but with massive data throughput, ultra-low latency inter-processor communication, and the ability to treat the memory of multiple GPUs as a single, coherent pool.9
This fundamental difference in requirements explains the trajectory of multi-GPU technology. The very architectures and rendering methods that proved inadequate and were ultimately abandoned for the consumer gaming market (SLI and CrossFire) were not evolved but entirely superseded by new interconnect technologies, most notably NVIDIA’s NVLink, which was designed from the ground up to address the specific, data-centric challenges of the professional world.12 The failure in one domain directly informed the revolutionary design of the successor in another, marking a definitive shift from the enthusiast’s gaming rig to the scientist’s and data scientist’s supercluster.
Section 2: The Era of Consumer Multi-GPU: SLI and CrossFire
2.1 NVIDIA’s Scalable Link Interface (SLI): Architecture and Requirements
Following its acquisition of 3dfx’s assets, NVIDIA reintroduced the SLI brand in 2004, rebranding it as the Scalable Link Interface.3 This technology was designed to allow users to link multiple NVIDIA graphics cards to collaboratively process a graphical workload, with the primary goal of improving performance in demanding games and 3D applications.5
Setting up an SLI configuration involved a strict set of hardware requirements. First, the system required two or more compatible NVIDIA graphics cards, which ideally had to be identical models with the same GPU and VRAM configuration to ensure stable operation.5 These cards had to be installed in an SLI-certified motherboard equipped with multiple PCI Express (PCIe) x16 slots.18 A physical “SLI bridge” connector was necessary to link the top edges of the cards, providing a dedicated, high-speed communication path for synchronizing data and combining the final rendered output.5 Finally, the system needed a power supply unit (PSU) with sufficient wattage and the requisite number of power connectors to handle the significantly increased power demands of running multiple high-performance GPUs.5 Once the hardware was in place, users would enable SLI mode within the NVIDIA Control Panel, at which point the driver would treat the multiple physical GPUs as a single logical device for rendering.18
2.2 AMD’s CrossFire: A More Flexible, but Ultimately Similar, Approach
AMD, through its acquisition of ATI Technologies, introduced its own multi-GPU solution, CrossFire (later rebranded as CrossFireX), to compete directly with NVIDIA’s SLI.6 While the fundamental goal was identical—to combine the processing power of multiple GPUs for enhanced graphics performance—CrossFire was architected with a greater degree of flexibility.6
A key differentiator was CrossFire’s less stringent requirement for matching GPUs. While using identical cards was recommended for optimal performance, CrossFire allowed users to pair different Radeon GPU models, provided they were from the same generation and architectural family (e.g., two different cards from the 5800 series).6 In such a configuration, the performance would typically be limited by the capabilities of the less powerful card in the pair.6
CrossFire also underwent a significant architectural evolution in its interconnect method. Early implementations, much like SLI, required a physical bridge connector to synchronize the GPUs.6 However, later generations of CrossFireX transitioned to a bridgeless design that utilized the existing high-speed PCI Express bus for inter-card communication, a technology known as XDMA (CrossFire Direct Memory Access).6 This eliminated the need for a separate physical connector, simplifying the setup process and reducing system clutter.6 Despite these differences in flexibility and interconnect hardware, the core operational principles and the methods for distributing rendering workloads were largely analogous to those used by SLI.
2.3 Workload Distribution for Gaming: Rendering Modes in Detail
To divide the complex task of rendering a 3D scene, both SLI and CrossFire relied on a set of distinct rendering modes managed by the graphics driver. The choice of mode was critical, as it determined how the workload was partitioned and had profound implications for both performance and the perceived smoothness of the final output.
2.3.1 Alternate Frame Rendering (AFR)
Alternate Frame Rendering was the most widely used and often the default mode for both SLI and CrossFire configurations.6 Its method of workload division was temporal: the GPUs would render entire frames in a sequential, alternating pattern. For a two-GPU setup, the first GPU would render frame 1, the second GPU would render frame 2, the first GPU would render frame 3, and so on.4
The primary advantage of AFR was its potential for excellent performance scaling. In scenarios where the workload of consecutive frames was relatively consistent and the system was GPU-bound, AFR could approach a near-doubling of the average frame rate.17 This made it the preferred mode for achieving high scores in benchmarking software and was a key focus of marketing efforts, as the resulting high average FPS numbers were easy to advertise.18 However, as will be discussed, this focus on a simple numerical metric came at a significant cost to the actual user experience.
2.3.2 Split Frame Rendering (SFR)
Split Frame Rendering offered a spatial approach to workload division. Instead of alternating frames, SFR partitioned each individual frame into two or more sections, with each GPU assigned to render a specific portion.4 For example, one GPU might render the top half of the screen while the second GPU renders the bottom half.4 To optimize performance, the dividing line between these sections could be dynamically adjusted by the driver to balance the geometric complexity and rendering load between the GPUs.17
The main benefit of SFR was that it avoided the temporal synchronization issues inherent to AFR, as all GPUs were contributing to the same frame at the same time.23 This meant it was not susceptible to the micro-stuttering phenomenon. However, SFR had its own significant drawbacks. The process of dividing a frame, rendering the parts, and then recompositing them into a final image incurred higher communication and synchronization overhead between the GPUs.18 Furthermore, achieving a perfect and dynamic load balance was technically much more difficult than the simple frame-swapping of AFR. As a result, SFR typically exhibited poorer performance scaling and delivered lower average frame rates, making it a less desirable option from a raw performance perspective.18
2.3.3 Specialized Modes (SLIAA, Hybrid SLI)
Beyond the two primary performance-oriented modes, specialized modes existed for other purposes. SLI Antialiasing (SLIAA), for instance, did not aim to increase the frame rate but to improve image quality.17 In this mode, both GPUs would render the same frame, but each would apply an antialiasing sample pattern slightly offset from the other. The final results were then combined to produce a much higher quality, smoother image than a single GPU could achieve, offering options like SLI 8x or SLI 16x antialiasing.17
Hybrid SLI (and its AMD counterpart, Hybrid CrossFire) was a technology designed to combine the power of a low-power integrated GPU (IGP) on the motherboard with a discrete, add-in GPU.17 This was primarily used in laptops and budget desktop systems to provide a modest performance boost or to allow the system to switch to the low-power IGP to save energy when high-performance graphics were not needed.17
Table 1: Comparative Analysis of AFR vs. SFR Rendering Modes
| Feature | Alternate Frame Rendering (AFR) | Split Frame Rendering (SFR) |
| Workload Division Method | Temporal: GPUs render sequential, whole frames (e.g., odd/even).4 | Spatial: Each frame is divided into sections, with each GPU rendering a portion.4 |
| Ideal Performance Scaling | High; can approach near-linear scaling (e.g., up to 1.9x for two GPUs).17 | Lower; suffers from higher overhead and load-balancing challenges.18 |
| VRAM Usage | Mirrored: All scene data is duplicated in each GPU’s VRAM. Effective VRAM is that of a single card.23 | Mirrored: All scene data must be available to each GPU, so VRAM is also duplicated.23 |
| Primary Advantage | Maximizes average frames per second (FPS) in ideal, GPU-bound scenarios.18 | Avoids temporal artifacts; not susceptible to micro-stuttering.23 |
| Primary Disadvantage | High susceptibility to micro-stuttering due to inconsistent frame delivery times.17 | Difficult to balance workload perfectly, leading to lower efficiency and performance gains.18 |
| Susceptibility to Micro-stuttering | High; this is the primary cause of the phenomenon.24 | None; frames are constructed collaboratively and presented as a single unit.23 |
| Communication Overhead | Lower; requires less inter-GPU communication as each GPU works on a whole frame independently.18 | Higher; requires significant communication to divide the frame and composite the final image.18 |
2.4 The Inherent Flaw: A Technical Explanation of Micro-stuttering
The single greatest failing of consumer multi-GPU technology, and the primary reason for its poor reputation among enthusiasts, was the phenomenon known as micro-stuttering.7 This issue was a direct and unavoidable consequence of the Alternate Frame Rendering (AFR) mode that both SLI and CrossFire predominantly relied upon for performance gains.24
While benchmarking tools like FRAPS would report a high and seemingly smooth average frame rate (e.g., 60 FPS), the actual on-screen experience was often choppy and inconsistent, feeling more like a much lower frame rate.7 This discrepancy between the measured average performance and the perceived visual smoothness is the essence of micro-stuttering. It is caused not by a low number of frames being produced, but by the inconsistent time intervals between when those frames are delivered to the display.24
In a single-GPU system, one processor is responsible for rendering every frame. While the time to render each frame will vary based on scene complexity, the delivery cadence is generally consistent. In a dual-GPU AFR setup, however, the two GPUs work asynchronously.24 GPU A renders frame 1, and GPU B renders frame 2. The time it takes GPU A to render frame 1 may be different from the time it takes GPU B to render frame 2. This can lead to a sequence where two frames are produced in rapid succession, followed by a long delay before the next pair of frames arrives. For example, a frame from GPU A might be ready after 10 milliseconds ($ms$), and the subsequent frame from GPU B might be ready just 6.6 $ms$ later, but the next frame from GPU A might take 26.6 $ms$ to appear. The average time between frames might be 16.6 $ms$ (equivalent to 60 FPS), but the user experiences a “fast” frame followed by a “slow” one, creating a jarring, stuttering effect.24
This problem reveals a fundamental conflict at the heart of the consumer multi-GPU value proposition. The industry chose to prioritize AFR because it produced the highest average FPS numbers, which were simple to measure and effective for marketing.18 However, this choice came at the direct expense of consistent frame pacing, a metric that is harder to quantify but is far more critical to the subjective experience of smooth gameplay. The pursuit of impressive benchmark figures led to the adoption of a rendering model that was experientially flawed, ultimately undermining the technology’s core promise of a superior gaming experience.
Section 3: The Unraveling of a Paradigm: Why SLI and CrossFire Failed in the Consumer Market
The era of consumer multi-GPU gaming, once the pinnacle of enthusiast PC building, ultimately collapsed under the weight of its own technical limitations, economic impracticalities, and a failing software ecosystem. The promise of scalable performance gave way to a reality of diminishing returns, frustrating user experiences, and waning support from all corners of the industry.
3.1 The Law of Diminishing Returns and Escalating Costs
The most immediate and tangible failure of SLI and CrossFire was their inability to provide a justifiable return on investment. The core premise—that adding a second GPU would lead to a commensurate increase in performance—was never fully realized. In the best-case scenarios, with a well-optimized game and no other system bottlenecks, a second GPU might provide a performance uplift of 50-80%.8 However, in many titles, the gain was significantly less, and in some cases, performance could even decrease due to driver overhead or poor optimization.27 This meant users were paying 100% of the cost for a second high-end graphics card for a fractional and unreliable performance benefit.8
This problem was critically exacerbated by a fundamental architectural limitation of the dominant AFR rendering mode: VRAM mirroring. Because each GPU in an AFR setup rendered a whole, independent frame, it needed a complete copy of all the game’s assets (textures, models, shaders) in its own local video memory.3 This meant that a system with two 8 GB graphics cards did not have an effective 16 GB of VRAM; it was limited to the 8 GB of a single card. This limitation acted as a ticking time bomb. The primary motivation for multi-GPU setups was to enable gaming at higher resolutions like 1440p and 4K, which demand significantly more VRAM.5 As games evolved and their VRAM requirements at these high settings began to exceed the capacity of a single card, the multi-GPU configuration offered no advantage in this crucial resource. The entire system would be bottlenecked by the VRAM of one card, rendering the immense computational power of the second GPU moot. This architectural flaw ensured that the technology was least effective in the very high-end scenarios it was marketed to solve, making its long-term failure inevitable.
3.2 The Software Ecosystem Collapse
While the hardware had its flaws, the ultimate death knell for consumer multi-GPU was the collapse of its software support structure. This happened in two distinct phases, culminating in a near-total abandonment by the game development industry.
3.2.1 The Burden of Support and the API Shift
Initially, the responsibility for ensuring game compatibility rested with NVIDIA and AMD. They maintained dedicated driver teams that created and tested specific multi-GPU “profiles” for nearly every major game release.29 This was a laborious and continuous effort, often requiring game-specific tweaks to force a particular rendering mode or work around engine-level bugs.30
This paradigm was upended with the introduction of modern, low-level graphics APIs like DirectX 12 and Vulkan. These APIs were designed to give game developers more direct control over the hardware. A major consequence of this shift was that the responsibility for implementing multi-GPU support moved from the GPU vendor’s driver to the game engine itself.20
3.2.2 Developer Apathy and Economic Reality
The transition to developer-led multi-GPU support acted as a forcing function that exposed the technology’s non-existent business case. While NVIDIA and AMD had a vested interest in selling more hardware, game developers operate on a different economic model where investment must be justified by the addressable market. Multi-GPU users consistently represented a tiny, sub-1% niche of the PC gaming market.31 When faced with the high cost and significant technical complexity of implementing and debugging multi-GPU support in their engines for such a minuscule audience, the decision for nearly every developer was simple and rational: to ignore it completely.31
This economic reality was compounded by a growing technical incompatibility. Modern rendering techniques, particularly Temporal Anti-Aliasing (TAA) and other effects that rely on data from previous frames to construct the current one, are fundamentally at odds with the basic premise of AFR, which treats each frame as an independent task.31 Re-architecting a modern rendering pipeline to accommodate the asynchronous nature of AFR was an immense technical hurdle that developers had no incentive to overcome. The API shift did not just make multi-GPU support harder; it moved the decision-making from hardware vendors to game studios, who promptly and logically abandoned the feature.
3.3 Physical and Practical Barriers
Beyond the software and value proposition issues, a host of practical hardware challenges made multi-GPU setups increasingly untenable for the average consumer.
3.3.1 Power and Thermal Demands
Running two high-performance graphics cards simultaneously places an enormous strain on a system’s power and cooling infrastructure.2 Power consumption could easily double, necessitating the purchase of expensive, high-wattage PSUs (often 1000W or more) and potentially straining residential electrical circuits.21 The heat output was also doubled, creating a thermal challenge within the PC case.2 In a typical tower configuration, the top GPU would draw in the hot air exhausted by the bottom GPU, leading to thermal throttling and reduced performance, negating the benefits of the second card.21 This often required investment in specialized chassis with high-airflow designs or complex liquid cooling solutions.26
3.3.2 Chassis and Motherboard Constraints
As GPUs grew larger and more powerful, the physical space within a standard PC case became a premium. Many high-end cards occupied two or even three expansion slots, making it physically difficult to install two of them on a standard motherboard without compromising airflow.21 Furthermore, motherboards needed not only multiple physical PCIe x16 slots but also the underlying PCIe lanes from the CPU to run them at sufficient speed. For optimal performance, a dual-GPU setup required the slots to run in at least an x8/x8 configuration, a feature often limited to more expensive enthusiast-grade motherboards.27
3.4 The Official End
Faced with a collapsing software ecosystem, a poor value proposition, and mounting hardware complexities, the GPU manufacturers officially sunsetted their consumer-focused technologies. AMD effectively retired the CrossFire brand in 2017 with the release of its RX Vega series GPUs.20 NVIDIA began phasing out SLI with its RTX 20 series, restricting the feature to only its most expensive top-tier cards (e.g., the RTX 2080 and 2080 Ti) and replacing the old SLI bridge with a new, more expensive NVLink bridge.12 By the RTX 30 series, SLI support was limited to only the flagship RTX 3090, effectively signaling its end for the gaming market and repositioning its underlying interconnect technology for professional workloads.37 The era of dual graphics cards for gaming had come to a definitive close.
Section 4: The Professional Renaissance: NVIDIA NVLink and the Dawn of High-Bandwidth Interconnects
As the viability of multi-GPU for consumer gaming disintegrated, a parallel evolution was occurring in the world of professional computing. The insatiable demands of high-performance computing (HPC) and artificial intelligence (AI) for data throughput created the impetus for a new class of interconnect technology. This led to the development of NVIDIA NVLink, an architecture that represents not an incremental improvement over SLI, but a fundamental paradigm shift designed for an entirely different class of workload.
4.1 A New Architecture for a New Workload
NVLink was engineered from the ground up to address the specific bottlenecks encountered in data-intensive, parallel computing tasks.9 Whereas SLI was a solution retrofitted for graphics, focused on synchronizing fully rendered frames, NVLink is a high-speed, low-latency fabric designed for the rapid and continuous exchange of raw data between processors.12 Its primary applications are not in gaming, but in scientific simulation, deep learning, large-scale data analytics, and high-end 3D rendering—workloads where the volume of data shared between GPUs far exceeds what can be efficiently handled by the traditional PCI Express bus.10
4.2 Architectural Deep Dive: NVLink vs. SLI
The architectural differences between SLI and NVLink underscore the magnitude of this technological leap. The comparison is not one of degrees, but of orders of magnitude, reflecting the shift from a graphics-centric to a data-centric design philosophy.
4.2.1 Bandwidth and Latency
The most striking difference lies in raw data throughput. The high-bandwidth SLI bridges provided a connection speed of approximately 1-2 GB/s.12 While this was sufficient for transferring display data, it was a crippling bottleneck for compute workloads. The PCI Express bus, the alternative communication path, offered more bandwidth—up to 32 GB/s for a PCIe Gen 4 x16 link—but came with higher latency as data had to be routed through the system’s CPU or a PCIe switch.9
NVLink obliterates these limitations. It provides a direct, point-to-point communication path between GPUs, bypassing the PCIe bus entirely.9 The bandwidth it provides is staggering in comparison. The third-generation NVLink used in the Ampere A100 GPU delivered 600 GB/s of total bidirectional bandwidth per GPU. The fourth-generation NVLink in the Hopper H100 GPU increased this to 900 GB/s, and the fifth-generation in the Blackwell architecture doubles it again to 1.8 TB/s.38 This represents a more than 14-fold increase over a PCIe Gen 5 connection and is hundreds of times faster than the old SLI bridge, enabling the massive data transfers required for training modern AI models.42
Table 2: Architectural and Performance Comparison: SLI vs. NVLink
| Feature | SLI (High-Bandwidth Bridge) | NVLink (4th Gen, H100) |
| Primary Use Case | Consumer Gaming, Graphics Rendering 5 | AI, HPC, Data Center Workloads 9 |
| Interconnect Type | Point-to-point bridge or PCIe bus 12 | High-speed, bidirectional direct mesh interconnect 12 |
| Typical Bandwidth | ~2 GB/s (bridge); up to 32 GB/s (PCIe 4.0) 12 | Up to 900 GB/s total bidirectional bandwidth per GPU 9 |
| Communication Path | GPU -> Bridge -> GPU or GPU -> PCIe -> CPU -> PCIe -> GPU 9 | Direct GPU-to-GPU, bypassing the CPU and PCIe bus 9 |
| Memory Model | Mirrored/Discrete: Each GPU has its own separate VRAM.12 | Unified/Pooled: Enables a shared memory space across multiple GPUs.9 |
| Scalability Limit | Typically 2-4 GPUs, with significant diminishing returns.17 | Scales to 8 GPUs per node and beyond with NVSwitch fabric.38 |
| Software Dependency | Game-specific driver profiles required for compatibility and performance.29 | Utilized directly by compute frameworks (CUDA, PyTorch, TensorFlow) and libraries (NCCL).12 |
4.3 The Memory Revolution: Unified Memory and Pooling
Perhaps the most transformative feature of NVLink is its ability to enable a unified memory architecture.9 This capability directly addresses and solves the critical VRAM mirroring limitation that plagued SLI. With NVLink, the discrete memory pools of multiple GPUs can be treated by the software as a single, large, coherent address space.44
This is a revolutionary shift in the programming model. For an AI researcher training a large language model, the combined memory of eight H100 GPUs can appear as one massive pool of VRAM. This allows for the training of models with trillions of parameters—models that are far too large to fit into the memory of a single GPU.44 The NVLink fabric handles the complexities of data placement and access, allowing developers to focus on their algorithms rather than on manually managing data transfers between separate memory spaces.47 This abstraction, moving from a model of “coordinating separate processors” to “programming a single, massive parallel processor,” is what makes modern, large-scale AI feasible. It is not just that data can be moved quickly; it is that the entire multi-GPU system can be programmed as a single, coherent computational entity.
4.4 Scaling Beyond the Server: The NVSwitch Fabric
To scale the benefits of NVLink beyond a handful of GPUs, NVIDIA developed the NVSwitch, a specialized silicon chip that acts as the backbone for large multi-GPU systems.38 The NVSwitch functions as a non-blocking crossbar switch, connecting all GPUs in a server or cluster and enabling simultaneous, all-to-all communication at full NVLink speed.45
The existence of this complex and expensive piece of hardware is a testament to a critical shift in the landscape of high-performance computing. As the computational power of individual GPUs has soared, the primary bottleneck for scaling large, distributed workloads has moved from on-chip computation to inter-GPU communication.45 Tasks like synchronizing gradients in distributed AI training require frequent, high-volume data exchanges between every GPU in the system.45 Without a switch, each GPU’s total interconnect bandwidth would have to be divided among its peers, creating a severe bottleneck as the number of GPUs increases.45 The NVSwitch solves this problem by providing a dedicated, high-radix fabric that ensures communication bandwidth does not degrade as the system scales. It is the key enabling technology behind NVIDIA’s DGX and HGX server platforms, which form the building blocks of the world’s most powerful AI supercomputers.14
Table 3: Evolution of NVLink Bandwidth by Generation
| NVLink Version | Associated GPU Architecture | Year Introduced | Data Rate per Lane | Links per GPU | Total Bidirectional Bandwidth per GPU |
| NVLink 1.0 | Pascal (P100) | 2016 | 20 Gbit/s | 6 | 300 GB/s 41 |
| NVLink 2.0 | Volta (V100) | 2017 | 25 Gbit/s | 6 | 300 GB/s 41 |
| NVLink 3.0 | Ampere (A100) | 2020 | 50 Gbit/s | 12 | 600 GB/s 38 |
| NVLink 4.0 | Hopper (H100) | 2022 | 100 Gbit/s (PAM4) | 18 | 900 GB/s 38 |
| NVLink 5.0 | Blackwell (B200) | 2024 | 100 GB/s (per link) | 18 | 1.8 TB/s 42 |
4.5 The AMD Counterpart: Infinity Fabric and the Future with AFL
AMD’s answer to the challenge of high-speed interconnects is its Infinity Fabric technology.52 Architecturally, Infinity Fabric is designed with a broader focus on heterogeneous computing, serving as a scalable interconnect not only for GPU-to-GPU communication but also for linking CPU cores, GPUs, and memory controllers within AMD’s ecosystem of EPYC processors and Instinct accelerators.52
While its unified architectural approach offers flexibility, current generations of Infinity Fabric generally provide lower raw GPU-to-GPU bandwidth compared to NVIDIA’s NVLink. For example, Infinity Fabric 3.0 in the MI300 series supports up to 896 GB/s of bidirectional bandwidth, which is competitive with NVLink 4.0, but NVIDIA’s subsequent generation has already doubled that figure.52
Looking forward, AMD is collaborating with partners like Broadcom to create a more direct competitor to the NVLink/NVSwitch ecosystem.57 The plan involves developing a new standard called Accelerated Fabric Link (AFL), which will extend the Infinity Fabric protocol over next-generation, high-lane-count PCIe switches.57 With the advent of the PCIe Gen7 standard, which promises another doubling of bandwidth, this switched fabric approach could allow AMD to build large-scale, multi-GPU systems with the kind of all-to-all connectivity that is currently NVIDIA’s key advantage in the AI infrastructure market.57
Table 4: Technical Comparison of Interconnect Technologies: NVLink vs. AMD Infinity Fabric vs. PCIe 5.0
| Feature | NVIDIA NVLink 4.0 (H100) | AMD Infinity Fabric 3.0 (MI300) | PCI Express 5.0 (x16) |
| Peak Bidirectional Bandwidth | 900 GB/s per GPU 9 | Up to 896 GB/s (GPU-GPU) 56 | 128 GB/s 9 |
| Primary Use Case | GPU-to-GPU communication in AI/HPC clusters.9 | Heterogeneous CPU-GPU and GPU-GPU communication.52 | General-purpose system component interconnect.9 |
| Architectural Focus | Optimized for massive GPU-centric parallelism.54 | Unified fabric for entire AMD CPU/GPU ecosystem.52 | Standardized I/O bus for diverse peripherals.59 |
| Latency Profile | Ultra-low; direct GPU-to-GPU path.9 | Low, but optimized for both CPU-GPU and GPU-GPU paths.52 | Higher; typically requires traversal through CPU or PCIe switch.9 |
| Ecosystem | Proprietary to NVIDIA GPUs and select partner CPUs (e.g., IBM POWER).54 | Primarily for AMD’s ecosystem, but with plans for open extension via AFL.54 | Open industry standard supported by all major hardware vendors.59 |
Section 5: Modern Workload Distribution and Parallelism Strategies
The transition from gaming to professional workloads necessitated the development of far more sophisticated strategies for distributing computation across multiple GPUs. While rendering modes like AFR and SFR were sufficient for graphics, the complex data dependencies and massive scale of AI and HPC applications required new paradigms for parallelism, supported by a robust and multi-layered software stack.
5.1 Beyond Rendering: Data Parallelism vs. Model Parallelism
In the context of training AI and machine learning models, two primary strategies for workload distribution have become dominant:
5.1.1 Data Parallelism
Data parallelism is the most common and straightforward approach to distributed training.60 The core idea is to replicate the entire AI model on each GPU in the system. The training dataset is then divided into smaller, independent mini-batches. Each GPU processes its own mini-batch simultaneously, calculating the forward pass (inference) and backward pass (gradient computation) for its slice of the data.50 After each step, the gradients computed by all GPUs are synchronized, averaged together, and used to update the model’s weights on every replica. This ensures that all model copies remain identical. This method is highly effective at speeding up training time, as it allows a much larger amount of data to be processed in parallel.14
5.1.2 Model Parallelism
Model parallelism, also known as tensor parallelism, is employed when the AI model itself is too large to fit into the memory of a single GPU.60 In this strategy, the model is partitioned, with different layers or segments of layers being placed on different GPUs.45 A single batch of data is then fed through this distributed model, with the intermediate results (activations) being passed from one GPU to the next as the computation proceeds through the network’s layers. This approach is inherently more complex and communication-intensive than data parallelism, as it requires frequent, low-latency data transfers between GPUs to pass the activations forward and the gradients backward. It is for this reason that model parallelism is highly dependent on ultra-fast interconnects like NVLink to be effective.45
5.2 The Software Stack: Orchestrating Distributed Workloads
Achieving efficient multi-GPU scaling is not merely a hardware problem; it relies on a sophisticated, multi-layered software stack where each component is optimized to work with the others. This integrated ecosystem is a key differentiator from the fragmented and poorly supported software environment of the consumer multi-GPU era.
- Programming Models (CUDA): At the foundation lies NVIDIA’s CUDA (Compute Unified Device Architecture), the parallel computing platform and programming model that first unlocked the potential of GPUs for general-purpose computing beyond graphics.14 CUDA provides developers with direct access to the GPU’s virtual instruction set and parallel computational elements.
- Frameworks (PyTorch, TensorFlow): High-level deep learning frameworks such as PyTorch and TensorFlow provide an essential layer of abstraction for developers.50 They offer simple APIs, like PyTorch’s DistributedDataParallel (DDP) or TensorFlow’s MirroredStrategy, that automate the complexities of model replication, data sharding, and gradient synchronization. This allows researchers to implement distributed training with minimal changes to their single-GPU code.60
- Communication Libraries (NCCL, Horovod): To execute the communication-intensive steps of distributed training, frameworks rely on specialized libraries. The most prominent is NVIDIA’s Collective Communications Library (NCCL).50 NCCL provides highly optimized implementations of “collective” operations, such as All-Reduce (which is used for averaging gradients), Broadcast, and All-Gather.61 Crucially, NCCL is topology-aware; it intelligently selects the most efficient communication algorithm (e.g., ring-based or tree-based) based on the underlying hardware interconnects (PCIe, NVLink, NVSwitch) to maximize bandwidth and minimize latency.61 Open-source alternatives like Horovod provide a framework-agnostic layer that can leverage NCCL or other backends like MPI to simplify multi-node training.50 This robust and standardized software stack is the critical ingredient that enables hardware like NVLink to be used effectively, a stark contrast to the ad-hoc, game-by-game profile system that failed SLI and CrossFire.
5.3 Resource Maximization: GPU Partitioning and Virtualization
The evolution of multi-GPU systems has also led to a philosophical shift in resource management, moving from simply “scaling up” a single task to also “scaling out” by maximizing the utilization of a single, powerful accelerator. This is exemplified by NVIDIA’s Multi-Instance GPU (MIG) technology.63
MIG allows a single, data-center-grade GPU, such as an A100 or H100, to be partitioned into up to seven smaller, fully independent GPU instances.63 Each MIG instance has its own dedicated, hardware-isolated set of resources, including compute cores, memory, and memory bandwidth.50 This addresses a key economic challenge in cloud and data center environments: not every workload (e.g., a small inference task) requires the full power of a flagship GPU. Leaving such a powerful and expensive resource underutilized is highly inefficient. MIG allows a cloud provider to securely and concurrently serve multiple smaller, independent workloads or tenants on a single physical GPU, guaranteeing quality of service through hardware isolation and dramatically increasing the overall resource utilization and return on investment for each accelerator.50
5.4 The System Backbone: The Evolving Role of PCI Express
While NVLink has become the dominant interconnect for high-speed, intra-node GPU communication, the PCI Express (PCIe) bus remains an essential backbone for the entire system.9 PCIe provides the critical connectivity between the GPU cluster and the other system components, including the host CPU, system RAM, high-speed storage (NVMe SSDs), and networking interface cards (NICs) for inter-node communication.59
The continuous evolution of the PCIe standard is therefore crucial for preventing system-level bottlenecks. Each new generation of PCIe effectively doubles the available bandwidth. PCIe 5.0, for example, provides up to 128 GB/s of bidirectional bandwidth over an x16 link, double that of PCIe 4.0.58 The forthcoming PCIe 6.0 standard is set to double this again to 256 GB/s, while also introducing more advanced signaling techniques like PAM4 to maintain signal integrity.58 This escalating bandwidth is vital for feeding the massive datasets required by modern AI workloads from storage to the GPU memory and for enabling high-speed communication between different server nodes in a large cluster.64
Section 6: Conclusion – The Future of Multi-GPU Computing
The trajectory of multi-GPU technology illustrates a classic case of technological evolution driven by a fundamental shift in market demands. The initial paradigm, focused on enhancing consumer gaming, ultimately failed due to a combination of flawed rendering methodologies, an unsustainable software support model, and an unfavorable cost-benefit ratio. This failure, however, paved the way for a professional renaissance, where the acute needs of HPC and AI spurred the development of new architectures that prioritized data throughput and memory coherence over simple frame-rate scaling.
6.1 Synthesis of the Paradigm Shift
The journey from SLI/CrossFire to NVLink is not a linear progression but a complete redefinition of purpose. The consumer-era technologies were constrained by the VRAM mirroring of AFR, plagued by the poor user experience of micro-stuttering, and ultimately abandoned by a software ecosystem that had no economic incentive to support them. The limitations of this approach—particularly the crippling communication bottlenecks of the PCIe bus and the inability to pool memory—directly informed the design of NVLink.
The modern data-center-centric paradigm, built on NVLink and its associated NVSwitch fabric, succeeded precisely where its predecessors failed. It delivered orders-of-magnitude increases in bandwidth, introduced a revolutionary unified memory model that simplified programming and enabled massive models, and was supported by a robust, standardized software stack (CUDA, NCCL, PyTorch/TensorFlow) that allowed its capabilities to be efficiently harnessed. This shift transformed multi-GPU from a niche enthusiast hobby into the foundational architecture of the modern AI revolution.
6.2 Future Trajectories and Emerging Challenges
Looking ahead, the evolution of multi-GPU computing is set to continue at a relentless pace, driven by the exponential growth in the complexity of AI models and scientific computations. Several key trends will define the next era of this technology.
- Interconnect Evolution: The scaling of interconnect bandwidth remains a primary focus. NVIDIA’s fifth-generation NVLink already delivers 1.8 TB/s of bandwidth per GPU with its Blackwell architecture, and future generations will continue to push this boundary.42 Concurrently, the broader industry is moving towards more open standards, with AMD’s plans to create a competitive switched fabric through its Accelerated Fabric Link (AFL) over PCIe Gen7 signaling a potential shift towards a more diverse and competitive high-performance interconnect market.57
- CPU-GPU Integration and the Rise of the “Superchip”: The future of system architecture is converging on a “superchip” model, where the traditional boundaries between discrete components are dissolving. Products like NVIDIA’s Grace Hopper Superchip, which integrates a high-performance CPU and a powerful GPU on a single package connected by a coherent, high-bandwidth NVLink-C2C interconnect, represent this future.38 This design effectively eliminates the historical CPU-to-GPU bottleneck of the PCIe bus, creating a single, unified memory domain at the chip level. This trend suggests a future where a computational “node” is no longer a collection of parts on a motherboard but a highly integrated package of processors that functions as a single, coherent unit.
- The Enduring Software Challenge: As hardware scales to clusters containing tens of thousands of GPUs, the most significant challenges will increasingly be in software.61 The primary architectural boundary is shifting from connecting components within a node to connecting these powerful “superchip” nodes to each other using external fabrics like InfiniBand or next-generation NVSwitch systems.38 Effectively programming and managing these exascale systems requires the development of more advanced algorithms, resource schedulers, and communication libraries that can orchestrate work across this vast sea of processors without being crippled by synchronization overhead, network latency, or fault tolerance issues.50 The continued advancement of multi-GPU computing will depend as much on innovations in software and systems-level design as it does on the raw performance of the next generation of silicon.