1. Introduction: The Paradox of Overparameterization
In the contemporary landscape of deep learning, a singular, pervasive dogma has dictated the design of neural architectures: scale is the primary driver of performance. From the early success of AlexNet to the recent dominance of Large Language Models (LLMs) boasting hundreds of billions of parameters, the field has operated under the assumption that massive overparameterization is a prerequisite for successful optimization. This paradigm posits that a vast excess of parameters—far exceeding the information theoretic content of the training data—is required to smooth the non-convex loss landscape, preventing Stochastic Gradient Descent (SGD) from stagnating in suboptimal local minima. Consequently, the computational cost of training and inference has grown exponentially, creating a significant barrier to deployment in resource-constrained environments and raising fundamental questions about the efficiency of biological versus artificial intelligence.
However, this foundational assumption was challenged by the formulation of the Lottery Ticket Hypothesis (LTH) by Frankle and Carbin in 2019. Their seminal work presented a counter-intuitive empirical finding: dense, randomly-initialized, feed-forward networks contain sparse subnetworks—termed “winning tickets”—that, when trained in isolation, reach test accuracies comparable to, and often exceeding, the original dense network in a similar number of iterations.1 This hypothesis reframes the role of overparameterization not as a necessity for representation, but as a necessity for initialization. It suggests that the dense network functions as a vast combinatorial search space from which the optimizer identifies a highly efficient, sparse topology capable of solving the task.1
The implications of the LTH are profound and multifaceted. If the dense training phase is merely a mechanism for architectural search, could the computational waste of overparameterization be circumvented entirely? Does the existence of these subnetworks imply that neural networks are not learning distributed representations in the way previously thought, but are instead converging on specific, sparse functional circuits? This report provides an exhaustive analysis of the Lottery Ticket Hypothesis, dissecting its theoretical underpinnings, the mechanisms of subnetwork discovery, the stability of optimization trajectories, and the translation of these principles to modern architectures like Vision Transformers (ViTs) and Large Language Models. We explore the transition from “weak” lottery tickets to “strong” supermasks, the intersection with Neural Architecture Search (NAS), and the practical challenges of the “Hardware Lottery” that dictates the viability of sparse computing.
1.1 Defining the Hypothesis and Its Variants
The rigorous formulation of the Lottery Ticket Hypothesis considers a dense neural network $f(x; \theta)$ with initial parameters $\theta_0 \sim \mathcal{D}_{\theta}$. The hypothesis asserts the existence of a binary mask $m \in \{0, 1\}^{|\theta|}$ such that the subnetwork $f(x; m \odot \theta_0)$—initialized with the same specific random weights as the dense network—can be trained to a performance $\mathcal{A}_{sub}$ such that $\mathcal{A}_{sub} \geq \mathcal{A}_{dense}$, using a comparable optimization budget.1
Crucially, this performance is contingent on the combination of the mask $m$ and the specific initialization $\theta_0$. If the mask $m$ is applied to a different random initialization $\theta’_0$, the subnetwork typically fails to train or converges to a significantly lower accuracy. This dependence indicates that the “winning ticket” is not merely a robust architecture, but a specific alignment between the network topology and the initial weights in the optimization landscape.2
As research has progressed, the LTH has evolved into several distinct interpretations, each with unique implications for optimization theory:
- The Weak LTH: This refers to the original formulation where winning tickets are identified retrospectively via pruning and must be retrained from their specific initial values to achieve matching performance. This view emphasizes the “initialization lottery”.4
- The Strong LTH: This stronger conjecture posits that sufficiently overparameterized networks contain subnetworks that perform well at initialization, without any gradient updates. In this view, the “training” process is entirely replaced by the selection of a subnetwork (masking), effectively finding a functional subgraph within the random noise.4
- The Generalized LTH: This variant suggests that winning tickets capture inductive biases that are transferable across datasets and tasks. A ticket found on a large dataset (like ImageNet) provides a “universal” sparse backbone that can be fine-tuned for disparate downstream tasks, decoupling the architectural search from the specific target distribution.8
2. The Mechanics of Discovery: Iterative Magnitude Pruning (IMP)
The primary methodological tool for uncovering winning tickets is Iterative Magnitude Pruning (IMP). While computationally expensive—often requiring training the full dense network multiple times—IMP serves as the “existence proof” generator for the LTH, consistently finding subnetworks that simpler methods fail to identify. Understanding the granular mechanics of IMP is essential for interpreting why standard pruning techniques often result in difficult-to-train models.
2.1 The Algorithmic Framework of IMP
IMP operates on the heuristic that weight magnitude is a robust proxy for importance. While second-order methods like Optimal Brain Damage (OBD) or Hessian-based pruning offer theoretically superior selection criteria, magnitude pruning has proven remarkably effective and stable in the context of the LTH.10 The procedure unfolds in a cyclical manner:
- Initialization: The network is initialized with parameters $\theta_0$.
- Training: The network is trained for $T$ iterations to reach parameters $\theta_T$.
- Pruning: A fraction $p$ (e.g., 20%) of the weights with the lowest magnitudes in $\theta_T$ are masked out (set to zero).
- Rewinding (The Critical Step): The remaining unpruned weights are reset to their values in $\theta_0$.
- Iteration: Steps 2-4 are repeated until the desired sparsity level is reached.
The distinction between One-Shot Pruning (pruning to the target sparsity in a single step) and Iterative Pruning is non-trivial. Experimental evidence consistently demonstrates that iterative pruning identifies “winning tickets” at significantly higher sparsity levels (e.g., 90-95%) than one-shot approaches.2 The iterative process likely allows the network to gradually adapt its topology to the loss landscape, “annealing” the architecture into a global optimum that is inaccessible via a sudden reduction in capacity.
2.2 The Importance of Initialization: $\theta_0$ vs. Random Reinitialization
A defining characteristic of a “winning ticket” is its sensitivity to initialization. To validate the LTH, researchers conduct a control experiment where the discovered mask $m$ is applied to a new random initialization $\theta’_0$. In almost all cases, the performance of the reinitialized subnetwork $f(x; m \odot \theta’_0)$ is significantly inferior to the winning ticket $f(x; m \odot \theta_0)$.2
This disparity highlights a critical insight: the topology of the sparse network alone explains only part of the performance. The remaining “magic” lies in the specific initial values of the weights. The weights that survive the pruning process are those that, during the initial dense training, moved effectively to reduce loss. By resetting them to $\theta_0$, IMP preserves the “potential energy” of these specific connections—their favorable position in the optimization landscape relative to the loss basin.3
Zhou et al. (2019) extended this analysis by investigating “Deconstructing Lottery Tickets.” Their findings suggest that for many tasks, preserving the exact magnitude of $\theta_0$ is less critical than preserving the sign of the weights. If a winning ticket is reinitialized such that the signs match $\theta_0$ but the magnitudes are constant (or re-sampled), the network often retains its trainability. This implies that the mask effectively selects a specific “orthant” in the parameter space, and the geometry of the optimization landscape is largely defined by these sign configurations.15
Table 1: Performance Comparison of Pruning Strategies on CIFAR-10 (ResNet-18)
| Pruning Strategy | Initialization | Training Method | Sparsity Limit (Acc Maintenance) | Key Observation |
| Standard Pruning | Random Re-init | Fine-tuning | ~80% | Requires dense training first; efficient inference only. |
| Winning Ticket (IMP) | Original $\theta_0$ | From Scratch (Reset) | ~90-95% | Matches dense accuracy; trains efficiently from start. |
| Random Ticket | Random Re-init | From Scratch | ~70-80% | Fails at high sparsity; topology alone is insufficient. |
| Sign-Preserved Ticket | Constant Sign($\theta_0$) | From Scratch | ~90% | Sign is the dominant factor in initialization quality. |
3. The Stability Gap and the Necessity of Late Rewinding
The original LTH findings were primarily validated on smaller datasets (MNIST, CIFAR-10) and shallower networks. When researchers attempted to scale the hypothesis to ResNet-50 on ImageNet or Transformer models, a significant anomaly emerged: winning tickets found at initialization $\theta_0$ failed to outperform random pruning. This limitation led to the discovery of Late Rewinding, a crucial modification that has since become standard for scaling the LTH.13
3.1 Instability Analysis and SGD Noise
The failure of $\theta_0$ in large-scale settings is attributed to the inherent instability of neural network training in its earliest phases. Frankle et al. (2020) conducted rigorous “instability analyses,” demonstrating that the optimization trajectory of large networks is highly sensitive to stochastic noise (e.g., data ordering, augmentation) during the first few epochs.19
If two copies of a dense network are initialized with the same $\theta_0$ but trained with different SGD noise seeds, their final weights will diverge significantly. Crucially, they diverge into different basins of attraction that are not Linearly Mode Connected (LMC). This means that interpolating between the two final solutions results in a barrier of high loss. Because the network’s final destination is determined by stochastic noise after initialization, the mask $m$ derived from the end of training is uncorrelated with the specific values at $\theta_0$. The mask reflects a destination that $\theta_0$ had not yet “committed” to reaching.21
3.2 The Mechanism of Late Rewinding
Late Rewinding addresses this by resetting the weights not to $\theta_0$, but to $\theta_k$, the state of the network at epoch $k$ (typically 0.1% to 5% into the training process). By epoch $k$, the network has undergone a phase transition from chaotic exploration to stable optimization.
- The Stability Gap: The period between epoch 0 and epoch $k$ is the “stability gap.” During this time, the network selects a specific linearly connected mode (a broad basin in the loss landscape).
- Mode Locking: Once the network reaches $\theta_k$, the final outcome is largely determined up to linear interpolation. Regardless of subsequent SGD noise, the network will converge to a solution within the same basin.19
By rewinding to $\theta_k$, IMP ensures that the mask $m$ (derived from $\theta_T$) and the weights $\theta_k$ are aligned within the same optimization basin. This modification allows the LTH to hold for virtually any architecture, including ResNet-50 on ImageNet and BERT on NLP tasks.13 The “lottery” for large networks is not won at initialization, but rather in the first few thousand iterations of training.
4. Pruning at Initialization (PaI) and the “Sanity Check” Crisis
The existence of winning tickets raises a tantalizing practical possibility: if we could identify the mask $m$ before training, we could bypass the expensive dense training phase entirely, reducing the computational cost of Deep Learning by an order of magnitude. This goal spawned the sub-field of Pruning at Initialization (PaI), which seeks “Zero-Cost Proxies” to predict weight importance at step zero.23
4.1 Zero-Cost Proxies: SNIP, GraSP, and SynFlow
PaI methods rely on computing a saliency score for each weight using a single forward/backward pass. The underlying assumption is that the gradient signals at initialization contain sufficient information to identify the trainable subnetwork.
- SNIP (Single-shot Network Pruning): Proposes that important connections are those with the highest “connection sensitivity,” defined as the magnitude of the loss gradient with respect to the weight: $S_w = | \frac{\partial L}{\partial w} \odot w |$. SNIP aims to preserve weights that, if removed, would cause the largest spike in loss.23
- GraSP (Gradient Signal Preservation): Critique SNIP for focusing on the magnitude of the loss rather than the trainability of the network. GraSP uses the Hessian-gradient product to preserve the gradient flow, aiming to maximize the reduction of loss over future iterations rather than just the instantaneous loss.23
- SynFlow (Synaptic Flow): Addresses the “layer collapse” issue where gradient-based methods might prune entire layers, rendering the network untrainable. SynFlow computes a score based on the product of weights along a path, iteratively conserving the total “flow” of signal through the network without referencing any training data (using an all-ones input).24
4.2 The “Sanity Check” Crisis
Despite the theoretical elegance of PaI, empirical scrutiny has revealed significant flaws in these methods. A landmark paper by Su et al. (2020) and subsequent work by Frankle et al. (2021) performed “Sanity Checks” that fundamentally undermined the claims of many PaI algorithms.29
The researchers performed a simple randomization test: taking the mask generated by a method like SNIP and randomly shuffling the weights within each layer. If the method were truly identifying a specific, critical topology (i.e., “this specific weight connects feature A to feature B”), then shuffling the mask should destroy performance.
The results were startling:
- For SNIP and GraSP, shuffling the mask resulted in negligible performance loss. In some cases, the shuffled mask performed better than the computed mask.
- This indicates that these methods were not identifying a specific topology. Instead, they were merely acting as sparsity schedulers—calculating the optimal fraction of weights to keep in each layer, but not which weights.24
- SynFlow showed slightly more sensitivity to shuffling, but still failed to match the performance of IMP tickets, particularly at high sparsities.
- In stark contrast, IMP winning tickets failed catastrophically when shuffled, confirming that IMP identifies a genuine, structurally specific circuit.17
These findings suggest that the information available at initialization (gradients, Hessians) is largely insufficient to predict the complex optimization dynamics of deep training. The “winning ticket” is determined by the trajectory of training, which PaI methods fundamentally ignore.
4.3 Convergence with Neural Architecture Search (NAS)
While PaI methods faltered as standalone pruning algorithms, they found a second life in Neural Architecture Search (NAS). In the “Zero-Cost NAS” paradigm, metrics like SynFlow are used not to prune a single network, but to rank thousands of candidate architectures in a search space. Even if imperfect, these metrics correlate well with final accuracy, allowing researchers to filter out poor architectures without training.33
Frameworks like ProxyBO (Bayesian Optimization with Proxies) utilize these scores to accelerate the search for optimal topologies by orders of magnitude. This convergence underscores a key insight: the LTH and NAS are describing the same underlying phenomenon—the search for a subgraph structure that aligns with initialization to facilitate efficient gradient descent.35
5. Transferability and the Inductive Bias of Winning Tickets
If the LTH implies that training discovers a specific optimal topology, a natural question follows: Is this topology specific to the training data, or does it encode a general inductive bias? Research into the transferability of winning tickets suggests the latter, pointing toward the existence of “Universal Tickets.”
5.1 One Ticket to Win Them All
Morcos et al. (2019) investigated whether winning tickets found on one dataset could transfer to another. Their experiments revealed that tickets discovered on large, diverse datasets (like ImageNet) transfer remarkably well to smaller datasets (CIFAR-10, Fashion-MNIST), often outperforming tickets found directly on the target dataset.8
This phenomenon suggests that the winning ticket encodes a generic visual inductive bias. Just as the early layers of CNNs learn Gabor-like filters that are useful for all visual tasks, the sparse topology of an ImageNet ticket captures the structural connectivity required to represent these fundamental visual features.37 The “Universal Ticket” is essentially a better backbone than a dense network because it has already stripped away the redundant capacity that leads to overfitting on small data.
5.2 Universal Tickets in Natural Language Processing
In the domain of NLP, the transferability is even more pronounced. Chen et al. (2020) demonstrated that subnetworks found within pre-trained BERT models on the Masked Language Modeling (MLM) task transfer universally to downstream tasks like GLUE and SQuAD.39
This finding has significant implications for the lifecycle of Large Language Models. It suggests that the pre-training phase serves to “mine” the lottery, identifying a robust sparse structure capable of general language understanding. Fine-tuning is then merely the adaptation of weights within this established topology. This supports a “Pre-train, then Prune” paradigm, where a single universal ticket is deployed for multiple downstream applications, offering a path to efficient “Foundation Model” deployment.40
5.3 Disentangled Lottery Tickets (DiLT)
Recent work has refined the transferability concept through the Disentangled Lottery Ticket (DiLT) hypothesis. This framework proposes that a winning ticket mask is composed of two distinct components:
- The Core Ticket: A task-agnostic subgraph that encodes general features (e.g., edge detectors, syntax). This is the intersection of masks found on disjoint data partitions.
- The Specialist Ticket: A task-specific subgraph that encodes features unique to a specific distribution.
By isolating the “Core” ticket, researchers can create modular sparse networks that are highly transferable, while “Specialist” tickets can be swapped in for specific domains, resembling a modular “mixture of experts” approach at the topological level.42
6. Dynamic Sparse Training (DST): Rigging the Lottery
The primary criticism of the LTH is practical: finding a winning ticket via IMP is more expensive than standard training. If we must train the dense model to find the ticket, we haven’t saved any training compute. Dynamic Sparse Training (DST) seeks to resolve this paradox by maintaining a sparse network throughout the entire training process, dynamically updating the topology to find the winning ticket “on the fly”.43
6.1 The RigL Algorithm
The state-of-the-art in DST is RigL (Rigged Lottery), proposed by Evci et al. (2020). RigL avoids the dense pre-training step entirely. It starts with a random sparse network and periodically updates the topology using a “Drop-and-Grow” mechanism:
- Drop: Prune a fraction of weights with the smallest magnitudes (removing unimportant connections).
- Grow: Activate new connections based on the gradients of the zero-valued weights. If a pruned connection has a high gradient, it indicates that the loss function is sensitive to that connection, and it should be re-grown.44
By using gradient information to guide the growth phase, RigL effectively searches the super-network space for the winning ticket without ever instantiating the full dense model. RigL matches the performance of dense networks and IMP tickets while using a fraction of the FLOPs, realizing the dream of training sparse networks from scratch.43
6.2 Structured RigL (SRigL) and Hardware Acceleration
A major limitation of standard RigL is that it produces unstructured sparsity—random patterns of zeros that are notoriously difficult to accelerate on standard hardware (GPUs/TPUs). This issue, known as the “Hardware Lottery” (discussed in Section 8), renders theoretical FLOP reductions useless for wall-clock speedup.
Structured RigL (SRigL) adapts the algorithm to enforce N:M sparsity (e.g., 2:4 sparsity, where every block of 4 weights has at least 2 zeros). By constraining the “grow” step to respect these hardware-friendly patterns, SRigL achieves the inference speedups of structured pruning with the accuracy benefits of dynamic topology search. Recent benchmarks demonstrate that SRigL can achieve 3.4x inference speedups on CPUs and 1.7x on GPUs compared to dense baselines, effectively bridging the gap between theoretical LTH findings and practical deployment.43
7. The Strong Lottery Ticket Hypothesis: Pruning Is Training
While the “Weak” LTH focuses on finding subnetworks that are trainable, the Strong LTH proposes a more radical idea: sufficiently overparameterized networks contain subnetworks that perform well without any weight training at all.
7.1 Supermasks and Edge-Popup
Ramanujan et al. (2020) and Zhou et al. (2019) pioneered the search for Supermasks. By freezing the random weights $\theta_0$ and optimizing only the binary mask $m$ (using a method called “Edge-Popup”), they demonstrated that one can find subnetworks with accuracy far better than chance—sometimes matching trained networks.4
In this regime, the weights are merely distinct random values available for selection. The optimizer searches for a path through these random values that approximates the target function. This aligns with the “Edge of Chaos” theory in dynamical systems. Deep networks at initialization are poised between order (vanishing gradients) and chaos (exploding gradients). The Supermask algorithm extracts a signal propagation path that stays on this “critical line,” allowing information to propagate deeply without dissipation. This suggests that “learning” in deep networks is partly about discovering these naturally resonant paths within the random substrate.48
7.2 Theoretical Guarantees
Recent theoretical work has proven the Strong LTH for various architectures. Malach et al. (2020) proved that a random network of depth $2L$ and width polynomial in $d$ can approximate any target network of depth $L$ with high probability.4 This confirms that the “lottery” is statistically guaranteed to possess a winning ticket given sufficient overparameterization. The random weights act as a basis set; if the basis is large enough, a subset sum can approximate any target vector.5
8. The Hardware Lottery: Unstructured vs. Structured Sparsity
A recurring theme in the LTH literature is the disconnect between theoretical sparsity (reduction in FLOPs) and practical acceleration (reduction in latency). This phenomenon, termed the “Hardware Lottery” by Sara Hooker, posits that research ideas succeed not just on merit but on compatibility with dominant hardware architectures.52
8.1 The Failure of Unstructured Pruning
Most winning tickets found by IMP are unstructured: the zeros are scattered randomly throughout the weight matrices. On SIMD (Single Instruction, Multiple Data) hardware like GPUs, this irregularity prevents efficient memory access and parallelization. A GPU warp must wait for the slowest thread; if one weight is non-zero, the hardware performs the computation for all. Consequently, a 90% sparse unstructured model often runs slower than a dense model due to the overhead of sparse matrix indices and cache misses.52
8.2 Winning with N:M Sparsity
To win the hardware lottery, LTH research has pivoted toward N:M Structured Sparsity (specifically 2:4 sparsity supported by NVIDIA Ampere A100/H100 Tensor Cores). This pattern requires that in every block of 4 contiguous weights, at least 2 are zero. This regularity allows for dedicated hardware acceleration, effectively doubling throughput.55
Research into Structured Winning Tickets shows that while forcing structure constraints restricts the search space (potentially excluding the “optimal” ticket), the performance gap can be closed using dynamic training methods (like SRigL) or “best-combination” learning.54 This represents a pragmatic evolution of the LTH: the winning ticket is no longer the absolute best subnetwork, but the best subnetwork that fits the hardware constraints.
9. LTH in the Era of Large Language Models (LLMs)
The advent of LLMs (GPT-3, LLaMA) has introduced a new constraint: retraining is often impossible due to compute costs. The classic IMP loop (Train $\to$ Prune $\to$ Retrain) is infeasible for models with 70B+ parameters. This has shifted the focus to Post-Training Pruning (PTP)—finding tickets in fully trained models without subsequent retraining.
9.1 SparseGPT: Second-Order Pruning at Scale
SparseGPT adapts the classic Optimal Brain Surgeon (OBS) approach to massive scale. It solves the layer-wise reconstruction problem: finding a sparse mask and updated weight values that minimize the error in layer output relative to the dense model.57 SparseGPT can prune LLaMA-65B to 50% sparsity in a few hours on a single GPU with minimal accuracy loss. While technically not a “retraining” method, SparseGPT effectively locates a “winning configuration” in the local neighborhood of the pre-trained weights, validating that the dense representation is highly redundant.57
9.2 Wanda: Pruning by Weights and Activations
Wanda (Pruning by Weights and Activations) challenges the magnitude-based pruning metric used in classic LTH. Sun et al. (2023) observed that in LLMs, feature activations often contain outliers with massive magnitudes. Pruning weights solely based on weight magnitude ($|W|$) ignores the fact that a small weight multiplied by a huge activation can have a significant impact on the output.
Wanda prunes weights based on the product $|W| \cdot \|A\|$, where $\|A\|$ is the norm of the input activations. This simple metric, requiring no gradient updates, achieves state-of-the-art pruning results for LLaMA and other LLMs.60 The success of Wanda suggests that in the LLM regime, the definition of a “ticket” must account for the input distribution (activations) more explicitly than in vision models. The winning subnetwork is defined by its interaction with the data manifold, not just its static weight topology.62
9.3 Lottery Ticket Adaptation (LoTA)
Bridging LTH and Parameter-Efficient Fine-Tuning (PEFT), Lottery Ticket Adaptation (LoTA) proposes fine-tuning only a sparse subnetwork for downstream tasks. Chen et al. (2024) demonstrate that identifying a sparse, task-specific subnetwork prevents catastrophic forgetting and enables multi-task merging.64 This reinforces the “Universal Ticket” concept: the pre-trained LLM contains multiple overlapping sparse subnetworks, each capable of solving a different task. LoTA effectively activates the relevant ticket for the task at hand, offering a more efficient alternative to Low-Rank Adaptation (LoRA).66
10. Conclusion and Future Outlook
The Lottery Ticket Hypothesis has evolved from a surprising empirical observation into a rigorous framework for understanding neural network topology. The evidence overwhelmingly supports the existence of sparse subnetworks that match full model performance, fundamentally altering our understanding of optimization.
The hypothesis implies that overparameterization is a mechanism for exploration, allowing SGD to find a stable sparse manifold (the winning ticket) early in training. Once this structure is found, the massive parameter count becomes redundant. The failure of simple Pruning-at-Initialization methods highlights that this structure is not evident in the static gradients at step zero, but emerges from the complex dynamics of the early training phase (the stability gap).
Looking forward, the integration of Dynamic Sparse Training (like SRigL) with Hardware-Aware Structures (N:M sparsity) promises to realize the actual efficiency gains that LTH has long promised. In the era of Foundation Models, the LTH manifests as Lottery Ticket Adaptation, suggesting that the future of AI is not in training ever-larger dense models, but in mastering the art of activating the correct sparse circuits within them. The “Lottery” is no longer a game of chance, but a solvable search problem.
Table 2: Summary of Key LTH Technologies and Their Impact
| Technology | Core Mechanism | Primary Benefit | Limitation |
| IMP | Train, Prune, Reset | Finds best tickets (Gold Standard) | Extremely expensive (Retraining) |
| Late Rewinding | Reset to Epoch $k$ | Scales LTH to ResNet/BERT | Requires “stability gap” analysis |
| PaI (SNIP/SynFlow) | Gradient/Flow scoring | Zero-cost discovery | Fails sanity checks (Low precision) |
| RigL / DST | Drop-and-Grow | Sparse training from scratch | Unstructured sparsity (slow on GPU) |
| SRigL | N:M Structured constraints | Hardware acceleration | Slightly restricted search space |
| Wanda / SparseGPT | Weight $\times$ Activation | Post-training pruning for LLMs | Focuses on inference, not training |
| LoTA | Sparse Fine-tuning | Multi-task adaptation | Task-specific mask storage |
The convergence of these diverse streams of research—from theoretical mean-field analysis to practical GPU kernel design—indicates that sparsity is not merely a compression technique, but a fundamental property of learnable systems. We are moving away from the brute-force lottery of dense initialization toward a future of intelligent, sparse architectural design.