The theoretical understanding of deep neural networks has undergone a fundamental transformation over the last decade. Historically, statistical learning theory relied on concepts such as the Vapnik-Chervonenkis (VC) dimension and Rademacher complexity to explain the generalization capabilities of machine learning models. These classical frameworks suggested a trade-off between model complexity and generalization: as the number of parameters increases, the model’s capacity to fit the training data improves, but the risk of overfitting\u2014memorizing noise rather than learning signal\u2014escalates correspondingly. This view implies a “sweet spot” of model complexity, beyond which test performance should degrade.<\/span>1<\/span><\/p>\n However, the empirical reality of modern deep learning stands in stark contradiction to this classical wisdom. State-of-the-art neural networks are massively overparameterized, often possessing parameters numbering in the billions or trillions, far exceeding the number of training samples available. Yet, rather than overfitting, these models exhibit a phenomenon known as “double descent”: as model size increases beyond the interpolation threshold (where training error reaches zero), test error frequently continues to decrease, defying the U-shaped curve predicted by traditional bias-variance trade-offs.<\/span>1<\/span><\/p>\n To resolve this paradox, the theoretical community turned to asymptotic analysis, specifically studying the behavior of neural networks as their width\u2014the number of neurons in hidden layers\u2014approaches infinity. This “infinite-width limit” serves a role analogous to the thermodynamic limit in statistical physics: by taking the number of components to infinity, individual fluctuations average out, revealing deterministic macroscopic laws that govern the system’s behavior.<\/span>3<\/span><\/p>\n This line of inquiry led to the discovery of the Neural Tangent Kernel (NTK) in 2018, a mathematical object that describes the training dynamics of infinitely wide networks as a linear evolution in function space.<\/span>4<\/span> The NTK framework provided the first rigorous proof that massive, overparameterized networks can be trained to global optimality. It established a duality between training neural networks with gradient descent and performing kernel ridge regression, suggesting that deep learning could be understood through the lens of kernel methods.<\/span>4<\/span><\/p>\n Despite its mathematical elegance, the NTK regime\u2014often termed “lazy training”\u2014failed to capture the arguably most critical aspect of deep learning: the ability to learn data-dependent features. Empirical evidence demonstrated a significant performance gap between NTK predictors and finite-width networks on complex tasks like ImageNet, suggesting that practical networks operate in a different regime.<\/span>5<\/span> This realization catalyzed the development of alternative scaling theories, most notably the Maximal Update Parameterization ($\\mu$P) and the Mean-Field limit, which preserve feature learning and representation alignment even as width approaches infinity.<\/span>7<\/span><\/p>\n This report provides an exhaustive analysis of these regimes. It details the mathematical foundations of the Gaussian Process (GP) limit at initialization, the freezing of the kernel in the NTK regime, and the specific scaling laws required to unlock feature learning. It further explores the application of these theories to modern architectures, including Residual Networks (ResNets) and Transformers, identifying critical scaling adjustments (such as Depth-$\\mu$P and attention head scaling) necessary to maintain stability and expressivity in deep, large-scale models.<\/span><\/p>\n Before analyzing the dynamics of training, it is essential to understand the statistical properties of neural networks at initialization. The foundational link between wide neural networks and probabilistic methods was established by Radford Neal in 1994, fundamentally shifting the perspective from geometric to probabilistic analysis.<\/span>3<\/span><\/p>\n Neal proved that a single-hidden-layer neural network with random weights converges to a Gaussian Process (GP) as the number of hidden units approaches infinity.<\/span>3<\/span> The intuition relies on the Central Limit Theorem (CLT). Consider a neuron in the first hidden layer. Its pre-activation value is a weighted sum of the inputs. If the weights are independent and identically distributed (i.i.d.) with zero mean and finite variance, and the number of inputs is large, the pre-activation distribution approaches a Gaussian.<\/span><\/p>\n When we extend this to a network with an infinite number of hidden units, the output of the network becomes a sum of infinitely many i.i.d. variables (the contributions of each hidden neuron). Consequently, the distribution of functions represented by the network at initialization converges to a Gaussian Process. This means that for any finite set of input points $\\{x_1, \\dots, x_k\\}$, the joint distribution of the network outputs $\\{f(x_1), \\dots, f(x_k)\\}$ is a multivariate Gaussian distribution.<\/span>3<\/span><\/p>\n This result implies that, at initialization, a wide network is simply a random field defined entirely by its mean function (usually zero) and its covariance function (kernel). This “Neural Network Gaussian Process” (NNGP) allows researchers to perform exact Bayesian inference with infinite-width networks without ever training a finite model, simply by computing the kernel and applying standard GP regression formulas.<\/span>10<\/span><\/p>\n For deep networks, the Gaussian behavior propagates through the layers. This can be formalized recursively. Let $n_l$ be the width of layer $l$. As $n_l \\to \\infty$ sequentially (or simultaneously), the pre-activations $z^{(l)}$ at each layer behave as Gaussian processes.<\/span>3<\/span><\/p>\n The covariance kernel $\\Sigma^{(l)}(x, x’)$ at layer $l$ describes the similarity between the network’s representations of two inputs $x$ and $x’$. For a standard Multilayer Perceptron (MLP) with nonlinearity $\\sigma$, the kernel propagates as follows:<\/span><\/p>\n This recursive formula reveals that the covariance at layer $l+1$ is determined by the expected product of the activations from layer $l$.<\/span>3<\/span> This propagation allows for the analytical computation of the NNGP kernel for arbitrary depths, effectively solving the “forward pass” of the infinite network in closed form.<\/span><\/p>\n The properties of the NNGP kernel at deep layers depend critically on the choice of initialization variances ($\\sigma_w^2, \\sigma_b^2$) and the nonlinearity $\\sigma$. Research has identified an “Order-to-Chaos” phase transition in deep networks.<\/span>13<\/span><\/p>\n This analysis of the NNGP provides the static picture. However, deep learning is fundamentally about <\/span>dynamics<\/span><\/i>\u2014how the network changes to fit the data. This requires the introduction of the Neural Tangent Kernel.<\/span><\/p>\n While the GP limit describes the network <\/span>before<\/span><\/i> training, the Neural Tangent Kernel (NTK) describes how it evolves <\/span>during<\/span><\/i> training. Introduced by Jacot et al. in 2018, the NTK framework was a breakthrough because it extended the infinite-width analysis to the optimization trajectory of gradient descent.<\/span>4<\/span><\/p>\n Consider a neural network $f(x; \\theta)$ parameterized by a vector $\\theta \\in \\mathbb{R}^P$. We train this network to minimize a loss function $\\mathcal{L}$ using gradient descent with a learning rate $\\eta$:<\/span><\/p>\n <\/p>\n $$\\theta_{t+1} = \\theta_t – \\eta \\nabla_\\theta \\mathcal{L}(\\theta_t)$$<\/span><\/p>\n In the continuous time limit (gradient flow), the evolution of the parameters is $\\frac{d\\theta}{dt} = -\\nabla_\\theta \\mathcal{L}$. Crucially, we are often interested not in the parameters themselves, but in the evolution of the network’s output $f(x; \\theta)$ in function space. Using the chain rule, the time derivative of the output for an input $x$ is:<\/span><\/p>\n <\/p>\n $$\\frac{df(x; \\theta_t)}{dt} = \\nabla_\\theta f(x; \\theta_t)^T \\frac{d\\theta_t}{dt} = – \\nabla_\\theta f(x; \\theta_t)^T \\nabla_\\theta \\mathcal{L}$$<\/span><\/p>\n If we consider the Mean Squared Error loss over a dataset, the gradient $\\nabla_\\theta \\mathcal{L}$ is a sum over training points $x_j$:<\/span><\/p>\n <\/p>\n $$\\nabla_\\theta \\mathcal{L} = \\sum_{j=1}^N (f(x_j; \\theta_t) – y_j) \\nabla_\\theta f(x_j; \\theta_t)$$<\/span><\/p>\n Substituting this back, we obtain the evolution equation for the function:<\/span><\/p>\n <\/p>\n $$\\frac{df(x; \\theta_t)}{dt} = – \\sum_{j=1}^N \\underbrace{\\langle \\nabla_\\theta f(x; \\theta_t), \\nabla_\\theta f(x_j; \\theta_t) \\rangle}_{\\Theta_t(x, x_j)} (f(x_j; \\theta_t) – y_j)$$<\/span><\/p>\n The term $\\Theta_t(x, x’) = \\langle \\nabla_\\theta f(x; \\theta_t), \\nabla_\\theta f(x’; \\theta_t) \\rangle$ is the <\/span>Neural Tangent Kernel<\/b> at time $t$.<\/span>4<\/span> It is a symmetric, positive semi-definite kernel that governs the dynamics of the network outputs.<\/span><\/p>\n The most profound discovery of NTK theory is the behavior of $\\Theta_t$ as the network width $n$ tends to infinity.<\/span><\/p>\n This “frozen kernel” property arises because in a sufficiently wide network, individual weights need to change only by an infinitesimal amount $O(1\/\\sqrt{n})$ to effect a finite change in the output. Since the weights barely move, the feature map $\\nabla_\\theta f(x; \\theta)$ (which depends on the weights) also remains approximately constant.<\/span>1<\/span><\/p>\n Because the kernel $\\Theta_t$ is constant ($\\Theta_t \\approx \\Theta_\\infty$), the training dynamics become linear. The differential equation governing the output becomes:<\/span><\/p>\n <\/p>\n $$\\frac{df_t(x)}{dt} = – \\eta \\sum_{j=1}^N \\Theta_\\infty(x, x_j) (f_t(x_j) – y_j)$$<\/span><\/p>\n This is a linear ordinary differential equation (ODE) that can be solved analytically. The solution path $f_t(x)$ is identical to the path taken by gradient descent on a linear model with fixed features $\\phi(x) = \\nabla_\\theta f(x; \\theta_0)$.<\/span><\/p>\n Furthermore, if the loss function is convex (like MSE), the network converges to a unique global minimum. The final function learned by the infinite-width network is exactly the kernel ridge regression solution using the NTK:<\/span><\/p>\n <\/p>\n $$f_{\\infty}(x) = \\Theta_\\infty(x, X_{train}) (\\Theta_\\infty(X_{train}, X_{train}))^{-1} Y_{train}$$<\/span><\/p>\n <\/p>\n (assuming zero initialization for simplicity).4<\/span><\/p>\n This equivalence provided a rigorous explanation for the training stability of overparameterized networks. It showed that “training” a massive network in this regime is effectively just projecting the target function onto the tangent space defined by the random initialization.<\/span><\/p>\n Like the NNGP, the NTK can be computed recursively for deep networks. For fully connected networks, the recursion involves both the covariance of the activations ($\\Sigma^{(l)}$) and the derivative of the nonlinearity ($\\dot{\\sigma}$).<\/span>3<\/span><\/p>\n Let $\\dot{\\Sigma}^{(l)}(x, x’) = \\sigma_w^2 \\mathbb{E}[\\dot{\\sigma}(u)\\dot{\\sigma}(v)]$. The NTK recursion is:<\/span><\/p>\n <\/p>\n $$\\Theta^{(l+1)}(x, x’) = \\Theta^{(l)}(x, x’) \\dot{\\Sigma}^{(l+1)}(x, x’) + \\Sigma^{(l+1)}(x, x’)$$<\/span><\/p>\n The first term represents the backpropagation of gradients through the weights, while the second term captures the contribution of the weights in the current layer. This recursive structure allows efficient computation of the exact infinite-width limit for various architectures, including CNNs (CNTK) and Graph Neural Networks.<\/span>6<\/span><\/p>\n While the NTK framework provides a powerful theoretical tool, it introduced a significant dissonance. Empirical studies quickly revealed that the NTK approximation does not fully capture the performance of state-of-the-art neural networks. Real networks, especially those trained with standard parameterizations and hyperparameters, often outperform their corresponding NTK kernels, sometimes by large margins.<\/span>5<\/span><\/p>\n This discrepancy led to the categorization of training regimes into two distinct phases: <\/span>Lazy Training<\/b> (the NTK regime) and <\/span>Rich Training<\/b> (the Feature Learning regime).<\/span>18<\/span><\/p>\n In the lazy regime, the network behaves as a linearized model.<\/span><\/p>\n In the rich regime, the network significantly departs from its initialization, and the internal representations evolve to adapt to the data structure.<\/span><\/p>\n The transition between Lazy and Rich regimes can be controlled by a scaling parameter $\\alpha$ that governs the magnitude of the model’s output at initialization.20<\/span><\/p>\n Consider a network parameterized as $f(x) = \\alpha \\sum_{i=1}^n v_i \\sigma(w_i^T x)$.<\/span><\/p>\n Chizat et al. demonstrated that by simply varying this scale, one can interpolate between the kernel regime and the feature learning regime, showing that they are two ends of a continuum.<\/span>20<\/span><\/p>\n The general theory of infinite-width networks applies to all architectures, but the specific mechanics of convergence and the quality of the limit depend heavily on the architecture type (ResNet vs. Transformer vs. CNN) and the specific parameterization used.<\/span><\/p>\n ResNets introduce skip connections ($x_{l+1} = x_l + f_l(x_l)$), which fundamentally alter signal propagation.<\/span><\/p>\n Transformers pose a unique challenge because they possess two distinct dimensions that can be taken to infinity: the embedding width $N$ (or $d_{model}$) and the number of attention heads $H$.<\/span><\/p>\n2. The Gaussian Process Limit at Initialization<\/b><\/h2>\n
2.1 Neal\u2019s Theorem and the Central Limit Theorem<\/b><\/h3>\n
2.2 Recursive Covariance Propagation<\/b><\/h3>\n
\n
\n<\/span>
\n<\/span>$$\\Sigma^{(1)}(x, x’) = \\frac{\\sigma_w^2}{d_{in}} \\langle x, x’ \\rangle + \\sigma_b^2$$<\/span>
\n<\/span>
\n<\/span>Here, $\\sigma_w^2$ and $\\sigma_b^2$ are the variances of the weights and biases, respectively.<\/span><\/li>\n
\n<\/span>
\n<\/span>$$\\Sigma^{(l+1)}(x, x’) = \\sigma_w^2 \\mathbb{E}_{(u, v) \\sim \\mathcal{N}(0, \\Lambda^{(l)})} [\\sigma(u)\\sigma(v)] + \\sigma_b^2$$<\/span>
\n<\/span>
\n<\/span>where $\\Lambda^{(l)}$ is the covariance matrix of the pre-activations at the previous layer:<\/span>
\n<\/span>
\n<\/span>$$\\Lambda^{(l)} = \\begin{pmatrix} \\Sigma^{(l)}(x, x) & \\Sigma^{(l)}(x, x’) \\\\ \\Sigma^{(l)}(x’, x) & \\Sigma^{(l)}(x’, x’) \\end{pmatrix}$$<\/span><\/li>\n<\/ol>\n2.3 The Phase Transition: Order vs. Chaos<\/b><\/h3>\n
\n
3. The Neural Tangent Kernel (NTK): Linearization of Training<\/b><\/h2>\n
3.1 The Geometry of Parameter Space vs. Function Space<\/b><\/h3>\n
3.2 The “Frozen” Kernel Property<\/b><\/h3>\n
\n
3.3 Equivalence to Kernel Ridge Regression<\/b><\/h3>\n
3.4 Computing the NTK<\/b><\/h3>\n
4. The Lazy vs. Rich Dichotomy: When Theory Meets Practice<\/b><\/h2>\n
4.1 Characteristics of the Lazy (NTK) Regime<\/b><\/h3>\n
\n
4.2 Characteristics of the Rich (Feature Learning) Regime<\/b><\/h3>\n
\n
4.3 The Role of Initialization Scale (<\/b>$\\alpha$<\/b>)<\/b><\/h3>\n
\n
5. Architectural Limits and Scaling Laws<\/b><\/h2>\n
5.1 Deep Residual Networks (ResNets)<\/b><\/h3>\n
\n
5.2 Transformers and Attention Collapse<\/b><\/h3>\n
\n
\n