bundle-course-sap-ariba-sourcing-procurement-contract-management-administration By Uplatz<\/a><\/h3>\nAn Overview of Model Compression Strategies<\/b><\/h3>\n
<\/p>\n
Model compression encompasses a diverse set of strategies aimed at reducing the size, computational complexity, and energy consumption of neural networks.<\/span>4<\/span> These techniques primarily operate by identifying and eliminating redundancy within the model’s parameters and computations. While this report focuses on quantization, it is essential to understand its place within the broader landscape of model compression. The primary families of techniques include:<\/span><\/p>\n\n- Pruning:<\/b> This technique involves removing superfluous parameters from a trained network. Parameters\u2014which can be individual weights, neurons, channels, or even entire layers\u2014that contribute minimally to the model’s output are identified and set to zero. This creates a sparse model that can be stored more efficiently and, with appropriate hardware or software support for sparse matrix operations, can lead to faster inference.<\/span>4<\/span><\/li>\n
- Knowledge Distillation:<\/b> In this paradigm, knowledge from a large, complex, and high-performing “teacher” model is transferred to a smaller, more efficient “student” model. The student model is trained not only on the ground-truth labels but also to mimic the output distributions (e.g., logits) of the teacher model. This process allows the compact student to learn the nuanced “dark knowledge” captured by the teacher, often achieving performance far superior to what it could attain if trained from scratch on the labels alone.<\/span>4<\/span><\/li>\n
- Low-Rank Factorization\/Decomposition:<\/b> Many layers in a neural network, particularly fully connected and convolutional layers, can be represented as large matrices. Low-rank factorization techniques approximate these large weight matrices by decomposing them into the product of two or more smaller, lower-rank matrices. This can significantly reduce the number of parameters and the computational cost of matrix multiplication operations with a manageable impact on accuracy.<\/span>4<\/span><\/li>\n
- Quantization:<\/b> This is the technique of reducing the numerical precision of the numbers used to represent a model’s parameters (weights and biases) and, during inference, its activations. Instead of using high-precision 32-bit floating-point numbers, quantization represents these values with lower-bit formats, such as 16-bit floats or, more commonly, 8-bit integers.<\/span>4<\/span> This method is the central focus of this report due to its profound and consistent impact on model efficiency.<\/span><\/li>\n<\/ul>\n
<\/p>\n
Core Principles of Quantization: Mapping High-Precision to Low-Precision Representations<\/b><\/h3>\n
<\/p>\n
At its core, quantization is the process of mapping values from a large, often continuous set to a smaller, discrete set.<\/span>2<\/span> In the context of deep learning, this involves converting the 32-bit floating-point ($FP32$) numbers that are standard during training into lower-precision data types like 16-bit floating-point ($FP16$), 8-bit integers ($INT8$), or even more aggressive 4-bit or 2-bit formats.<\/span>2<\/span><\/p>\nThis mapping is governed by a set of quantization parameters that define the transformation. The most common scheme is affine or asymmetric quantization, which is defined by two key parameters: a scale factor ($S$) and a zero-point ($Z$). The scale factor is a positive real number that determines the step size of the quantization, while the zero-point is an integer that ensures the real value of zero can be perfectly represented by a quantized integer. The relationship is expressed by the fundamental quantization equation 9:<\/span><\/p>\n <\/p>\n
$$\\text{real\\_value} = S \\times (\\text{quantized\\_value} – Z)$$<\/span><\/p>\n <\/p>\n
The process involves two steps:<\/span><\/p>\n\n- Quantization:<\/b> A floating-point value $x$ is mapped to its integer representation $x_q$ via $x_q = \\text{round}(x\/S) + Z$.<\/span><\/li>\n
- Dequantization:<\/b> The integer value $x_q$ is mapped back to an approximate floating-point value $\\hat{x}$ via $\\hat{x} = S \\times (x_q – Z)$.<\/span><\/li>\n<\/ol>\n
This transformation is inherently lossy. The difference between the original value $x$ and the dequantized value $\\hat{x}$ is known as the <\/span>quantization error<\/b>.<\/span>10<\/span> This error arises from two sources: <\/span>clipping<\/b>, where values outside the chosen quantization range are clipped to the minimum or maximum representable value, and <\/span>rounding<\/b>, where values within the range are rounded to the nearest discrete level. Minimizing this quantization error while maximizing the benefits of lower precision is the central challenge in the field of quantization.<\/span>15<\/span><\/p>\n <\/p>\n
The Primary Benefits: A Trifecta of Efficiency<\/b><\/h3>\n
<\/p>\n
The widespread adoption of quantization is driven by a powerful combination of three primary benefits, which collectively address the challenges posed by large-scale models.<\/span><\/p>\n\n- Reduced Model Size & Memory Footprint:<\/b> This is the most direct and intuitive advantage. By reducing the number of bits required to store each parameter, quantization significantly shrinks the overall model size. For example, converting a model from $FP32$ to $INT8$ theoretically results in a 4x reduction in its storage footprint (from 32 bits per parameter to 8 bits). This reduction has a profound impact on deployment, making it feasible to store complex models on devices with limited memory. Furthermore, it reduces memory bandwidth requirements, as less data needs to be moved from memory to the processing units during inference, which is often a critical performance bottleneck.<\/span>8<\/span><\/li>\n
- Accelerated Inference:<\/b> Quantization can dramatically increase inference speed, leading to lower latency and higher throughput. This speedup stems from two main factors. First, as mentioned, the reduced memory bandwidth means that processors spend less time waiting for data. Second, and more importantly, arithmetic operations on low-precision integers are fundamentally faster and more efficient than their floating-point counterparts on most modern hardware. CPUs, GPUs, and especially specialized AI accelerators (like Google’s TPUs or Apple’s Neural Engine) contain dedicated hardware units optimized for high-throughput integer matrix multiplication, delivering performance gains that can range from 2x to 4x or more.<\/span>9<\/span><\/li>\n
- Lower Power Consumption:<\/b> The efficiency of integer arithmetic also translates directly to reduced energy consumption. Floating-point operations are more complex and require more energy to execute than integer operations. By shifting the bulk of a model’s computations to the integer domain, quantization lowers the overall power draw of the inference process. This is a critical consideration for battery-operated devices like smartphones, wearables, and drones, where extending operational life is paramount. On a larger scale, in data centers serving millions of inference requests, these energy savings can lead to substantial reductions in operational costs and a smaller environmental carbon footprint.<\/span>4<\/span><\/li>\n<\/ul>\n
<\/p>\n
A Methodological Taxonomy of Quantization<\/b><\/h2>\n
<\/p>\n
The landscape of quantization is diverse, with numerous techniques developed to address different constraints and objectives. These methods can be systematically categorized based on several key design choices, which provides a clear framework for understanding their respective trade-offs in terms of accuracy, computational cost, and implementation complexity.<\/span><\/p>\n <\/p>\n
Training-Involvement Strategies: The PTQ vs. QAT Dichotomy<\/b><\/h3>\n
<\/p>\n
The most fundamental distinction among quantization methods is the point at which quantization is introduced relative to the model training process. This leads to two primary paradigms: Post-Training Quantization (PTQ) and Quantization-Aware Training (QAT).<\/span><\/p>\n <\/p>\n
Post-Training Quantization (PTQ)<\/b><\/h4>\n
<\/p>\n
PTQ is a process where quantization is applied to a neural network <\/span>after<\/span><\/i> it has already been fully trained to convergence in high precision (e.g., $FP32$).<\/span>5<\/span> This approach is designed to be a lightweight, post-hoc optimization step that does not require retraining the model or having access to the original training pipeline.<\/span><\/p>\nThe typical PTQ workflow involves a <\/span>calibration<\/b> phase. During this phase, a small, representative dataset (often just 100-500 samples) is passed through the high-precision model.<\/span>25<\/span> The purpose is to observe and record the statistical distribution (typically the minimum and maximum values) of the activation tensors at various points in the network. These observed ranges are then used to calculate the optimal quantization parameters (scale and zero-point) for the activations, which are dynamic and input-dependent. The weights, being static, can have their ranges determined directly without a calibration set.<\/span>5<\/span><\/p>\nThe primary advantage of PTQ is its simplicity and efficiency. It is computationally cheap, fast to execute, and does not require the original training dataset or a complex training environment, making it highly accessible.<\/span>5<\/span> However, its main drawback is a potential for significant accuracy degradation. Because the model’s weights were learned without any knowledge of quantization, the precision reduction can introduce noise that the model is not robust to. This accuracy drop becomes particularly pronounced for highly sensitive models or when quantizing to very low bit-widths (e.g., 4-bit or less).<\/span>10<\/span><\/p>\n <\/p>\n
Quantization-Aware Training (QAT)<\/b><\/h4>\n
<\/p>\n
In contrast to PTQ, QAT integrates the quantization process directly into the model training or fine-tuning loop.<\/span>5<\/span> The core idea is to simulate the effects of low-precision inference during training, allowing the model to learn parameters that are inherently robust to quantization noise.<\/span><\/p>\nThis is achieved by inserting “fake” quantization and de-quantization operations into the model’s computation graph. During the forward pass of training, weights and activations are quantized to a lower precision and then immediately de-quantized back to high precision before being used in subsequent computations.<\/span>16<\/span> This “fake quantization” step models the rounding and clipping errors that will occur during actual quantized inference. The model’s loss is then calculated based on these perturbed values, and the gradients are computed.<\/span><\/p>\nA key challenge in QAT is that the rounding function inherent in quantization is non-differentiable (its gradient is zero almost everywhere), which would halt the backpropagation of gradients. To overcome this, QAT employs the <\/span>Straight-Through Estimator (STE)<\/b>. The STE acts as a proxy for the gradient of the rounding function, typically by simply passing the incoming gradient through unchanged, as if the rounding operation were an identity function during the backward pass.<\/span>35<\/span> This allows the model’s high-precision weights to be updated in a way that accounts for the simulated quantization error.<\/span><\/p>\nThe main benefit of QAT is its superior accuracy. By adapting to the constraints of low-precision arithmetic during training, QAT can often recover model performance to a level nearly identical to the original full-precision model, even under aggressive quantization schemes.<\/span>5<\/span> The downside is its significant computational cost and complexity. QAT requires a full retraining or fine-tuning cycle, access to the complete training dataset, and modifications to the training pipeline, making it a much more involved and resource-intensive process than PTQ.<\/span>5<\/span><\/p>\nThe distinction between these two approaches is not always binary. The high cost of full QAT has driven innovation in advanced PTQ methods that incorporate small amounts of data-driven, layer-wise optimization, blurring the lines. These “PTQ++” techniques aim to capture some of QAT’s accuracy recovery benefits with a fraction of the computational cost, suggesting a convergence toward a spectrum of training-like effort rather than a strict dichotomy.<\/span>28<\/span><\/p>\n\n\n\n| Feature<\/b><\/td>\n | Post-Training Quantization (PTQ)<\/b><\/td>\n | Quantization-Aware Training (QAT)<\/b><\/td>\n<\/tr>\n\n| Core Concept<\/b><\/td>\n | Quantization applied to a fully trained model; no retraining.<\/span><\/td>\n | Quantization effects are simulated during training or fine-tuning.<\/span><\/td>\n<\/tr>\n\n| Computational Cost<\/b><\/td>\n | Low; requires only a brief calibration step.<\/span><\/td>\n | High; requires a full retraining or extensive fine-tuning cycle.<\/span><\/td>\n<\/tr>\n\n| Data Requirement<\/b><\/td>\n | Small, unlabeled calibration dataset (~100-500 samples).<\/span><\/td>\n | Full training dataset with labels.<\/span><\/td>\n<\/tr>\n\n| Typical Accuracy<\/b><\/td>\n | Can have a moderate to significant accuracy drop, especially at low bit-widths.<\/span><\/td>\n | Often recovers accuracy to near full-precision levels.<\/span><\/td>\n<\/tr>\n\n| Implementation Complexity<\/b><\/td>\n | Simple; can often be applied with a few lines of code.<\/span><\/td>\n | Complex; requires modifying the model architecture and training loop.<\/span><\/td>\n<\/tr>\n\n| Best For<\/b><\/td>\n | Rapid deployment, scenarios without access to the training pipeline, or when a small accuracy drop is acceptable.<\/span><\/td>\n | Maximizing accuracy, quantizing sensitive models, and aggressive low-bit quantization.<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n <\/p>\n Activation Handling Strategies: Static vs. Dynamic Quantization<\/b><\/h3>\n <\/p>\n Within the PTQ paradigm, a further distinction arises based on how the activation tensors are handled during inference. Since activations are input-dependent, their value ranges are not known ahead of time. This leads to two different strategies for determining their quantization parameters. Model weights, in contrast, are always known before inference and are therefore always quantized statically.<\/span>16<\/span><\/p>\n <\/p>\n Static Quantization<\/b><\/h4>\n <\/p>\n In static quantization, the quantization parameters (scale and zero-point) for the activations are pre-calculated and fixed before inference.<\/span>12<\/span> This is achieved through the calibration process described earlier, where a representative dataset is used to estimate the typical range of each activation tensor.<\/span>19<\/span><\/p>\nThe primary advantage of this approach is performance. With fixed quantization parameters for both weights and activations, the entire computation graph can be executed using highly efficient, integer-only arithmetic. This minimizes runtime overhead and allows the model to leverage specialized integer hardware to its full potential, resulting in the fastest possible inference speed.<\/span>19<\/span> The main drawback is its reliance on the calibration dataset. If the data encountered during real-world deployment has a significantly different distribution from the calibration data, the pre-calculated ranges may be suboptimal, leading to increased clipping errors and a degradation in model accuracy.<\/span>9<\/span><\/p>\n <\/p>\n Dynamic Quantization<\/b><\/h4>\n <\/p>\n In dynamic quantization, the model’s weights are quantized offline, but the quantization parameters for the activations are calculated “on the fly” for each input during the inference pass.<\/span>12<\/span> For each activation tensor, its minimum and maximum values are computed at runtime, and these values are used to determine its scale and zero-point for that specific inference instance.<\/span>19<\/span><\/p>\nThe key benefit of dynamic quantization is its flexibility and robustness. It does not require a calibration dataset and can adapt to varying input distributions, which can lead to better accuracy preservation than static quantization if the data distribution is unpredictable.<\/span>26<\/span> However, this flexibility comes at the cost of performance. The runtime calculation of activation statistics introduces computational overhead, making dynamic quantization slower than its static counterpart.<\/span>19<\/span> Furthermore, because the activation quantization parameters are not known ahead of time, it prevents a fully integer-only pipeline and may not be efficiently supported by all hardware accelerators.<\/span>43<\/span> It is often employed for models like LSTMs and Transformers where the bottleneck is memory bandwidth (loading the large weights) rather than computation, making the runtime overhead less impactful.<\/span>16<\/span><\/p>\n <\/p>\n Value-Mapping Strategies: Uniform vs. Non-Uniform Quantization<\/b><\/h3>\n <\/p>\n Another critical dimension of quantization is the strategy used to map continuous values to the discrete quantization levels. This choice affects the representational capacity of the quantized format and has significant implications for hardware efficiency.<\/span><\/p>\n <\/p>\n Uniform Quantization<\/b><\/h4>\n <\/p>\n Uniform quantization is the most common and straightforward approach. It divides the value range into evenly spaced intervals, meaning the step size between any two adjacent quantization levels is constant.<\/span>10<\/span> This linear mapping is simple to implement and aligns perfectly with the capabilities of standard integer arithmetic logic units (ALUs) found in most CPUs and GPUs. This hardware compatibility makes it extremely efficient to execute.<\/span>45<\/span><\/p>\nThe main limitation of uniform quantization is its inefficiency in representing data with non-uniform distributions. The weights and activations in neural networks often follow a bell-shaped or Laplacian distribution, with most values clustered near zero and a few outlier values in the tails. Uniform quantization allocates its representational capacity equally across the entire range, effectively “wasting” precision on sparsely populated regions while not providing enough precision for the dense regions around zero.<\/span>45<\/span><\/p>\n <\/p>\n Non-Uniform Quantization<\/b><\/h4>\n <\/p>\n Non-uniform quantization addresses this limitation by spacing the quantization levels unevenly. It allocates more discrete levels to regions of the value range where data points are dense (e.g., near zero) and fewer levels to sparse regions (e.g., the tails of the distribution).<\/span>10<\/span> Common methods to achieve this include applying a logarithmic scale or using clustering algorithms like k-means to determine the optimal placement of quantization levels.<\/span>9<\/span><\/p>\nThe primary advantage of this approach is its superior representational capacity. By better matching the underlying data distribution, non-uniform quantization can achieve higher accuracy than uniform quantization for the same bit-width.<\/span>45<\/span> However, this theoretical advantage is often overshadowed by practical implementation challenges. Standard hardware is built for uniform, linear arithmetic. Executing operations with non-uniformly quantized values typically requires special hardware or, more commonly, the use of software-based look-up tables (LUTs) to map the quantized indices to their corresponding real values before computation. This LUT-based approach can introduce significant latency and memory overhead, often negating the performance benefits of quantization and making it slower than the less accurate but hardware-friendly uniform approach.<\/span>47<\/span> This reality underscores a critical theme in model compression: the choice of algorithm is heavily dictated by the constraints of the target hardware. A theoretically superior method is of little practical value if it cannot be executed efficiently on available processors.<\/span><\/p>\n <\/p>\n Granularity Strategies: Balancing Accuracy and Overhead<\/b><\/h3>\n <\/p>\n Quantization granularity refers to the scope over which a single set of quantization parameters (a scale and zero-point pair) is applied. The choice of granularity represents a trade-off between the accuracy of the quantized representation and the overhead associated with storing and using the quantization parameters themselves.<\/span>5<\/span><\/p>\nThe different levels of granularity are:<\/span><\/p>\n\n- Per-Tensor (or Layer-wise):<\/b> This is the coarsest level, where a single scale and zero-point are used for an entire weight or activation tensor within a layer. It is the simplest method with the lowest memory overhead for storing the quantization parameters.<\/span>12<\/span><\/li>\n
- Per-Channel (or Channel-wise):<\/b> This is a finer granularity, commonly used for the weights of convolutional and linear layers. A separate scale and zero-point are calculated for each output channel of the weight tensor. This approach can significantly improve accuracy because it can adapt to the fact that different output channels often learn features with very different statistical distributions and value ranges.<\/span>5<\/span><\/li>\n
- Group-wise:<\/b> This is an even finer level of granularity where a single set of parameters is shared across a small, contiguous block of weights (e.g., a group of 64 or 128 values). This technique has become particularly important in the quantization of LLMs, as it allows the model to isolate and handle problematic outlier values within a tensor by giving them their own quantization range, without corrupting the precision for the rest of the values.<\/span>5<\/span><\/li>\n<\/ul>\n
The general trade-off is clear: finer granularity (group-wise > per-channel > per-tensor) allows the quantization scheme to more closely adapt to the local statistics of the data, which typically leads to lower quantization error and higher model accuracy. However, this comes at the cost of increased storage overhead, as more scale and zero-point values must be stored alongside the model. This can also introduce additional computational complexity during inference, as different parameters must be applied to different parts of the tensor.<\/span>5<\/span><\/p>\n <\/p>\n Advanced Techniques in Post-Training Quantization<\/b><\/h2>\n <\/p>\n The inherent simplicity of PTQ makes it an attractive option for deployment, but its potential for accuracy degradation has spurred the development of more sophisticated techniques. These advanced PTQ methods aim to bridge the accuracy gap with QAT by incorporating intelligent, data-driven optimizations that make the model more robust to quantization, all without the need for end-to-end retraining.<\/span><\/p>\n <\/p>\n Data-Free and Low-Data Methods for Pre-Processing<\/b><\/h3>\n <\/p>\n A powerful class of PTQ techniques involves pre-processing the full-precision model to make it more “quantization-friendly” before the quantization step is even applied. These methods modify the model’s weights and biases in a mathematically equivalent way, ensuring the output of the full-precision model remains unchanged while altering its internal properties to reduce the eventual quantization error.<\/span><\/p>\n <\/p>\n Cross-Layer Equalization (CLE)<\/b><\/h4>\n <\/p>\n A significant challenge in quantization arises when consecutive layers in a network have vastly different weight tensor ranges. For example, one layer might have weights in the range $[-0.1, 0.1]$ while the next has weights in $[-10, 10]$. This disparity makes it difficult to quantize both layers effectively, as the first layer will suffer from underutilization of the quantization grid, while the second will suffer from excessive clipping.<\/span>51<\/span><\/p>\nCross-Layer Equalization (CLE)<\/b> is a technique designed to mitigate this issue by balancing the dynamic ranges of weights across consecutive layers.<\/span>29<\/span> It leverages the scale-equivariance property of common activation functions like ReLU, where $ReLU(s \\cdot x) = s \\cdot ReLU(x)$ for a positive scaling factor $s$. CLE identifies pairs of consecutive layers (e.g., two convolution layers) and introduces a scaling factor. It scales down the weights of the first layer’s output channels by this factor and scales up the weights of the second layer’s corresponding input channels by the same factor. This operation leaves the mathematical output of the two-layer block unchanged in full precision but redistributes the dynamic range more evenly between them.<\/span>51<\/span> By equalizing the weight ranges, CLE makes the model inherently more robust to subsequent quantization. It is a data-free method and has proven particularly effective for architectures that rely on depth-wise separable convolutions, such as the MobileNet family.<\/span>29<\/span><\/p>\n <\/p>\n Bias Correction<\/b><\/h4>\n <\/p>\n Quantization is not an unbiased process. The combination of rounding and, more importantly, the clipping of outlier values can introduce a systematic error, or bias, that shifts the mean of a layer’s output distribution.<\/span>54<\/span> This error then propagates through the subsequent layers of the network, accumulating and potentially leading to a significant drop in overall model accuracy.<\/span>52<\/span><\/p>\nBias Correction<\/b> is a technique that aims to compensate for this quantization-induced shift.<\/span>29<\/span> It operates on a layer-by-layer basis, typically after other methods like CLE have been applied. The process requires a small calibration dataset. For each layer, the method calculates the mean output of the original full-precision layer and the mean output of the quantized layer over the calibration data. The difference between these two means represents the average error, or bias, introduced by quantization. This error value is then subtracted from the layer’s bias term, effectively re-centering the output distribution of the quantized layer to more closely match that of the original model.<\/span>29<\/span> This simple yet effective technique can often recover a significant portion of the accuracy lost due to quantization bias.<\/span><\/p>\n <\/p>\n Calibration-Driven Optimization<\/b><\/h3>\n <\/p>\n While pre-processing methods make the model more amenable to quantization, another class of techniques focuses on optimizing the quantization process itself, using calibration data to make more intelligent decisions than simple heuristics.<\/span><\/p>\n <\/p>\n | | | | | | |
![\"\"](\"https:\/\/uplatz.com\/blog\/wp-content\/uploads\/2025\/11\/Neural-Network-Quantization-1024x576.jpg\")
<\/p>\n