SAS Viya Platform Administration<\/a><\/p>\n1.1 Architectural Principles of Modern Multimodal Models (LMMs)<\/b><\/h4>\n
<\/p>\n
At the heart of the current wave of multimodal AI are Large Multimodal Models (LMMs), sophisticated systems exemplified by industry-leading models such as Google’s Gemini, OpenAI’s GPT-4o, and Anthropic’s Claude 3.<\/span>1<\/span> These models are predominantly built upon the Transformer architecture, a design that has proven exceptionally effective at processing sequential data.<\/span>1<\/span> The architectural blueprint for a typical LMM follows a structured, multi-stage workflow designed to translate heterogeneous data into a unified, machine-readable format.<\/span><\/p>\nThe process begins with a set of <\/span>specialized encoders<\/b>. Each distinct data modality is channeled through its own dedicated encoder, which is specifically designed to handle the unique characteristics of that data type. For instance, a Vision Transformer (ViT) or a Convolutional Neural Network (CNN) might process images, while a separate text encoder handles natural language inputs.<\/span>1<\/span> The function of these encoders is to transform the raw input data into high-dimensional vector representations, commonly known as embeddings. These embeddings serve as a numerical proxy for the semantic content of the original input.<\/span><\/p>\nFollowing the encoding stage, the disparate embeddings from each modality must be integrated. This is accomplished through a <\/span>fusion mechanism<\/b>, a critical component that merges the modality-specific representations into a shared, coherent semantic space.<\/span>1<\/span> It is within this fusion layer that true cross-modal understanding is forged. The model learns to associate concepts across modalities\u2014for example, linking the visual features of a chart in a presentation with the corresponding textual explanation or spoken narration.<\/span>1<\/span> This integration is often facilitated by a powerful technique known as cross-attention, which allows the model to selectively focus on relevant parts of one modality based on context provided by another.<\/span>1<\/span> By natively processing these different data types without requiring intermediate conversions, LMMs can handle complex, real-world tasks with greater efficiency and generate richer, more nuanced insights than their unimodal predecessors.<\/span>1<\/span><\/p>\n <\/p>\n
1.2 The Transformer’s Engine: Deconstructing the Self-Attention Mechanism<\/b><\/h4>\n
<\/p>\n
The revolutionary success of the Transformer architecture, and by extension the LMMs built upon it, is attributable to its core computational engine: the self-attention mechanism.<\/span>5<\/span> This mechanism allows the model to weigh the importance of different tokens within a sequence relative to each other, enabling it to capture complex, long-range dependencies. The mathematical formulation of the most common form, scaled dot-product attention, is given by the equation:<\/span><\/p>\nAttention(Q,K,V)=softmax(dk\u200b\u200bQKT\u200b)V<\/span><\/p>\nHere, the input sequence is projected into three distinct matrices: Query (Q), Key (K), and Value (V).<\/span>5<\/span> The<\/span><\/p>\nQuery vector represents what a particular token is “looking for,” the Key vector represents what a token “offers,” and the Value vector contains the actual content or semantic information of the token. The attention function computes the dot product of the Query matrix with the transpose of the Key matrix (QKT), which results in a matrix of similarity scores between every query token and every key token.<\/span>7<\/span> For an input sequence of length<\/span><\/p>\nn, this operation produces an n\u00d7n attention matrix.<\/span>8<\/span><\/p>\nThese scores are then scaled by the square root of the dimension of the key vectors (dk\u200b\u200b) to stabilize gradients during training.<\/span>7<\/span> A softmax function is applied to normalize the scores, converting them into a probability distribution where the weights for each query sum to one. Finally, this attention weight matrix is multiplied by the<\/span><\/p>\nValue matrix, producing an output where each token’s representation is a weighted sum of all other tokens’ values in the sequence, with the weights determined by the learned attention scores.<\/span>6<\/span><\/p>\nTo allow the model to capture diverse types of relationships simultaneously (e.g., syntactic, semantic, positional), this process is parallelized in what is known as <\/span>multi-head attention<\/b>. The model learns multiple independent sets of Q, K, and V projection matrices, each constituting an “attention head.” The outputs of these parallel heads are then concatenated and linearly projected to form the final output.<\/span>5<\/span> While this enhances the model’s expressive power, it also multiplies the computational workload.<\/span><\/p>\n <\/p>\n
1.3 The Quadratic Complexity Problem: Why Dense Attention Limits Scale and Scope<\/b><\/h4>\n
<\/p>\n
The elegant design of the self-attention mechanism conceals a fundamental limitation that has become the single greatest bottleneck in scaling AI models: its quadratic complexity. The computation of the full n\u00d7n attention matrix, where every token must attend to every other token, results in a computational cost that scales quadratically with the sequence length n, denoted as O(n2).<\/span>8<\/span> This quadratic growth means that doubling the sequence length quadruples the computational requirement, making the processing of very long sequences computationally intractable.<\/span>8<\/span> For context, in modern LLMs, the attention computation can account for as much as 70-80% of the total latency when processing sequences of 64,000 tokens.<\/span>13<\/span><\/p>\nThis computational burden is mirrored by a quadratic growth in memory requirements. Storing the full attention matrix itself has a complexity of O(n2), and during autoregressive generation, the model must maintain a Key-Value (KV) cache that stores the key and value vectors for all previous tokens. While the KV cache grows linearly with sequence length, the overall memory footprint of the attention mechanism creates a severe bottleneck, particularly on hardware like GPUs with finite VRAM.<\/span>8<\/span><\/p>\nThis quadratic bottleneck is especially acute in the context of multimodal models. While a text document might consist of a few thousand tokens, a single high-resolution image can be tokenized into thousands of patches, and just a few seconds of high-frame-rate video can generate tens of thousands of spatio-temporal tokens.<\/span>12<\/span> When a model must simultaneously process long sequences from multiple modalities\u2014such as a long video, its audio track, and a detailed textual prompt\u2014the combined sequence length makes the<\/span><\/p>\nO(n2) cost of dense attention practically infeasible.<\/span>2<\/span><\/p>\nThe implications of this bottleneck extend beyond mere technical constraints; they impose significant economic and environmental costs on the development and deployment of advanced AI. The need for massive computational power to handle dense attention translates directly into a demand for more powerful and expensive hardware, such as large clusters of GPUs or TPUs. This high cost of entry creates a substantial financial barrier, effectively centralizing cutting-edge AI research and development within a handful of large, well-funded corporations and limiting broader access to these transformative technologies.<\/span>15<\/span> Furthermore, the immense energy consumption required for the training and inference of these large-scale models carries a significant environmental footprint.<\/span>16<\/span> Consequently, solving the attention bottleneck through methods like sparsity is not merely a technical optimization. It is a critical step toward democratizing AI, reducing the economic and environmental costs of innovation, and fostering a more sustainable and accessible technological future.<\/span><\/p>\n <\/p>\n
Section 2: The Rationale for Sparsity: From Efficiency to Efficacy<\/b><\/h3>\n
<\/p>\n
In response to the formidable challenge posed by the quadratic complexity of dense attention, the field has converged on a powerful solution: sparsity. Sparse attention mechanisms are designed to break the O(n2) scaling law by fundamentally rethinking the assumption that every token needs to interact with every other token. The initial motivation for this approach was rooted in computational efficiency, but subsequent research has unveiled a surprising and profound secondary benefit: sparsity can not only make models more efficient but also more effective.<\/span><\/p>\n <\/p>\n
2.1 Breaking the Quadratic Barrier: Computational and Memory Advantages<\/b><\/h4>\n
<\/p>\n
The core principle of sparse attention is to reduce the computational complexity by restricting each query token to interact with only a limited subset of k key tokens, where k is significantly smaller than the total sequence length n (k\u226an). By doing so, the computational complexity of the attention mechanism can be reduced from O(n2) to a more manageable O(n\u22c5k) or, in some structured cases, even O(nlogn).<\/span>8<\/span><\/p>\nThis reduction in complexity yields substantial efficiency gains. By calculating only a small fraction of the total possible attention scores, sparse methods drastically decrease the number of floating-point operations (FLOPs) required for the forward and backward passes.<\/span>19<\/span> This, in turn, leads to a significant reduction in memory access and storage requirements, as the model no longer needs to compute or hold the entire dense attention matrix in memory.<\/span>21<\/span> The practical impact of these gains is transformative. It enables models to process much longer input sequences, a capability that is essential for a wide range of real-world applications that were previously out of reach. These include the analysis of lengthy legal contracts or scientific papers, the processing of entire software code repositories, and the generation of high-resolution, long-duration videos.<\/span>11<\/span> The resulting improvements in training and inference speed make the deployment of large-scale models more feasible and cost-effective.<\/span>13<\/span><\/p>\n <\/p>\n
2.2 The Surprising Benefit: How Removing Redundant Information Can Enhance Performance<\/b><\/h4>\n
<\/p>\n
While sparsity was initially conceived as a practical compromise\u2014trading a degree of model performance for a significant gain in efficiency\u2014a growing body of empirical evidence has revealed a counter-intuitive phenomenon: in many cases, sparse attention can actually <\/span>improve<\/span><\/i> model accuracy and robustness.<\/span>20<\/span> This discovery challenges the long-held assumption that dense attention represents the “gold standard” for performance.<\/span><\/p>\nThe underlying reason for this surprising benefit lies in the filtering of noise and redundancy. A full, dense attention mechanism forces the model to consider every possible token-to-token interaction, many of which are irrelevant, redundant, or actively misleading.<\/span>25<\/span> This can lead to the model “wasting” a non-negligible portion of its attention capacity on irrelevant keys, which introduces noise into the feature aggregation process and can degrade the quality of the learned representations.<\/span>27<\/span><\/p>\nResearch from the LoRA-Sparse paper, for instance, demonstrated that removing what it termed “useless attention” is actively beneficial. Their method achieved a 0.8% performance improvement over a dense attention baseline with a selection ratio of just 50%, and other studies have reported similar gains.<\/span>20<\/span> By compelling the model to focus only on the most salient relationships, sparse attention effectively acts as a powerful form of regularization. It filters out distracting information, leading to cleaner, more discriminative feature representations, better generalization to unseen data, and more robust overall performance.<\/span>20<\/span> This reframes the objective of sparsity research. The goal is no longer simply to approximate the dense attention matrix as efficiently as possible, but rather to discover an optimal sparse connectivity pattern that is inherently superior to its dense counterpart. This transforms the problem from one of engineering approximation into one of scientific discovery, seeking the ideal computational graph for a given task.<\/span><\/p>\n <\/p>\n
2.3 A Natural Emergence: Theoretical Underpinnings of Sparsity in Transformers<\/b><\/h4>\n
<\/p>\n
Further bolstering the case for sparsity is the finding that it is not merely a contrived, practical heuristic but rather an intrinsic property of trained Transformer models. Analysis of the attention matrices in large, pre-trained models consistently reveals that they are naturally sparse. Even after being trained with a dense mechanism, the learned attention distributions are highly concentrated, with the vast majority of the probability mass assigned to a very small subset of key tokens.<\/span>17<\/span> Studies have documented sparsity levels as high as 96.8% in the attention heads of long-context LLMs, with a negligible impact on the model’s ability to recall information.<\/span>29<\/span><\/p>\nThis inherent sparsity appears to be deeply connected to the learning dynamics of Transformers. Researchers have observed that the formation of sparse attention patterns during the training process often coincides with the sudden emergence of new, complex capabilities, such as in-context learning and factual recall.<\/span>30<\/span> This suggests that the ability to learn to ignore irrelevant context and focus computational resources on a few critical tokens is not just an efficiency hack but a fundamental mechanism underlying the development of advanced reasoning in these models.<\/span><\/p>\nThe speed at which these sparse patterns emerge is also theoretically linked to the statistical properties of the training data. Specifically, the repetition of information, both within a single training example (termed “in-context repetition” or “burstiness”) and across the entire dataset (“cross-sample repetition”), has been shown to accelerate the formation of these crucial neural circuits.<\/span>30<\/span> This provides a compelling theoretical framework connecting the structure of the data, the model’s internal learning dynamics, and the emergence of both sparsity and sophisticated cognitive abilities.<\/span><\/p>\n <\/p>\n
Part II: A Taxonomy of Sparse Attention Mechanisms<\/b><\/h2>\n
<\/p>\n
The pursuit of efficient and effective attention has given rise to a diverse ecosystem of sparse attention mechanisms. These methods can be broadly categorized along a spectrum, from simple, pre-defined fixed patterns to complex, dynamic patterns that adapt to the input content at runtime. This evolution reflects a continuous search for the optimal balance between computational efficiency, architectural simplicity, and expressive power.<\/span><\/p>\n <\/p>\n
Section 3: Fixed and Structured Sparsity Patterns<\/b><\/h3>\n
<\/p>\n
The earliest and most straightforward approaches to sparse attention involve imposing a pre-defined, static sparsity pattern on the attention matrix. These patterns are fixed and do not change based on the input data. They represent a set of strong but potentially rigid inductive biases about which token interactions are most important. Their development marks a historical progression in the search for the “correct” set of assumptions to guide efficient attention.<\/span><\/p>\n <\/p>\n
3.1 Local and Sliding Window Attention<\/b><\/h4>\n
<\/p>\n
The simplest form of fixed sparsity is local or sliding window attention. This approach is based on the strong inductive bias of locality, which posits that the most relevant context for a given token is likely to be found in its immediate vicinity.<\/span>13<\/span> In this scheme, each token is restricted to attend only to a fixed-size window of its neighboring tokens.<\/span>21<\/span><\/p>\nThis method is highly effective for tasks where local context is paramount, such as in certain types of image processing where interactions between adjacent pixels are most critical. For example, the Swin Transformer, a highly successful architecture for computer vision, employs a local attention mechanism within shifted windows to efficiently model visual features.<\/span>17<\/span> However, the primary drawback of a purely local attention mechanism is its inability to capture the long-range dependencies that are a hallmark of the Transformer’s power.<\/span>13<\/span> Models like StreamingLLM, which use a moving window for efficient long-context inference, must employ special mechanisms to handle information flow beyond the local window.<\/span>13<\/span><\/p>\n <\/p>\n
3.2 Strided and Dilated (Atrous) Attention<\/b><\/h4>\n
<\/p>\n
To address the limited receptive field of local attention without increasing computational cost, researchers developed strided or dilated attention. Instead of attending to a contiguous block of neighbors, a token attends to other tokens at fixed intervals or strides, creating a “dilated” or “atrous” window.<\/span>8<\/span> This allows the model’s attention to span a much wider range of the input sequence while keeping the number of attended tokens constant.<\/span><\/p>\nStrided attention offers a more effective trade-off between local and global context modeling compared to a simple sliding window. It can capture relationships between more distant tokens, making it a more versatile fixed pattern. This type of pattern is often included as a component in more complex hybrid sparsity models.<\/span>11<\/span><\/p>\n <\/p>\n
3.3 Global and Hybrid Patterns (e.g., BigBird, Longformer)<\/b><\/h4>\n
<\/p>\n
The limitations of purely local or strided patterns led to the development of hybrid models that combine multiple fixed patterns to achieve a more comprehensive view of the input sequence. These models represent a more refined and weaker inductive bias, acknowledging the need for both local detail and global context. Canonical examples of this approach are Longformer and BigBird, which were among the first methods to make the processing of very long sequences practical.<\/span>11<\/span><\/p>\nThese hybrid architectures typically integrate three types of fixed attention patterns:<\/span><\/p>\n\n- Local Window Attention:<\/b> Each token attends to a local window of its neighbors, preserving the fine-grained local context.<\/span>23<\/span><\/li>\n
- Global Attention:<\/b> A small number of pre-selected tokens are designated as “global” tokens. These tokens can attend to all other tokens in the sequence, and all other tokens can attend to them. They function as information hubs or aggregators, ensuring that a pathway for global information flow is always maintained.<\/span>18<\/span><\/li>\n
- Random Attention:<\/b> To further enhance global connectivity and robustness, each token may also attend to a small, randomly selected set of other tokens across the sequence.<\/span>31<\/span><\/li>\n<\/ol>\n
By combining these patterns, models like Longformer and BigBird create a sparse attention matrix that is computationally efficient yet capable of modeling both local and global dependencies, significantly expanding the capabilities of Transformers on long-document tasks.<\/span>23<\/span><\/p>\n <\/p>\n
3.4 Observed Patterns in Practice: A-Shape, Vertical-Slash, and Block-Sparse<\/b><\/h4>\n
<\/p>\n
While the aforementioned patterns were largely human-designed based on intuition, extensive analysis of the attention matrices of trained long-context LLMs has revealed that certain stable, recurring sparse patterns emerge naturally during training.<\/span>29<\/span> The MInference framework identified three such general patterns that are particularly common and can be exploited for significant efficiency gains, especially during the compute-intensive pre-filling stage of inference.<\/span><\/p>\n\n- A-shape Pattern:<\/b> In these attention heads, the attention scores are heavily concentrated on two main areas: the very first few tokens of the sequence (a phenomenon known as the “attention sink,” which acts as a global information aggregator) and a local window of tokens immediately surrounding the current query token. This creates a pattern resembling the letter ‘A’.<\/span>29<\/span><\/li>\n
- Vertical-Slash Pattern:<\/b> This pattern is characterized by strong vertical lines, indicating that certain key tokens are highly attended to by many different query tokens throughout the sequence. This is combined with diagonal “slashes” that correspond to standard local attention.<\/span>29<\/span><\/li>\n
- Block-Sparse Pattern:<\/b> Here, the attention is not randomly scattered but is concentrated within specific rectangular blocks of the attention matrix. The locations of these important blocks can be efficiently approximated at runtime using techniques like mean pooling on the query and key matrices.<\/span>29<\/span><\/li>\n<\/ul>\n
The significance of these observed patterns is their stability and predictability. They tend to be specific to particular attention heads and layers and are relatively consistent across different inputs. This allows for a “kernel-aware search” to be performed offline, assigning the most efficient, specialized computational kernel to each head based on its dominant pattern. This approach of matching emergent structures to optimized hardware operations represents a key step in bridging the gap between theoretical sparsity and practical acceleration.<\/span>29<\/span> The evolution from human-designed biases (like local windows) to exploiting these naturally learned structures (like A-shapes) provides a clear motivation for the next step in the taxonomy: methods that allow the model to learn and adapt its sparsity patterns dynamically.<\/span><\/p>\n <\/p>\n
Section 4: Dynamic and Content-Aware Sparsity<\/b><\/h3>\n
<\/p>\n
While fixed sparsity patterns offer significant efficiency gains, their inherent rigidity is a major limitation. The optimal way to connect tokens is not static but depends heavily on the specific content of the input. This realization spurred the development of dynamic and content-aware sparsity mechanisms, which represent a fundamental shift from model-centric to data-centric optimization. These methods empower the model to determine the most relevant attention patterns at runtime, tailoring its computational graph to the unique demands of each input sequence.<\/span><\/p>\n <\/p>\n
4.1 Learned Sparsity: Routing, Clustering, and Expert-Choice Mechanisms<\/b><\/h4>\n
<\/p>\n
This class of methods aims to make the sparsity pattern itself a learnable component of the model. Instead of relying on pre-defined heuristics, the model learns a policy for how to allocate its attention resources based on the input data.<\/span><\/p>\nA leading example of this approach is the <\/span>Mixture of Sparse Attention (MoSA)<\/b>. Drawing inspiration from the Mixture-of-Experts (MoE) paradigm, MoSA treats each attention head as an “expert” with a specialized function.<\/span>32<\/span> It employs a lightweight, learnable “expert-choice” routing network that allows each head to dynamically select its preferred top-<\/span><\/p>\nk tokens from the input sequence to attend to.<\/span>32<\/span> This creates arbitrary, content-dependent sparse attention patterns that are tailored to the needs of each head. A key advantage of MoSA is its demonstrated ability to outperform dense attention baselines in an isoFLOPs setting\u2014that is, when given the same total computational budget. By saving compute on the attention calculation, MoSA can afford to have more attention heads, leading to greater specialization and, in some cases, up to a 27% improvement in perplexity over a dense model with the same FLOPs.<\/span>32<\/span><\/p>\nAnother approach in this category is the <\/span>Routing Transformer<\/b>, which uses online k-means clustering to group semantically similar tokens together. Attention is then confined to operate only within these dynamically formed clusters, ensuring that computation is focused on related concepts.<\/span>32<\/span><\/p>\n <\/p>\n
4.2 Approximation Methods: Low-Rank Approximations (LoRA-Sparse) and Hashing<\/b><\/h4>\n
<\/p>\n
A second family of dynamic methods seeks to avoid the cost of computing the full n\u00d7n attention matrix by first creating a cheap approximation of it. This approximation is then used to guide the selection of the most important token pairs for the final, high-fidelity sparse attention calculation.<\/span><\/p>\nThe <\/span>LoRA-Sparse<\/b> method is a prominent example of this technique.<\/span>20<\/span> It first projects the query and key vectors into a much lower-dimensional (low-rank) space and computes an approximate attention map there. Since this map is much smaller, its computation is significantly cheaper. The method then identifies the top-scoring query-key pairs from this low-rank approximation and uses this information to construct a sparse attention mask for the full-dimensional space. The final attention calculation is then performed only on these selected, high-importance pairs. A critical innovation in LoRA-Sparse is its “order-mimic” training objective, which explicitly trains the low-rank approximation to preserve the relative ordering of the attention scores from the full matrix, ensuring that the selection of important pairs is highly accurate.<\/span>20<\/span><\/p>\nOther methods in this category use <\/span>hashing<\/b> to achieve a similar goal. Models like Reformer and MagicPIG employ locality-sensitive hashing (LSH), a technique that groups similar vectors together with high probability. By assigning queries and keys to hash buckets and restricting attention to only occur between tokens within the same bucket, these models can approximate the full attention matrix with sub-quadratic complexity.<\/span>35<\/span><\/p>\n <\/p>\n
4.3 Selection-Based Methods: Top-k Token and Channel Selection<\/b><\/h4>\n
<\/p>\n
A third category of dynamic sparsity involves explicit, selection-based filtering. These methods calculate a measure of importance for all potential interactions and then use a hard selection criterion, such as top-k, to prune the irrelevant ones.<\/span><\/p>\nThe most direct form is <\/span>top-k token selection<\/b>. This is the core mechanism used in the MESAN model for Visual Question Answering.<\/span>36<\/span> In this approach, the model calculates the initial attention scores for all token pairs but then explicitly selects only the top-<\/span><\/p>\nk highest-scoring keys for each query to use in the final weighted sum of values. This directly filters out interactions deemed less relevant.<\/span><\/p>\nA more sophisticated variant is the <\/span>Double Sparsity<\/b> method, which combines two orthogonal types of sparsity for greater efficiency <\/span>37<\/span>:<\/span><\/p>\n\n- Token Sparsity:<\/b> This is the dynamic, content-aware selection of important tokens to attend to, similar to the top-k method.<\/span><\/li>\n
- Channel Sparsity:<\/b> This leverages the insight that, for a given attention calculation, only a small subset of the feature channels (dimensions) in the query and key vectors are actually significant. This channel-level sparsity is found to be relatively static and can be identified efficiently via a one-time, offline calibration process.<\/span><\/li>\n<\/ol>\n
![\"\"](\"https:\/\/uplatz.com\/blog\/wp-content\/uploads\/2025\/06\/ChatGPT-Image-Jul-4-2025-01_58_29-PM.png\")
<\/p>\n