https:\/\/uplatz.com\/course-details\/servicenow\/451<\/a><\/p>\n1. Introduction to Generative Models and Normalizing Flows<\/b><\/h3>\n
<\/p>\n
1.1. Contextualizing Normalizing Flows within Generative Modeling<\/b><\/h4>\n
<\/p>\n
Generative models represent a fundamental class of machine learning models whose primary objective is to learn the underlying probability distribution of a given dataset. Once this distribution is learned, these models can then generate new, synthetic data samples that closely resemble the original training data. Their utility spans a wide array of applications, including data synthesis, anomaly detection, and the learning of meaningful data representations.<\/span><\/p>\nNormalizing Flows (NFs) constitute a distinct category within generative models, specifically designed as likelihood-based models for continuous inputs.<\/span>2<\/span> Over time, NFs have consistently shown promising results in both density estimation and generative modeling tasks.<\/span>2<\/span><\/p>\n <\/p>\n
1.2. Fundamental Principles of Normalizing Flows<\/b><\/h4>\n
<\/p>\n
At its core, a Normalizing Flow operates by transforming a complex, often intractable, data distribution into a simpler, known prior distribution\u2014typically a standard Gaussian noise distribution. This transformation is achieved through a sequence of invertible and differentiable mappings.<\/span>2<\/span> The mathematical cornerstone that enables NFs to precisely track the likelihood of data points throughout this intricate transformation process is the “change of variable formula”.<\/span>2<\/span><\/p>\nThe inherent design of Normalizing Flows bestows upon them several unique and appealing properties. These include the ability to perform exact likelihood computation, operate with deterministic objective functions during training, and efficiently compute both the data generator (forward pass) and its inverse (reverse pass).<\/span>2<\/span> This makes them bijective mappings between inputs and latent representations <\/span>13<\/span>, where their structure inherently facilitates analytical log-likelihood computation.<\/span>10<\/span><\/p>\n <\/p>\n
1.3. Historical Context and Recent Resurgence<\/b><\/h4>\n
<\/p>\n
Despite their theoretical elegance and a set of unique properties, Normalizing Flows had, for a period, received comparatively little attention in recent years.<\/span>2<\/span> Their practical adoption remained limited <\/span>3<\/span>, especially when contrasted with the rapid advancements and widespread popularity of other generative models such as Diffusion Models and Large Language Models. The state-of-the-art in Normalizing Flows had, regrettably, not kept pace with the swift progress observed in these alternative generative techniques.<\/span>3<\/span><\/p>\nThe research indicates that the prior underperformance of Normalizing Flows, leading to their being largely overlooked, stemmed not from fundamental theoretical flaws but rather from limitations in the expressive power of the underlying transformations used to implement the invertible mappings.<\/span>10<\/span> While the core mathematical principles of NFs, such as exact likelihood and invertibility, were always theoretically sound, the practical implementation of the invertible functions often lacked the necessary capacity to effectively model complex, high-dimensional data. TarFlow emerges as a pivotal development in this context, directly addressing this issue by integrating powerful neural network backbones, specifically the Transformer architecture. This architectural upgrade represents a critical shift: TarFlow’s innovation lies not in altering the core NF principle but in dramatically enhancing the function approximator utilized within the flow, moving from simpler masked MLPs to the highly expressive Transformer architecture.<\/span>3<\/span><\/p>\nThis breakthrough aims to demonstrate that Normalizing Flows are, in fact, more powerful than previously believed.<\/span>2<\/span> Such a development holds the potential to reopen an alternative path to powerful generative modeling <\/span>3<\/span>, initiating a new era for this class of models. This validation of the theoretical rigor inherent in Normalizing Flows suggests that their previous limitations were more a matter of engineering and architectural design rather than fundamental theoretical constraints. This opens the door for Normalizing Flows to gain wider adoption in applications where exact likelihoods and guaranteed invertibility are paramount, such as anomaly detection, scientific data analysis, and rigorous model comparison <\/span>15<\/span>, fields where other generative models often fall short.<\/span><\/p>\n <\/p>\n
2. Transformer-Based Normalizing Flows (TarFlow): Architecture and Core Principles<\/b><\/h3>\n
<\/p>\n
2.1. Definition and Conceptual Foundation<\/b><\/h4>\n
<\/p>\n
TARFLOW, an acronym for Transformer AutoRegressive Flow, is introduced as a powerful and highly scalable Normalizing Flow architecture.<\/span>2<\/span> It builds upon the conceptual foundation of Masked Autoregressive Flows (MAFs) but fundamentally enhances their capabilities by leveraging the robust architecture of Transformers.<\/span>1<\/span><\/p>\n <\/p>\n
2.2. Detailed Architectural Components and Integration<\/b><\/h4>\n
<\/p>\n
The core of TarFlow\u2019s architecture is a stack of autoregressive Transformer blocks applied to image patches.<\/span>1<\/span> A crucial design element that contributes to its invertibility and expressive power is the alternating direction of autoregression between successive layers.<\/span>1<\/span><\/p>\nEach autoregressive flow transformation within TarFlow is implemented using a causal Vision Transformer (ViT) operating on a sequence of image patches.<\/span>2<\/span> This design choice facilitates powerful non-linear transformations across all image patches while critically maintaining a parallel computational graph during the training phase.<\/span>2<\/span> This parallelism during training is a key enabler for building large, high-capacity models.<\/span><\/p>\nThe fundamental distinction of TarFlow from traditional MAFs lies in its deployment of a powerful masked Transformer that operates in a block autoregression fashion. This means it predicts a block of dimensions at a time, in contrast to the simpler masked Multi-Layer Perceptrons (MLPs) used in MAFs, which factorize inputs on a per-dimension basis.<\/span>3<\/span> This block-wise processing is particularly vital for efficiently handling high-resolution images. To ensure robust and stable training, the architecture incorporates two types of residual connections: one over hidden layers inside the causal Transformer, and another over latents.<\/span>2<\/span> These connections are instrumental in achieving training stability, making TarFlow as straightforward to train as a standard Transformer.<\/span>2<\/span><\/p>\n <\/p>\n
2.3. Mathematical Underpinnings<\/b><\/h4>\n
<\/p>\n
The underlying mathematical principle of Normalizing Flows, the change of variable formula, is central to TarFlow\u2019s ability to compute exact likelihoods.<\/span>2<\/span> The block autoregressive architecture is inspired by prior autoregressive normalizing flows.<\/span>10<\/span> This design enables end-to-end training with a single loss function, ensuring consistency between encoding and decoding processes.<\/span>10<\/span> The causal masking within the Transformer blocks is essential for enforcing the autoregressive property, which in turn ensures the tractability of Jacobian determinants\u2014a core mathematical component required for Normalizing Flows.<\/span><\/p>\nThe synergistic power of Transformers and Normalizing Flows is evident in TarFlow’s design. Normalizing Flows provide a robust mathematical framework for exact likelihood computation and guaranteed invertibility, properties highly desirable for probabilistic models. Transformers, particularly Vision Transformers, offer unparalleled expressive power and scalability for modeling complex, long-range dependencies in high-dimensional data such as images.<\/span>2<\/span> The strategic integration of the causal Vision Transformer is particularly clever; its causal masking precisely enables the efficient implementation of the autoregressive property, where each part of the output depends only on previously generated parts. Crucially, this design allows for a parallel computational graph during training <\/span>2<\/span>, which historically was a major bottleneck for previous autoregressive models that often required sequential processing even during training. The block autoregression fashion further enhances this efficiency by processing groups of pixels or patches rather than individual pixels, making it highly suitable for high-resolution image generation.<\/span>3<\/span> This represents a compelling illustration of how combining two powerful, complementary paradigms can overcome the individual limitations of each, resulting in a more robust and performant system. This architectural innovation not only makes Normalizing Flows competitive with other state-of-the-art generative models but also suggests a broader, emerging trend in AI research: the strategic integration of strong, general-purpose neural network architectures (like Transformers) into specialized probabilistic models to significantly enhance their capabilities. This approach is often more fruitful than attempting to invent entirely new model classes from scratch, as it leverages established strengths while addressing specific weaknesses.<\/span><\/p>\nHowever, a fundamental trade-off exists between training parallelism and inference sequentiality in TarFlow. While the research consistently highlights the advantages of TarFlow’s design, such as its parallel computational graph during training <\/span>2<\/span> and the resulting improved scalability and training stability <\/span>2<\/span>, a significant counterpoint is also consistently emphasized: the causal form of attention inherently requires sequential computation, making TarFlow’s sampling process extremely slow.<\/span>8<\/span> This inherent sequentiality during inference, where each new patch or block depends on all previously generated patches, restricts parallel computation during inference, leading to slow generation that impedes practical deployment.<\/span>10<\/span> This clearly illustrates a fundamental design constraint in highly expressive autoregressive generative models: optimizing for efficient and scalable training often introduces a bottleneck in the inference (sampling) phase. This is a critical design challenge that subsequent research, such as the development of the GS-Jacobi method, actively aims to mitigate. This trade-off is not unique to TarFlow but is a common challenge across many complex generative models. It reveals that optimizing for one phase, such as training efficiency and scalability, can inadvertently introduce significant limitations in another, such as inference speed. Future research in generative AI will likely continue to explore innovative methods to decouple these dependencies or to find clever approximations and iterative solvers that allow for faster sampling without sacrificing the high quality or theoretical guarantees achieved during training. This ongoing tension between training and inference efficiency will drive significant advancements in the field.<\/span><\/p>\n <\/p>\n
3. Enhancing TarFlow Performance: Key Techniques and Optimizations<\/b><\/h3>\n
<\/p>\n
3.1. Techniques for Improving Sample Quality<\/b><\/h4>\n
<\/p>\n
TarFlow’s impressive generative capabilities are not solely due to its architectural design but are also significantly enhanced by several key techniques aimed at improving sample quality:<\/span><\/p>\n\n- Gaussian Noise Augmentation During Training:<\/b> This technique involves adding a moderate amount of Gaussian noise to the input data during the model’s training phase. The research indicates that this is critical for producing high-quality samples.<\/span>2<\/span> This strategy is deemed essential for perceptual quality <\/span>3<\/span> and effectively enriches the support of the training distribution, thereby improving the generalization of the inverse model.<\/span>2<\/span> The observation that using narrow uniform noise, commonly employed for dequantization, leads to constant numerical issues and an inability to produce sensible outputs during sampling <\/span>2<\/span> further underscores that the type and magnitude of noise are not minor implementation details but crucial design choices that profoundly impact the model’s ability to capture the true underlying data distribution and generate high-fidelity samples. This also hints at a deeper conceptual connection to diffusion models, which inherently rely on controlled noise processes for their generative capabilities. This finding provides a crucial guideline for future research in Normalizing Flow training strategies, suggesting that noise augmentation should be viewed as a fundamental component for achieving high generative quality, rather than just a technical workaround for data types. It encourages a re-evaluation of how noise influences the optimization landscape and the model’s capacity to generalize, potentially leading to more sophisticated noise scheduling or adaptive noise strategies.<\/span><\/li>\n
- Post-Training Denoising Procedure:<\/b> Following the training of the model, a straightforward, training-free technique is applied to effectively denoise the generated samples.<\/span>4<\/span> This procedure utilizes only the TarFlow model itself <\/span>2<\/span> and is specifically designed to address the challenge of models trained on noisy distributions potentially mimicking noisy training examples in their outputs.<\/span>2<\/span><\/li>\n
- Effective Guidance Methods:<\/b> TarFlow incorporates guidance methods that are applicable in both class-conditional and unconditional generation settings.<\/span>4<\/span> These models are compatible with guidance methods, offering similar flexibility to diffusion models <\/span>2<\/span>, including a principled score-based guidance algorithm <\/span>15<\/span>, which enhances the model’s ability to seek out specific modes in the data distribution and provides greater control during inference.<\/span><\/li>\n<\/ul>\n
<\/p>\n
3.2. Addressing Sampling Efficiency: The Sequential Bottleneck and Iterative Solutions<\/b><\/h4>\n
<\/p>\n
Despite its parallel training capabilities, the autoregressive nature of TarFlow fundamentally limits parallel computation during inference. This is because the causal form of attention requires sequential computation, which makes TarFlow’s sampling process extremely slow.<\/span>7<\/span> This sequential modeling inherently restricts parallel computation during inference, leading to slow generation that impedes practical deployment.<\/span>10<\/span><\/p>\nTo overcome this significant sampling bottleneck, the Gauss-Seidel-Jacobi (GS-Jacobi) iteration method has been introduced. This technique substantially accelerates TarFlow sampling <\/span>7<\/span> by transforming the nonlinear recurrent neural network inherent in the TarFlow sampling phase into a diagonalized nonlinear system that can be solved iteratively.<\/span>7<\/span><\/p>\nFor optimizing this iterative sampling process, two crucial metrics have been developed: the Convergence Ranking Metric (CRM) and the Initial Guessing Metric (IGM). Researchers discovered that blocks within the TarFlow model exhibit varying importance.<\/span>7<\/span> Some blocks play a major role and are sensitive to initial values, making them prone to numerical overflow, while others are more robust.<\/span>7<\/span><\/p>\n\n- CRM<\/b> is utilized to identify whether a TarFlow block is “simple” (converges in few iterations) or “tough” (requires more iterations).<\/span>7<\/span><\/li>\n
- IGM<\/b> evaluates the suitability of the initial value for the iterative process, which helps reduce the probability of numerical overflow and accelerates convergence.<\/span>7<\/span><\/li>\n<\/ul>\n
Leveraging these observations, the Selective Jacobi Decoding (SeJD) strategy was proposed. This advanced strategy capitalizes on the finding that models tend to exhibit low dependency redundancy in the initial layers and higher redundancy in subsequent layers.<\/span>10<\/span> By applying parallel iterative optimization specifically on layers with higher redundancy <\/span>10<\/span>, SeJD significantly accelerates autoregressive inference.<\/span>10<\/span> This method boasts a superlinear convergence rate and guarantees that the number of iterations required is no greater than the original sequential approach.<\/span>10<\/span><\/p>\nExperiments have demonstrated substantial speed improvements, achieving up to 4.7 times faster inference while maintaining the generation quality and fidelity.<\/span>10<\/span> Specific speed-ups include 4.53x in Img128cond, 5.32x in AFHQ, 2.96x in Img64uncond, and 2.51x in Img64cond, all achieved without degrading Frechet Inception Distance (FID) scores.<\/span>7<\/span><\/p>\nThis significant algorithmic effort, specifically designed to overcome the sampling bottleneck, highlights the importance of interdisciplinary research, particularly the convergence of deep learning architecture design with principles from numerical analysis, such as iterative solvers. The understanding that blocks in the TarFlow model have varying importance and that dependency redundancy varies substantially across different layers allows for a selective and adaptive acceleration strategy. This approach demonstrates a deep understanding of the model’s internal computational graph and how to exploit its properties for efficiency. Such an approach highlights that even with a fixed, powerful architecture, substantial performance gains can still be achieved through clever algorithmic design and numerical methods. This development underscores the critical importance of optimizing the decoding or sampling process, suggesting that future advancements in generative models, especially those with inherent sequential inference steps, may increasingly come from these types of algorithmic innovations rather than solely from further architectural changes. This opens up promising avenues for applying similar iterative acceleration techniques to a wider range of autoregressive models across different domains.<\/span><\/p>\nTable 1: TarFlow Sampling Acceleration Results with GS-Jacobi Iteration<\/b><\/p>\n\n\n\n| Model Configuration<\/span><\/td>\n | Speed-up Factor<\/span><\/td>\n | FID Score Maintained?<\/span><\/td>\n | Source<\/span><\/td>\n<\/tr>\n\n| Img128cond<\/span><\/td>\n | 4.53x<\/span><\/td>\n | Yes<\/span><\/td>\n | 7<\/span><\/td>\n<\/tr>\n\n| AFHQ<\/span><\/td>\n | 5.32x<\/span><\/td>\n | Yes<\/span><\/td>\n | 7<\/span><\/td>\n<\/tr>\n\n| Img64uncond<\/span><\/td>\n | 2.96x<\/span><\/td>\n | Yes<\/span><\/td>\n | 7<\/span><\/td>\n<\/tr>\n\n| Img64cond<\/span><\/td>\n | 2.51x<\/span><\/td>\n | Yes<\/span><\/td>\n | 7<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n <\/p>\n 4. Performance Benchmarks and State-of-the-Art Results<\/b><\/h3>\n <\/p>\n 4.1. Likelihood Estimation Performance<\/b><\/h4>\n <\/p>\n TarFlow has established new state-of-the-art results in likelihood estimation for images, significantly surpassing previous methods by a considerable margin.<\/span>2<\/span> A landmark achievement is its pioneering success in reaching a sub-3 BPD (bits per dimension) on ImageNet 64×64, specifically reporting 2.99 BPD.<\/span>2<\/span> This performance markedly outperforms prior leading methods.<\/span><\/p>\nThe achievement of a sub-3 BPD on ImageNet 64×64 for the first time is a highly specific and quantitative milestone in generative modeling. BPD is a direct, information-theoretic measure of how effectively a model learns to compress or represent the true underlying data distribution. A lower BPD signifies a more accurate and efficient model of the data’s true probability density. Breaking the “3 BPD” barrier on a complex dataset like ImageNet 64×64 represents a substantial improvement in the fidelity of the learned distribution, which is foundational not only for density estimation but also for generating high-quality samples. This is a crucial scientific benchmark that speaks directly to the model’s fundamental capacity to understand and represent complex image statistics. This result solidifies TarFlow’s position as a leading model for density estimation, a capability that extends beyond mere image generation into diverse applications such as anomaly detection, data compression, and various scientific modeling tasks where a precise understanding of probability distributions is required. It implicitly confirms that the architectural innovations, specifically the use of Transformers, and the refined training techniques, such as Gaussian noise augmentation, are exceptionally effective in capturing the intricate, high-dimensional data distributions found in real-world images.<\/span><\/p>\nTable 2: TarFlow Likelihood Estimation Performance (Bits Per Dimension – BPD) on ImageNet 64×64 (Unconditional)<\/b><\/p>\n\n\n\n| Model Type<\/span><\/td>\n | BPD \u2193<\/span><\/td>\n | Source<\/span><\/td>\n<\/tr>\n\n| TARFLOW<\/b><\/td>\n | 2.99<\/b><\/td>\n | 2<\/span><\/td>\n<\/tr>\n\n| NFDM<\/span><\/td>\n | 3.20<\/span><\/td>\n | 2<\/span><\/td>\n<\/tr>\n\n| Flow Matching<\/span><\/td>\n | 3.31<\/span><\/td>\n | 2<\/span><\/td>\n<\/tr>\n\n| VDM<\/span><\/td>\n | 3.40<\/span><\/td>\n | 2<\/span><\/td>\n<\/tr>\n\n| Improved DDPM<\/span><\/td>\n | 3.54<\/span><\/td>\n | 2<\/span><\/td>\n<\/tr>\n\n| Sparse Transformer<\/span><\/td>\n | 3.44<\/span><\/td>\n | 2<\/span><\/td>\n<\/tr>\n\n| Routing Transformer<\/span><\/td>\n | 3.43<\/span><\/td>\n | 2<\/span><\/td>\n<\/tr>\n\n| SPN<\/span><\/td>\n | 3.52<\/span><\/td>\n | 2<\/span><\/td>\n<\/tr>\n\n| PixelCNN<\/span><\/td>\n | 3.83<\/span><\/td>\n | 2<\/span><\/td>\n<\/tr>\n\n| Flow++<\/span><\/td>\n | 3.69<\/span><\/td>\n | 2<\/span><\/td>\n<\/tr>\n\n| Glow<\/span><\/td>\n | 3.81<\/span><\/td>\n | 2<\/span><\/td>\n<\/tr>\n | | | | | | | | | | | | | | | | |
![\"\"](\"https:\/\/uplatz.com\/blog\/wp-content\/uploads\/2025\/06\/Blog-images-new-set-A-10-1.png\")
<\/p>\n