Architectures and Strategies for Scalable Multi-Model Serving

Executive Summary

This report provides a comprehensive analysis of multi-model serving (MMS), a critical paradigm for efficiently deploying large numbers of machine learning models on shared infrastructure. We deconstruct the limitations of traditional single-model deployments and establish the business imperative for MMS, focusing on significant cost, resource, and energy savings. The report details foundational architectural patterns—including dynamic model management, overcommit strategies, multiplexing, and composition—and provides an in-depth comparative analysis of leading open-source frameworks (NVIDIA Triton, KServe/ModelMesh, Seldon Core, Ray Serve, BentoML) and managed cloud services (AWS, GCP, Azure). We address advanced challenges such as handling heterogeneous workloads and present strategic recommendations derived from performance benchmarks and real-world case studies. This document is intended to serve as a definitive guide for MLOps engineers and architects tasked with building scalable, robust, and cost-effective AI inference platforms.

bundle-combo—sap-finance-fico-and-s4hana-finance By Uplatz

The Imperative for Multi-Model Serving

Deconstructing the Single-Model Serving Paradigm and Its Limitations

The standard and most straightforward pattern for productionizing machine learning (ML) models involves packaging a single model artifact within a container image. This container is then deployed as a microservice, typically managed by a service orchestration framework such as Kubernetes.1 This approach creates a direct, one-to-one mapping between a deployed container and the ML model being served. While this pattern is effective for organizations deploying a handful of high-value models, its inherent inefficiencies become pronounced and operationally untenable as the number of models scales into the dozens, hundreds, or even thousands.1

The primary limitations of the single-model serving paradigm are rooted in resource inefficiency and operational friction at scale:

• Resource Over-provisioning and Cost Inefficiency: Each model container reserves a dedicated slice of computational resources, including CPU, memory, and often expensive specialized hardware like GPUs or TPUs. When an organization needs to operationalize a large number of models—many of which may have sporadic or low traffic—this pattern leads to systemic underutilization. Resources are provisioned and paid for regardless of whether the model is actively processing inference requests, forcing a setup where the infrastructure is massively over-provisioned, unnecessarily increasing both capital expenditure and operational costs.1
• Infrastructure Overhead and Scalability Ceilings: Beyond the resources required by the model itself, each container introduces its own operational overhead, including the operating system, server process, and monitoring agents. This overhead, while negligible for a few deployments, becomes a significant resource drain when multiplied across hundreds or thousands of containers.1 Furthermore, this approach can encounter hard technological limits imposed by the orchestration layer. For instance, Kubernetes has a default limit of 110 pods per node. To deploy 20,000 models using a single-pod approach would necessitate nearly 200 nodes, consuming a minimum of 200 GB of memory even with the smallest cloud instances, a figure that is a significant underestimation in practice.1
• High Latency from Cold Starts: In scenarios where models are not kept running continuously to save costs, the single-model-per-container pattern suffers from severe “cold start” latency. Before an on-demand inference request can be served, the entire container image must be downloaded from a registry, and the model server process must be initiated. This process can take on the order of tens of minutes, which is unacceptable for any real-time or interactive application.1

The adoption of a multi-model serving architecture is therefore not merely a cost-saving tactic but a necessary architectural evolution for any organization seeking to achieve MLOps maturity. As companies move from deploying a few “hero” models to leveraging hundreds of personalized or segmented models—a pattern increasingly common in domains like e-commerce, content recommendation, and finance—the single-model approach becomes an operational bottleneck.2 MMS represents a strategic shift from treating models as monolithic, resource-heavy applications to managing them as lightweight, dynamic artifacts within a shared, efficient serving fabric.

 

Defining Multi-Model Serving: A Paradigm for Efficiency and Scale

 

Multi-model serving (MMS) is an architectural paradigm designed to overcome the limitations of single-model deployment by intelligently scheduling and hosting multiple, distinct ML models on shared server infrastructure.1 The fundamental principle of MMS is to break the rigid one-to-one mapping between a model and its containerized environment. Instead of deploying a dedicated container for each model, MMS utilizes long-lived, shared model inference servers that are capable of dynamically loading, serving, and unloading multiple models within the same process or pool of processes.1

This paradigm shift moves the unit of resource sharing from the coarse-grained container level to the much more fine-grained model level.1 A single, resource-intensive server process, running on a powerful instance with CPU or GPU resources, can concurrently serve inference requests for a multitude of models. This is analogous to the evolution of computing infrastructure from physical servers to virtual machines and subsequently to containers, where each successive step unlocked greater resource density, flexibility, and operational efficiency. MMS applies this same principle of consolidation and shared tenancy directly to the domain of machine learning inference.

This approach is particularly critical for managing modern, heterogeneous model portfolios. Organizations today rarely deploy a single type of model. Their portfolios often include a diverse mix of large, GPU-intensive Large Language Models (LLMs), smaller CPU-based classical ML models (e.g., from scikit-learn or XGBoost), and various deep learning models for computer vision or natural language processing.1 Some of these models may handle high, consistent traffic, while others are invoked only sporadically. The single-model pattern’s “one-size-fits-all” resource allocation is profoundly inefficient for this reality. MMS, by its nature, is designed to handle this heterogeneity by pooling diverse hardware resources and allocating them dynamically based on real-time demand, making it the default architectural choice for managing a modern AI/ML platform at scale.

 

Core Objectives and Business Value: Cost, Resource, and Energy Optimization

 

The primary driver and central objective of adopting a multi-model serving strategy is to enable the scalable deployment of a large and growing portfolio of ML models on a significantly smaller infrastructure footprint. This consolidation directly translates into substantial and quantifiable business value through cost reduction, enhanced resource utilization, and improved energy efficiency.1

The key benefits derived from this approach include:

• Drastic Reduction in Memory and Compute Footprint: By sharing the base resources of the server process and underlying hardware, the aggregate memory footprint of the system can be reduced by an order of magnitude compared to deploying each model in its own container.1 This allows organizations to host hundreds of models on a small cluster of nodes, rather than requiring hundreds of nodes.
• Maximized Utilization of Specialized Hardware: Expensive and powerful hardware accelerators like GPUs are often underutilized in a single-model deployment, especially if the model’s traffic is not high enough to fully saturate the device’s computational capacity. MMS allows multiple models to share a single GPU, processing inference requests concurrently and thereby maximizing the return on investment for this specialized hardware.1
• Elimination of Container-Level Cold Start Latency: Because MMS relies on long-lived, continuously running inference servers, the time-consuming process of downloading a container image and initializing a server for each model deployment is eliminated. While there may be a minor latency penalty for loading a new model into the server’s memory for the first time, this is orders of magnitude faster than a full container cold start, which can take several minutes.1
• Substantial Cost and Energy Savings: The cumulative effect of reduced infrastructure footprint, higher resource density, and maximized hardware utilization leads directly to significant reductions in cloud computing bills and overall operational expenditure. This efficiency also translates into lower energy consumption, contributing to an organization’s sustainability goals.1

 

Key Performance Metrics in a Multi-Model Context: Latency, Throughput, and Staleness

 

When evaluating the performance of a multi-model serving system, it is essential to consider a set of key metrics that capture the trade-offs inherent in this architecture. While these metrics are common to all serving systems, their interplay is particularly nuanced in an MMS environment.10

• Latency: Defined as the time elapsed from the moment an inference request is sent to when the prediction is received by the user. In an MMS context, latency is not a single static value. It is critically affected by the state of the requested model within the server’s cache. A request for a “hot” model already loaded in memory will have very low latency. However, a request for a “cold” model that has been evicted from memory or has never been loaded will incur an additional latency penalty for loading the model from storage into memory before inference can occur.10 Managing and minimizing this cold-start latency is a central challenge in MMS design.
• Throughput: Defined as the number of predictions the system can generate per unit of time (e.g., inferences per second). MMS architectures can significantly enhance throughput by employing techniques such as dynamic request batching, which groups multiple requests into a single batch for more efficient, vectorized computation on hardware like GPUs, and parallel execution, where multiple model instances process requests simultaneously.10
• Staleness and Time Sensitivity: Staleness refers to the age of the data or model used to generate a prediction relative to the time of the request. This is most relevant in asynchronous or batch prediction architectures, where predictions are pre-computed.10 MMS typically operates in an online, synchronous serving pattern, where predictions are computed in real-time upon user request. This makes MMS ideal for use cases with high time sensitivity, where the value of a prediction decays rapidly with time and requires the most up-to-date input data.6

 

Foundational Architectural Patterns and Optimization Techniques

 

The effectiveness of a multi-model serving system hinges on a set of core architectural patterns and optimization techniques designed to manage resources intelligently and process requests efficiently. These strategies address the fundamental tension between maximizing model density to reduce costs and minimizing latency to ensure a responsive user experience.

 

Model Co-hosting and Resource Pooling

 

The foundational principle of MMS is the co-hosting of multiple models within a single, shared environment, enabling them to draw from a common pool of computational resources.1 Instead of isolating each model in its own resource-siloed container, models are managed by a central inference server process. This server is deployed on an instance with a substantial allocation of CPU, memory, and potentially GPU resources. All models loaded by this server share these underlying resources.

This resource pooling is implemented through specialized, multi-model inference servers, which are engineered to handle the lifecycle—loading, serving, and unloading—of numerous models concurrently within a single process. Prominent examples of such servers include NVIDIA Triton Inference Server and MLServer, which often form the high-performance data plane for more comprehensive MLOps platforms.14 By consolidating models onto these powerful servers, organizations can achieve a high density of models per hardware unit, which is the primary mechanism for improving resource utilization and reducing infrastructure costs.

 

Dynamic Model Management: Intelligent Loading, Unloading, and Caching

 

A central challenge in co-hosting a large number of models is that their aggregate size often far exceeds the available memory (e.g., GPU VRAM) of the serving infrastructure. To solve this, MMS platforms employ dynamic model management, treating the server’s memory as a cache for model artifacts rather than as permanent storage.

This dynamic management involves two key processes:

1. Dynamic Loading: Models are not pre-loaded into memory when the server starts. Instead, a model is loaded on-demand, typically when the very first inference request for that specific model arrives at the server.12 The server fetches the model’s artifacts from a persistent storage location (like an S3 bucket), loads them into active memory, and then processes the request. This “lazy loading” approach introduces a one-time “cold start” latency penalty for the first request to an unloaded model but is essential for managing a large catalog of models that cannot all fit in memory simultaneously.11
2. Intelligent Unloading and Caching: To make room for newly requested models when memory is full, the server must decide which existing models to evict. This decision is governed by a caching policy.

 

The Role of Least Recently Used (LRU) Caching

 

The most prevalent eviction policy used in MMS systems is Least Recently Used (LRU). Under this policy, the server maintains a record of when each loaded model was last used for an inference request. When memory pressure requires a model to be unloaded, the server chooses the one that has been inactive for the longest period.1 This model is then evicted from high-speed memory (like GPU VRAM) and may be moved to a “warm” storage tier, such as system RAM or a local disk cache.20 If a request for this evicted model arrives later, it can be reloaded from this warm storage much faster than from the original remote object store, mitigating the latency penalty of a full cold start.1

 

The “Overcommit” Strategy: Serving Beyond Memory Constraints

 

The “Overcommit” strategy is a formalization of this dynamic caching mechanism, explicitly allowing a server to register and manage a catalog of models whose total memory requirement exceeds the provisioned physical memory.1 This approach is built on the assumption that not all models will be active at the same time; traffic patterns are often complementary, with different models being popular at different times of the day or week.1

By overcommitting resources, the system can provide access to a vast library of models while only paying for the infrastructure needed to serve the actively used subset at any given moment. The LRU cache acts as the dynamic arbiter, automatically promoting frequently used models to active memory and demoting inactive ones to warm storage, thus striking a dynamic balance between cost and performance.

 

Advanced Serving Patterns

 

Beyond simple co-hosting, MMS enables more sophisticated architectural patterns that allow for greater flexibility in routing requests and building complex applications.

 

Model Multiplexing: Routing by Request

 

Model multiplexing is a powerful pattern designed for scenarios with a large number of sparsely invoked models that are structurally similar (e.g., they share the same model architecture and input/output schema but have different trained weights, such as per-user or per-region personalization models).19

In this pattern, a single logical deployment, consisting of a pool of identical server replicas, is capable of serving any of the models in the collection. The key mechanism is request-based routing. The client application includes a specific model_id in the header of its inference request. The serving framework’s internal router inspects this header and intelligently directs the request to a replica that already has that particular model loaded and cached in its memory.19 This routing logic minimizes unnecessary model loading operations. If no replica currently has the requested model cached, the router selects one (e.g., the least loaded replica) to load the model from storage and serve the request. This pattern dramatically improves resource efficiency for use cases involving thousands of fine-tuned model variants.

 

Model Composition and Inference Graphs: Building Complex Pipelines

 

Model composition, also known as inference graphs or ensembles, is a pattern that moves beyond serving single models to building complex, multi-step inference pipelines by chaining multiple models and business logic components together.6

The entire workflow is defined as a Directed Acyclic Graph (DAG), where the output of one node (e.g., a pre-processing function or a model) serves as the input to the next node in the graph.6 This is essential for many real-world AI applications that require more than a single model prediction. For example, an optical character recognition (OCR) application might be composed as a two-stage graph: the first model detects bounding boxes of text in an image, and its output is then passed to a second model that recognizes and transcribes the text within those boxes.24 This pattern allows developers to build and deploy an entire end-to-end AI application as a single, servable unit, abstracting away the complexity of coordinating multiple independent services.

 

Performance Optimization at the Request Level

 

To maximize throughput and hardware utilization, MMS frameworks employ several optimizations at the level of individual requests.

 

Dynamic and Continuous Request Batching

 

Request batching is a critical technique for improving inference throughput, especially on hardware accelerators like GPUs that are designed for parallel computation. Instead of processing each incoming request individually, the server groups them together into a single batch and performs inference on the entire batch at once.13

• Dynamic Batching: This is the most common form of server-side batching. The server buffers incoming requests for a short, configurable time window (e.g., 10 milliseconds) or until a maximum batch size (e.g., 32 requests) is reached. Whichever condition is met first triggers the processing of the batch.13 This mechanism creates a dynamic trade-off: a longer wait time allows for larger, more efficient batches but increases the latency for the first requests in the batch.
• Continuous Batching: This is a more advanced optimization specifically for autoregressive models like LLMs, where the time to generate a full response varies significantly depending on the desired output length. In traditional dynamic batching, the entire batch is blocked until the longest-running request is complete. Continuous batching improves upon this by allowing new requests to be added to the batch as soon as slots become free from completed requests, thereby eliminating idle GPU time and significantly increasing overall throughput.33

 

Concurrency and Parallel Execution Models

 

High-performance inference servers are designed to exploit both concurrency (managing multiple tasks by interleaving them) and parallelism (executing multiple tasks simultaneously).34

In the context of MMS, concurrent model execution refers to the ability of a server like NVIDIA Triton to run inference for different models, or even multiple instances of the same model, in parallel on a single GPU.38 When requests for different models arrive, the server can schedule them onto the GPU’s hardware compute streams simultaneously. This allows a single GPU to be effectively shared by multiple active models at the same time, ensuring that its powerful processing capabilities are not left idle and are fully utilized to serve a diverse request load.

The interplay of these optimization techniques reveals a core architectural principle: there is a fundamental tension between resource efficiency (achieved through high model density) and request latency (achieved through predictable, low-latency access). Techniques like dynamic loading and overcommit prioritize density at the cost of potential cold-start latency. Conversely, pre-loading all models guarantees low latency but is economically unfeasible at scale. The choice of which patterns and optimizations to employ depends directly on the specific Service Level Objectives (SLOs) for an application and the organization’s tolerance for infrastructure costs versus performance variability.

Furthermore, the evolution from generic optimizations like dynamic batching to workload-aware techniques like continuous batching for LLMs indicates a clear trend. Future MMS optimizations will likely become increasingly specialized and “model-aware.” Instead of relying on simple heuristics like LRU, these systems may incorporate predictive models to forecast resource usage and execution times, enabling more intelligent and proactive scheduling and caching decisions.39

Finally, the model composition pattern elevates MMS from a simple hosting utility to a powerful framework for building applications. It allows developers to construct complex, distributed AI systems by composing models and business logic as if they were simple function calls, particularly within programmable, Python-native frameworks.23 This transforms the problem from merely hosting models to architecting sophisticated, distributed AI applications, representing a more powerful and flexible paradigm for production ML.

 

Deep Dive: Kubernetes-Native Serving Frameworks

 

For organizations that have standardized on Kubernetes as their orchestration platform, a class of Kubernetes-native serving frameworks has emerged. These tools leverage Kubernetes’ Custom Resource Definitions (CRDs) to provide a declarative, infrastructure-as-code approach to deploying and managing ML models at scale. They integrate deeply with the Kubernetes ecosystem for networking, scaling, and observability.

 

NVIDIA Triton Inference Server: High-Performance, Multi-Framework Execution

 

NVIDIA Triton Inference Server is an open-source inference serving software optimized for high-performance, low-latency inference, particularly on NVIDIA GPUs.27 While it can be run as a standalone server, it is often used as the high-performance “data plane” or execution engine within more comprehensive MLOps platforms like KServe and Seldon Core.26

 

Architecture: Backends, Schedulers, and Concurrent Model Execution

 

Triton’s architecture is designed for versatility and performance. Its core components include:

• Pluggable Backend System: Triton is framework-agnostic, supporting a wide array of ML frameworks such as TensorRT, PyTorch, TensorFlow, ONNX, and even custom Python backends through a standardized backend API.27 This allows it to serve a heterogeneous collection of models from a single server instance.
• Schedulers and Concurrent Execution: Triton’s architecture is built for parallelism. It can manage multiple instances of the same model and execute inference requests for different models concurrently on the same GPU, leveraging the hardware’s parallel processing capabilities to maximize throughput and utilization.38

 

Multi-Model Features

 

Triton is inherently a multi-model server, designed from the ground up to manage and serve a large number of models simultaneously.

• Model Repository: Triton uses a specific file-based model repository structure. Each model resides in its own subdirectory, which contains the model artifacts and a config.pbtxt file that defines its properties (e.g., input/output tensors, batching strategy, instance count). Triton monitors this repository and can dynamically load, unload, or update models without restarting the server.45
• Dynamic Batching: Dynamic batching is a first-class feature in Triton. It can be enabled and finely tuned on a per-model basis via the model configuration file. The server automatically batches incoming requests to form larger tensors for inference, which dramatically increases throughput on GPUs.13
• Model Ensembles and Business Logic Scripting (BLS): Triton supports the creation of complex inference pipelines, or “ensembles,” directly within the server. An ensemble is defined as a Directed Acyclic Graph (DAG) in a model’s configuration file, specifying how the output of one model should be routed as the input to another. This allows for entire multi-model workflows, including pre- and post-processing steps written in Python (using BLS), to be executed with a single request from the client, minimizing network latency.27

 

KServe and the ModelMesh Architecture: High-Density, Serverless Serving

 

KServe is a highly scalable, Kubernetes-native model serving platform that provides a standardized InferenceService CRD to simplify the deployment of ML models.43 It features a clean architectural separation between its control plane, which manages the lifecycle of deployments in Kubernetes, and its data plane, which handles the inference requests.47

 

Architecture: Control Plane, Data Plane, and the InferenceService CRD

 

• InferenceService CRD: This is the core abstraction in KServe. A data scientist or ML engineer defines a single YAML manifest specifying the model’s location, the required serving runtime, and other configurations. KServe’s controller then automates the creation of all the necessary underlying Kubernetes resources (Deployments, Services, etc.).46
• Serverless Deployment: By default, KServe integrates with Knative to provide serverless deployment capabilities. This enables request-based autoscaling, including the ability to scale the number of model replicas down to zero when there is no traffic, offering significant cost savings for models with intermittent or unpredictable workloads.46
• Pluggable Runtimes: KServe is not an inference server itself but an orchestration platform. It supports a variety of pluggable model serving runtimes, including NVIDIA Triton, TorchServe, and its own lightweight Python servers for common frameworks like Scikit-learn and XGBoost.43

 

ModelMesh Deep Dive: Distributed LRU Caching and Intelligent Placement

 

For use cases requiring very high model density and frequent model changes, KServe offers an alternative deployment mode called ModelMesh.20 ModelMesh is a specialized multi-model serving management layer optimized for hosting hundreds or thousands of models efficiently.

• Core Architecture: ModelMesh operates a pool of generic serving pods and treats their combined memory as a single, distributed LRU cache for models.20 Instead of deploying a dedicated pod for each model, ModelMesh intelligently loads models into this shared pool of pods as needed.
• Intelligent Placement and Caching: When a request for a model arrives, ModelMesh determines the optimal pod to load it onto, balancing factors like memory availability, request load, and the “age” of the cache on each pod.20 As memory becomes constrained, it automatically evicts the least recently used models from the cache across the entire cluster to make room for new ones.20 This distributed, cluster-aware caching is a more sophisticated and resilient approach than a simple per-pod cache, making ModelMesh exceptionally well-suited for high-density, dynamic environments.21

 

Seldon Core 2: Data-Centric MLOps and Advanced Routing

 

Seldon Core is a comprehensive, Kubernetes-native MLOps framework designed to package, deploy, monitor, and manage thousands of production ML models.22 It provides a rich set of features for building complex inference graphs and managing the model lifecycle in production.

 

Architecture: MLServer, Inference Graphs, and the Open Inference Protocol

 

• Inference Graphs: Seldon’s core abstraction is the SeldonDeployment CRD, which allows users to define complex inference graphs. These graphs can be composed of various components, including predictors (the models themselves), transformers (for pre/post-processing), and routers (for traffic splitting).25
• Integrated Model Servers: Seldon Core orchestrates the deployment but relies on specialized inference servers for execution. It is tightly integrated with MLServer, its own high-performance Python-based server, and also provides first-class support for NVIDIA Triton.26
• Standardized Protocol: Seldon Core uses the Open Inference Protocol (also known as the KFServing V2 protocol), which provides a standard specification for inference requests and responses over both REST and gRPC. This ensures interoperability between different components and servers in the inference graph.26

 

Multi-Model Features

 

Seldon Core is explicitly designed to support multi-model serving patterns and advanced MLOps workflows.

• Multi-Model Serving (MMS) and Overcommit: Seldon’s architecture fully embraces MMS to optimize resource utilization and reduce costs.26 Through its integration with MLServer, it supports the “overcommit” strategy, using an LRU cache to manage more models than can fit in memory and dynamically swapping them as needed.1
• Experiments and Advanced Routing: A key differentiator for Seldon is its first-class support for advanced deployment strategies and experimentation. The inference graph can include sophisticated routers that split traffic between different models or even different sub-graphs. This enables MLOps teams to easily implement canary deployments, A/B tests, and shadow deployments to safely validate and roll out new model versions in production with live traffic.22 This positions Seldon not just as a serving tool, but as a complete framework for continuous deployment in ML.

The architectures of these Kubernetes-native frameworks reveal a clear and powerful trend toward a layered or decoupled design. Orchestration platforms like KServe and Seldon Core excel at the “control plane” functions: providing standardized Kubernetes abstractions (CRDs), managing the model lifecycle, and handling complex traffic routing. They then delegate the high-performance “data plane” function—the actual, computationally intensive model inference—to specialized and highly optimized servers like NVIDIA Triton and MLServer. This separation of concerns allows organizations to leverage best-in-class tools for each layer of the stack, combining robust MLOps capabilities with raw inference performance.

 

Deep Dive: Python-Native and Developer-Centric Frameworks

 

In contrast to the Kubernetes-native, YAML-driven frameworks, another class of tools has emerged that prioritizes a Python-native, code-first developer experience. These frameworks allow ML engineers and data scientists to define, build, and deploy complex distributed serving applications directly in Python, often abstracting away the underlying infrastructure complexities.

 

Ray Serve: Scalable and Programmable Serving for Distributed Applications

 

Ray Serve is a scalable model serving library built on top of Ray, an open-source framework for building and running distributed applications in Python.9 It is designed to be highly flexible and programmable, making it particularly well-suited for complex, multi-model inference pipelines and end-to-end AI applications.

 

Architecture: Built on the Ray Actor Model

 

Ray Serve’s architecture is fundamentally based on Ray’s core primitive: the Actor. A Ray Actor is a stateful Python object that runs in its own process, can be accessed remotely, and can execute methods.58 A Ray Serve application is composed of several types of actors:

• Controller: A global actor that manages the state of the entire Serve application, including creating, updating, and destroying other actors.
• HTTP/gRPC Proxies: Actors that run web servers (like Uvicorn) to accept incoming requests from outside the cluster. They act as the ingress point and route requests to the appropriate model replicas.
• Replicas: These are the worker actors that contain the actual model and business logic. Each replica processes inference requests sent from the proxies or from other replicas.58

 

Multi-Model Patterns

 

Ray Serve’s programmable, Python-native API makes it exceptionally powerful for implementing sophisticated multi-model patterns.23

• Model Composition: This is Ray Serve’s primary pattern for building complex inference graphs. One deployment (a group of replicas) can be “bound” to another. At runtime, this binding is replaced with a DeploymentHandle, which can be used to make asynchronous calls to the other deployment as if it were a simple Python function call. This allows developers to compose intricate DAGs of models and business logic in a clean, programmatic way, offering superior flexibility compared to static YAML definitions.23
• Multi-Application Deployment: Ray Serve allows multiple, fully independent applications to be deployed and co-hosted on the same underlying Ray cluster. Each application has its own set of deployments, route prefix, and can be updated or scaled independently of the others. This pattern is ideal for multi-tenant scenarios or for improving hardware utilization by co-locating unrelated services on the same infrastructure.62
• Model Multiplexing: Ray Serve provides a serve.multiplexed API to efficiently serve many sparsely invoked models from a single deployment. A client specifies a model_id in the request header, and Ray Serve’s router directs the request to a replica that has that model cached. If the number of models exceeds a configured limit per replica, a Least Recently Used (LRU) policy is used to evict inactive models from the cache, managing memory usage dynamically.19

 

BentoML: A Unified Framework for Building and Deploying AI Applications

 

BentoML is a comprehensive, open-source platform designed to streamline the entire ML lifecycle, from model packaging to production deployment.63 It emphasizes a developer-friendly, Python-first workflow to bridge the gap between data science and production engineering.

 

Architecture: Services, Runners, and Distributed Deployments

 

BentoML’s architecture is built around a few core concepts:

• Services: The serving logic is defined in Python classes decorated with @bentoml.service. These classes contain API endpoints (defined with @bentoml.api) that expose the model’s functionality over a REST API.67
• Runners: Model inference logic is encapsulated within Runners. This abstraction decouples the model execution from the API serving logic, allowing BentoML to apply optimizations like adaptive batching and to manage resource allocation (e.g., assigning a model to a specific GPU) independently.65
• Bentos: The entire application—including the service code, model artifacts, dependencies, and configuration—is packaged into a standardized, versioned artifact called a “Bento.” This Bento is a self-contained, reproducible unit of deployment that can be easily containerized into a Docker image.64

 

Multi-Model Features

 

BentoML supports multi-model serving through its concept of distributed services and composable architectures.

• Distributed Services: A BentoML project can define multiple Services, each of which can be deployed as an independent microservice, potentially in its own container.68 This is the primary pattern for building multi-model pipelines, where, for example, a CPU-based pre-processing service can handle data transformation before calling a GPU-based inference service.68
• Interservice Communication: BentoML facilitates communication between these distributed services via the bentoml.depends() function. A service can declare a dependency on another service, and BentoML’s runtime handles service discovery, routing, and serialization, making the remote call appear as a simple, local Python method call.68
• Adaptive Batching: BentoML provides a built-in adaptive batching mechanism that can be enabled with a simple decorator on an API endpoint. This feature automatically groups incoming requests into micro-batches to improve throughput, especially for models running on GPUs.64

The key distinction of these Python-native frameworks lies in their “programmable infrastructure” paradigm. They prioritize developer experience by allowing the entire distributed serving application to be defined and reasoned about in a single, familiar language: Python. This contrasts sharply with the declarative, YAML-based approach of Kubernetes-native tools, which enforces a separation between infrastructure configuration and application code. This code-first approach can significantly lower the barrier to entry for ML teams and accelerate iteration cycles by empowering them to own more of the deployment stack without requiring deep Kubernetes expertise.

The choice between a framework like Ray Serve and BentoML often reflects a preference in architectural philosophy. Ray Serve, built on the highly flexible Ray actor model, excels at composing arbitrary and dynamic inference graphs, making it suitable for complex, research-oriented, or rapidly evolving pipelines. BentoML’s distributed services pattern, on the other hand, encourages a more structured, traditional microservices architecture. This can lead to more maintainable and easier-to-understand systems for well-defined applications but may offer less dynamic composition flexibility than Ray Serve. Ultimately, the decision to adopt a Kubernetes-native versus a Python-native framework reflects an organization’s philosophy on the roles and responsibilities of MLOps and ML engineering teams.

 

Comparative Analysis and Framework Selection

 

Choosing the right multi-model serving framework is a critical architectural decision that depends on a variety of factors, including team expertise, existing infrastructure, performance requirements, and the desired level of operational control. This section provides a comparative analysis of the leading frameworks to guide this selection process.

 

Qualitative Comparison: Usability, Framework Support, and Ecosystem Integration

 

• Usability and Developer Experience: Frameworks like BentoML and Ray Serve are consistently recognized for their developer-friendly, Python-native APIs, which lower the barrier to entry for data science and ML engineering teams.44 They allow for rapid iteration and local development that closely mirrors the production environment. In contrast, Kubernetes-native frameworks such as KServe and Seldon Core present a steeper learning curve, requiring familiarity with Kubernetes concepts like CRDs and YAML manifests. However, for teams with strong DevOps or MLOps expertise, their declarative nature provides a powerful and standardized way to manage infrastructure as code.44 NVIDIA Triton, while powerful, can be complex to configure optimally, especially for advanced features like custom backends or ensembles.56
• Framework and Format Support: NVIDIA Triton stands out for its extensive, out-of-the-box support for a wide range of ML frameworks and optimized formats, including TensorFlow, PyTorch, ONNX, and TensorRT.41 Seldon Core and KServe achieve broad compatibility by integrating Triton as a serving runtime, alongside their own native servers.16 BentoML also supports all major frameworks, providing utilities to package models from each.64 Ray Serve is inherently framework-agnostic, as it is designed to serve any arbitrary Python code, offering maximum flexibility.23
• Ecosystem Integration: The value of a serving framework is often amplified by its integration with the broader MLOps ecosystem. KServe is a core component of the Kubeflow ecosystem, making it a natural choice for organizations already invested in that platform.46 Seldon Core offers strong integrations with tools for model monitoring and explainability, such as Prometheus and Alibi.53 Ray Serve benefits from being part of the larger Ray ecosystem, which includes libraries for distributed data processing (Ray Data), training (Ray Train), and reinforcement learning (RLlib), enabling a unified stack for both training and serving.9

 

Performance Profile Analysis: Latency, Throughput, and Scalability Considerations

 

• High-Performance Inference: For raw inference performance, especially on GPUs, NVIDIA Triton is widely regarded as the industry leader. Its architecture is meticulously optimized for high throughput and low latency, leveraging features like concurrent model execution and highly configurable dynamic batching.42 MLPerf Inference benchmarks have demonstrated that Triton can achieve performance that is nearly identical to bare-metal, non-containerized submissions, confirming that its rich feature set does not come at the cost of performance.42
• Scalability Models: The frameworks offer different approaches to scaling. KServe, through its integration with Knative, provides a serverless scaling model. This includes request-based autoscaling and the ability to scale down to zero replicas, which is extremely cost-effective for workloads with sporadic or bursty traffic patterns.46 Seldon Core and Ray Serve typically employ a replica-based scaling model, where the number of pods or actors is adjusted based on metrics like CPU utilization or queue depth, managed by either a Kubernetes Horizontal Pod Autoscaler (HPA) or Ray’s native autoscaler.58 KServe’s ModelMesh offers a unique model, maintaining a fixed pool of serving pods and dynamically scaling the models within this pool through its intelligent placement and eviction logic.20

 

Architectural Trade-offs: Kubernetes-Native vs. Python-Native Approaches

 

The choice between a Kubernetes-native and a Python-native framework represents a fundamental trade-off in architectural philosophy.

• Declarative (YAML) vs. Imperative (Python): Kubernetes-native tools like Seldon Core and KServe utilize declarative YAML manifests to define the desired state of the serving infrastructure.25 This approach enforces a clean separation of concerns between the application code (the model) and its configuration, aligning well with established GitOps and infrastructure-as-code practices. In contrast, Python-native tools like Ray Serve and BentoML use an imperative, code-first approach where the entire distributed application, including its routing logic and scaling properties, is defined in Python.23 This offers greater programmatic flexibility and a more integrated development experience for Python developers but can blur the lines between application logic and infrastructure configuration.
• Operational Model and Team Structure: Kubernetes-native solutions naturally fit into an organizational structure with a dedicated MLOps or platform team responsible for managing the Kubernetes cluster and the serving platform. Data scientists then interact with this platform by providing model artifacts and configuration files.44 Python-native frameworks empower ML engineering teams to take ownership of a larger portion of the stack, defining not just the model but the entire serving application. This can accelerate development but requires the team to have a broader skill set encompassing aspects of distributed systems and operations.

 

Decision Framework: Selecting the Right Tool for Your Use Case

 

Based on the analysis, the following decision framework can guide the selection of a multi-model serving tool:

• If the primary driver is maximizing raw GPU throughput and minimizing latency for performance-critical applications, NVIDIA Triton is the optimal choice, either used standalone or as a serving runtime within KServe or Seldon Core.42
• If the organization is deeply invested in Kubernetes and requires robust, enterprise-grade MLOps capabilities, such as advanced canary deployments, A/B testing, and deep monitoring integrations, Seldon Core and KServe are the leading contenders.44
• If the specific use case involves serving thousands of frequently changing models with a focus on high density and resource efficiency, KServe with the ModelMesh backend is purpose-built for this challenge.21
• If the primary need is to build complex, programmable inference pipelines with custom business logic, and the team is Python-centric, Ray Serve offers unparalleled flexibility and a powerful composition API.23
• If the goal is to achieve the fastest and simplest path from a Python model script to a deployable, containerized service with a focus on developer experience, BentoML provides a highly streamlined and user-friendly workflow.44

The following table provides a structured summary of these comparisons, enabling a data-driven, trade-off-aware decision based on an organization’s specific context and priorities.

Feature	NVIDIA Triton	KServe (with ModelMesh)	Seldon Core (with MLServer)	Ray Serve	BentoML
Architecture Paradigm	High-Performance Inference Server	Kubernetes-Native Orchestration	Kubernetes-Native MLOps Framework	Python-Native Distributed Computing	Python-Native Application Framework
Primary Abstraction	Model Repository & config.pbtxt	InferenceService CRD	SeldonDeployment CRD (Inference Graph)	Deployment (Ray Actor)	Service & Runner
Multi-Model Patterns	Ensembles, Co-location	Co-location, InferenceGraph, ModelMesh (High-Density)	Co-location, InferenceGraph, Experiments (A/B, Canary)	Composition, Multi-Application, Multiplexing	Distributed Services, Composition
Dynamic Loading/Caching	Yes (Dynamic Loading)	ModelMesh (Distributed LRU Cache)	Yes (via MLServer, LRU Cache)	Yes (Multiplexing, LRU Cache)	Manual Implementation
Overcommit Support	No (Manual Management)	Yes (via ModelMesh)	Yes	No (Manual Management)	No
Request Batching	Dynamic (Highly Configurable)	Yes (via Runtimes like Triton)	Yes (via MLServer/Triton)	Dynamic (Decorator-based)	Adaptive (Decorator-based)
Concurrency Model	Concurrent Model Execution	Managed by Kubernetes/Knative	Managed by Kubernetes	Ray Actor Concurrency	Python Process/Thread Workers
Framework Compatibility	Excellent (Broadest Backend Support)	Very Good (Pluggable Runtimes)	Very Good (Pluggable Runtimes)	Excellent (Any Python Code)	Very Good (All Major Frameworks)
Deployment Environment	Docker, Kubernetes, Standalone	Kubernetes	Kubernetes	Ray Cluster (Anywhere), Kubernetes	Docker, Kubernetes, Serverless
Developer Experience	Configuration-Heavy (YAML/pbtxt)	Declarative (YAML)	Declarative (YAML)	Programmatic (Python API)	Programmatic (Python API)
Scalability Model	Replica-based	Serverless (Knative) or Pod Pool (ModelMesh)	Replica-based	Replica-based (Actors)	Replica-based
Advanced MLOps Features	Ensembling, BLS	Canary, Scale-to-Zero, Explainability	A/B, Canary, Shadow, Explainability, Outlier Detection	Model Composition	–
Ideal Use Case	High-throughput, low-latency GPU inference	Serverless & high-density serving on Kubernetes	Enterprise MLOps with advanced deployment strategies	Complex, programmable pipelines in Python	Rapid development and packaging of ML services

 

Multi-Model Serving in the Cloud: Managed Services

 

The major cloud providers offer managed services that simplify the deployment and operational management of machine learning models, including solutions for multi-model serving. These platforms abstract away much of the underlying infrastructure complexity, but they each embody different architectural philosophies and are suited for different use cases.

 

Amazon SageMaker Multi-Model Endpoints (MME)

 

Amazon SageMaker Multi-Model Endpoints are specifically designed for the use case of hosting a large number of models in a cost-effective manner.12 This service is architected for high-density co-hosting, making it ideal for scenarios with large model catalogs where individual models may have sparse or unpredictable traffic.

• Architecture: An MME uses a single endpoint backed by a fleet of instances running a shared serving container. This container runs a model server, such as AWS’s open-source Multi Model Server (MMS), which is responsible for the dynamic lifecycle management of the models.12 Models are not pre-loaded; instead, they are dynamically loaded from Amazon S3 into the container’s memory on the first invocation for that model. When an instance’s memory comes under pressure, the server automatically unloads infrequently used models to make space for new ones.12
• Key Features and Usage: To invoke a specific model, the client passes the model’s filename (e.g., model.tar.gz) in the TargetModel parameter of the invoke_endpoint API call.11 This architecture is highly cost-effective because you only pay for the compute instances, which are shared across all models. MMEs also support GPUs, allowing for the efficient hosting of deep learning models.75
• Limitations: The primary trade-off with MMEs is performance predictability. The on-demand loading of models introduces a “cold start” latency penalty for the first request to an unloaded model.11 For this reason, SageMaker MMEs are best suited for applications that can tolerate occasional latency spikes and for hosting homogeneous models that can run within the same container environment.12

 

Google Cloud Vertex AI Endpoints

 

Google Cloud’s Vertex AI provides a unified platform for MLOps, and its endpoint service allows for the deployment of multiple models to a single endpoint. However, its architectural approach differs significantly from SageMaker’s MME and is primarily oriented toward traffic management and safe deployments rather than high-density co-hosting.

• Architecture: When you deploy multiple models to a single Vertex AI endpoint, you typically associate dedicated compute resources (a set of virtual machines) with each model.76 The endpoint then acts as a unified gateway that can route incoming traffic to these different deployed models based on a configured traffic split.77 While it is possible to co-host multiple models on the same VM to improve resource utilization, the primary mechanism exposed to the user is traffic splitting.80
• Key Features and Usage: The main use case for deploying multiple models to a Vertex AI endpoint is to perform A/B testing or to execute a canary rollout of a new model version. For example, you can deploy a new model v2 alongside an existing model v1, initially directing 10% of the traffic to v2 and 90% to v1. You can then monitor the new model’s performance and gradually increase its traffic share until it handles 100% of requests, at which point the old model can be undeployed.77
• Limitations: This model is less focused on the cost-saving benefits of dynamic loading and unloading for large, sparse model catalogs. It is more of a production-grade traffic management tool for ensuring the safe and reliable updating of models.

 

Microsoft Azure Machine Learning with Triton Integration

 

Microsoft Azure Machine Learning offers managed online endpoints as a scalable solution for real-time model inference. A key aspect of its strategy for high-performance and multi-model serving is its first-class, streamlined integration with NVIDIA Triton Inference Server.

• Architecture: Azure ML provides a “no-code deployment” option for models that are packaged in the specific repository format required by Triton.45 When deploying such a model, Azure provisions the underlying compute resources and deploys a managed instance of the Triton server, configuring it to serve the specified model(s).
• Key Features and Usage: This approach allows users to leverage the full power and extensive feature set of Triton—including its multi-framework backend support, high-performance concurrent execution, and dynamic request batching—within a fully managed cloud environment. This significantly reduces the operational overhead of manually setting up, configuring, and maintaining a Triton server, while still providing access to its state-of-the-art performance capabilities.45 It represents a strategy of providing a managed, enterprise-grade wrapper around a best-in-class open-source technology.

The differing approaches of the major cloud providers highlight that “multi-model serving” is not a monolithic concept. AWS SageMaker MME is architected for high-density co-hosting to maximize cost-efficiency for large model catalogs. Google Cloud Vertex AI’s multi-model feature is architected for robust traffic management to enable safe deployment practices like A/B testing. Microsoft Azure ML’s strategy is one of managed integration, providing an easy on-ramp to a powerful open-source inference server. This underscores a broader trend of convergence, where cloud providers are increasingly building their managed services on top of powerful open-source foundations, offering users a balance of managed convenience and open-source flexibility.

 

Advanced Challenges and Strategic Recommendations

 

While multi-model serving offers a powerful solution for scaling ML deployments, its implementation in complex, real-world environments presents a unique set of advanced challenges. Addressing these challenges requires sophisticated strategies that go beyond basic co-hosting and caching, pushing towards more intelligent, adaptive, and workload-aware systems.

 

Taming Heterogeneity: Strategies for Mixed-Framework and Mixed-Size Workloads

 

One of the most significant challenges in production MMS is managing a heterogeneous portfolio of models. Real-world systems must serve models built with different ML frameworks (e.g., PyTorch, TensorFlow, XGBoost), with vastly different resource requirements (e.g., small CPU-bound models vs. large GPU-hungry LLMs), and varying performance characteristics.7 A naive, one-size-fits-all shared server is often suboptimal for this diversity.

Effective strategies for handling heterogeneity include:

• Multi-Framework Inference Servers: Employing versatile servers like NVIDIA Triton, which can load and run models from multiple frameworks simultaneously using its pluggable backend system, is a foundational step. This allows for the consolidation of different model types within a single server process, improving resource density.41
• Decoupled and Disaggregated Architectures: For particularly complex and heterogeneous models, such as Large Multimodal Models (LMMs), a more advanced approach is to disaggregate the model’s architecture into independently deployable and scalable components. For example, the image encoding stage (which is compute-intensive) and the text decoding stage (which is memory-intensive) of an LMM can be served as separate microservices. This decoupled architecture allows for fine-grained resource allocation, where each component can be scaled on the most appropriate hardware, leading to greater overall efficiency.7
• Spatio-Temporal GPU Sharing: To efficiently co-locate multiple heterogeneous models on a single GPU, advanced scheduling techniques are required. Spatio-temporal sharing involves both spatially partitioning the GPU’s resources (e.g., assigning specific streaming multiprocessors or memory blocks to different models) and temporally sharing them (e.g., time-slicing execution). This allows the scheduler to create virtual GPUs, or “gpulets,” tailored to the specific needs of each model, thereby maximizing the utilization of the underlying hardware while meeting performance SLOs.8

 

From Reactive to Proactive: Predictive Resource Allocation and Autoscaling

 

Traditional resource management in serving systems is largely reactive. Autoscalers add or remove replicas based on current metrics like CPU utilization or request queue length, and caching policies like LRU evict models based on past usage. The next frontier in MMS optimization is to move from this reactive posture to a proactive, predictive one, using machine learning to anticipate future needs.

• Predictive Resource Allocation: Instead of waiting for a resource bottleneck to occur, ML models can be trained on historical workload data to predict the resource requirements (CPU, memory, GPU) of incoming inference tasks in advance. These predictions enable a more intelligent scheduler to proactively allocate the optimal amount of resources, improving efficiency and preventing failures from misconfiguration.6
• Intelligent and Adaptive Caching: Simple LRU caching can be suboptimal. A more intelligent system can use ML to predict which models are likely to be requested in the near future based on temporal patterns (time of day), user behavior, or other contextual signals. This enables predictive loading, where the system pre-warms its cache by loading models just before they are needed, effectively eliminating cold-start latency for anticipated requests.87 Systems like Chameleon take this further with adaptive adapter caching and adapter-aware scheduling for LLMs, intelligently managing GPU memory to reduce latency and increase throughput.89

 

Insights from the Field: Lessons from Large-Scale Production Systems

 

Examining how large technology companies have tackled the “many models” problem provides invaluable practical insights that complement the theoretical capabilities of serving frameworks.

• Strategic Model Consolidation at Netflix: In their large-scale recommendation system, Netflix faced the challenge of managing hundreds of specialized models. Their solution was not just infrastructural but also strategic. They consolidated many of these bespoke models into a single, powerful multi-task learning model. This approach dramatically reduced technical debt, simplified maintenance, and accelerated the propagation of modeling improvements across different use cases.90 This unified model was then deployed in various serving environments, each with finely tuned parameters for latency, caching, and parallelism to meet the specific needs of different product surfaces. This case study highlights a critical lesson: before optimizing the infrastructure to serve thousands of models, organizations should first question whether a more sophisticated, unified modeling approach can reduce the number of models needed in the first place.
• The Platform Approach at Uber and Others: Companies like Uber (with its Michelangelo platform), Airbnb, and Canva have demonstrated the necessity of building a centralized MLOps platform to manage model deployment at a massive scale.91 These platforms provide a standardized, automated “paved road” for data scientists to deploy, monitor, and manage their models. They abstract away the complexities of the underlying serving infrastructure (which may itself be a multi-model system) and provide a consistent workflow, enabling rapid and reliable deployment of thousands of models making millions or billions of predictions per second.91

 

Strategic Recommendations for Implementation and Future-Proofing

 

For organizations embarking on the journey to scalable multi-model serving, the following strategic recommendations can help guide the implementation and ensure the architecture is robust and future-proof.

1. Standardize Interfaces and Formats: Adopt industry standards wherever possible. Standardizing on an inference protocol like the KServe V2 Open Inference Protocol ensures interoperability between different serving components. Similarly, using a standard model interchange format like ONNX can decouple your models from specific training frameworks, providing greater flexibility to leverage different inference runtimes.
2. Embrace a Layered, Decoupled Architecture: Avoid monolithic solutions. A best-practice architecture separates the orchestration/control plane (e.g., KServe, Seldon Core) from the high-performance inference/data plane (e.g., NVIDIA Triton). This layered approach provides flexibility, allowing you to select the best tool for each specific function and to evolve parts of the stack independently.
3. Prioritize Comprehensive Monitoring: Implement robust, end-to-end monitoring from the outset. Track key performance indicators (KPIs) such as end-to-end latency (P90, P99), throughput, hardware utilization (especially GPU memory and compute), and cache hit/miss rates. This data is invaluable for debugging, performance tuning, and making informed decisions about scaling policies and caching strategies.
4. Design for Heterogeneity: Do not assume your model portfolio will remain homogeneous. Architect your serving platform with the explicit assumption that you will need to support a mix of model frameworks, sizes, and hardware requirements. This may involve creating separate, specialized serving pools (e.g., a CPU-only pool for classical models, a high-VRAM GPU pool for LLMs) managed under a unified control plane.

The trajectory of MMS is clearly moving toward more intelligent, workload-aware systems. The disaggregation of models into independently served components, as seen in LMM and Dynamo research, suggests that the future of MMS will involve managing not just multiple models, but the distributed components of single, massive models. This evolution makes intelligent scheduling, routing, and predictive resource allocation even more critical for building the next generation of efficient and performant AI systems.

 

Conclusion

 

The transition from deploying individual machine learning models in isolated containers to hosting a multitude of models on shared, efficient infrastructure represents a fundamental and necessary evolution in the field of MLOps. Multi-model serving is no longer a niche optimization but a core competency for any organization seeking to scale its AI/ML capabilities in a cost-effective and operationally sustainable manner. The single-model paradigm, with its attendant resource over-provisioning, infrastructure overhead, and high latency, is operationally untenable for managing the large, heterogeneous model portfolios that modern AI applications demand.

This report has detailed the architectural patterns and optimization techniques that form the foundation of modern MMS, including dynamic model management with LRU caching, the “overcommit” strategy, model multiplexing, and complex inference graphs. A deep analysis of the leading open-source frameworks reveals a clear bifurcation in philosophy: Kubernetes-native platforms like KServe and Seldon Core offer declarative, infrastructure-centric control, while Python-native frameworks like Ray Serve and BentoML provide a programmable, developer-centric approach to building distributed serving applications. High-performance inference servers, most notably NVIDIA Triton, have become the de facto data plane, providing the raw computational power that these orchestration layers manage.

The key challenges ahead lie in taming heterogeneity and moving from reactive to proactive resource management. The future of MMS points toward increasingly intelligent systems that use machine learning to predict workloads, pre-warm caches, and dynamically allocate partitioned hardware resources. Furthermore, as models themselves become disaggregated into servable components, the role of the serving platform will evolve from a model host to a sophisticated orchestrator of distributed computational graphs.

For practitioners and architects, the path forward involves making strategic choices based on a clear understanding of these architectural trade-offs. The decision to adopt a specific framework or cloud service should be guided by the organization’s existing infrastructure, team skill sets, performance SLOs, and MLOps maturity. By embracing standardization, adopting a layered architecture, and prioritizing robust monitoring, organizations can build a future-proof serving platform capable of efficiently delivering AI-powered features at scale.