{"id":9503,"date":"2026-01-28T10:54:57","date_gmt":"2026-01-28T10:54:57","guid":{"rendered":"https:\/\/uplatz.com\/blog\/?p=9503"},"modified":"2026-01-28T10:54:57","modified_gmt":"2026-01-28T10:54:57","slug":"compaction-strategies-and-the-small-file-problem-in-object-storage-a-comprehensive-analysis-of-query-performance-optimization","status":"publish","type":"post","link":"https:\/\/uplatz.com\/blog\/compaction-strategies-and-the-small-file-problem-in-object-storage-a-comprehensive-analysis-of-query-performance-optimization\/","title":{"rendered":"Compaction Strategies and the Small File Problem in Object Storage: A Comprehensive Analysis of Query Performance Optimization"},"content":{"rendered":"

The modern data architecture has undergone a fundamental shift from local, block-based distributed file systems to cloud-native object storage. While this transition has enabled unprecedented scalability and decoupling of compute from storage, it has also amplified a structural inefficiency known as the small file problem. In distributed storage systems, particularly those supporting big data frameworks like Apache Spark, Trino, and Presto, a small file is typically defined as any file significantly smaller than the default block size, often 128 MB or 256 MB.<\/span>1<\/span> The proliferation of these files\u2014driven by high-frequency streaming ingestion, IoT sensor telemetry, and over-partitioned directory structures\u2014introduces severe performance degradation across the entire data processing lifecycle.<\/span>3<\/span> As analytics engines struggle to manage millions of discrete objects, compaction strategies have emerged as the primary mechanism for restructuring data to optimize metadata management, I\/O throughput, and query planning efficiency.<\/span><\/p>\n

The Genesis and Mechanics of the Small File Problem<\/b><\/h2>\n

The emergence of the small file problem is rarely a result of intentional design but is rather an emergent property of contemporary data ingestion patterns. Many enterprises deal with streaming sources such as networking equipment, application logs, and server event streams that generate thousands of event logs per second.<\/span>3<\/span> To achieve near-real-time data freshness, ingestion pipelines frequently commit these records to object storage in tiny increments, often as separate JSON, XML, or CSV files.<\/span>1<\/span> Over time, this results in a fragmented storage layout where the logical data volume is dwarfed by the administrative overhead of managing its physical manifestation.<\/span><\/p>\n

Metadata Inflation and System Bottlenecks<\/b><\/h3>\n

In traditional Hadoop Distributed File System (HDFS) environments, the small file problem was primarily a memory constraint on the NameNode. HDFS tracks every file, directory, and block as a metadata record in the NameNode’s memory, with each record occupying between 150 and 300 bytes.<\/span>1<\/span> A hundred million small files can consume hundreds of gigabytes of memory, eventually exhausting the NameNode\u2019s capacity and forcing architectural workarounds like federation.<\/span>3<\/span> Furthermore, because HDFS is optimized for large contiguous files, storing a 16 KB file consumes the same metadata resources as a 128 MB file, leading to massive inefficiencies in memory and storage utilization.<\/span>3<\/span><\/p>\n

Cloud object storage providers like Amazon S3, Azure Blob Storage, and Google Cloud Storage (GCS) decouple metadata from a central node, yet the performance penalties remain significant. The efficiency of a storage system is largely measured by IOPS (Input\/Output Operations Per Second), which includes seek time, read time, and data transmission time.<\/span>3<\/span> For mechanical media and even modern SSDs, sequential reads of large files are substantially faster than the random reads required to access thousands of small files.<\/span>3<\/span> Every file access involves a distinct metadata request to the storage provider\u2019s API, incurring latency for connection establishment, authentication, and metadata retrieval.<\/span>2<\/span><\/p>\n

The Computational Cost of Fragmentation<\/b><\/h3>\n

Distributed query engines are designed to parallelize workloads by splitting data into tasks. When a query scans a table composed of millions of small files, the engine must spawn an equivalent number of tasks, as each file often corresponds to at least one split.<\/span>2<\/span> This creates a massive scheduling overhead where the time spent on task initialization, metadata fetching, and context switching exceeds the time spent on actual computation.<\/span>1<\/span><\/p>\n\n\n\n\n\n\n\n
System Component<\/b><\/td>\nImpact of Small Files<\/b><\/td>\nImpact of Compacted Files<\/b><\/td>\n<\/tr>\n
Metadata Store<\/span><\/td>\nMemory exhaustion (HDFS NameNode)<\/span><\/td>\nLow overhead, efficient caching<\/span><\/td>\n<\/tr>\n
Network I\/O<\/span><\/td>\nHigh latency from thousands of API calls<\/span><\/td>\nHigh throughput from sequential reads<\/span><\/td>\n<\/tr>\n
Task Scheduling<\/span><\/td>\nExcessive task creation and imbalanced load<\/span><\/td>\nOptimized parallel execution<\/span><\/td>\n<\/tr>\n
Cost Efficiency<\/span><\/td>\nHigh API request costs and billable minimums<\/span><\/td>\nOptimized storage and reduced request fees<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Data derived from architectural comparisons of HDFS and cloud object storage performance.<\/span>1<\/span><\/p>\n

Architectural Constraints and Throttling in Cloud Environments<\/b><\/h2>\n

Cloud providers implement strict rate limits to maintain service stability across multi-tenant environments. In Amazon S3, request rates are typically limited per prefix, and high volumes of small-file reads can trigger 503 “Slow Down” errors.<\/span>9<\/span> These throttling events force engines to implement exponential backoff, which compounds query latency. Similarly, Azure Storage accounts may experience throttling at the account level for standard tiers or the share level for premium tiers if I\/O operations per second (IOPS) or ingress\/egress limits are exceeded.<\/span>11<\/span><\/p>\n

The Economic Implications of Small Files<\/b><\/h3>\n

The financial burden of small files is twofold: request costs and storage minimums. Cloud providers charge per 1,000 requests, and a query engine reading 100,000 tiny files will incur far higher API costs than one reading 100 large files.<\/span>8<\/span> Furthermore, storage tiers such as Amazon S3 Standard-Infrequent Access and Azure Blob Cool Tier often impose a minimum billable object size, typically 128 KB.<\/span>13<\/span> If an organization stores millions of 10 KB files, they are billed for 128 KB per file, essentially paying for “ghost” data that does not exist.<\/span>13<\/span><\/p>\n\n\n\n\n\n\n\n
Storage Tier \/ Feature<\/b><\/td>\nMinimum Billable Size<\/b><\/td>\nSmall File Impact<\/b><\/td>\n<\/tr>\n
S3 Standard-IA<\/span><\/td>\n128 KB<\/span><\/td>\nSignificant cost increase for < 128 KB objects<\/span><\/td>\n<\/tr>\n
S3 Glacier Instant Retrieval<\/span><\/td>\n128 KB<\/span><\/td>\nHigh overhead for small log archives<\/span><\/td>\n<\/tr>\n
S3 Intelligent-Tiering<\/span><\/td>\nNo minimum for storage<\/span><\/td>\nObjects < 128 KB always charged at Frequent Access rate<\/span><\/td>\n<\/tr>\n
Azure Blob Cool Tier<\/span><\/td>\nTypically 64 KB – 128 KB<\/span><\/td>\nHigher cost per actual byte stored<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Economic analysis based on cloud provider billing models for tiered storage.<\/span>13<\/span><\/p>\n

Fundamental Compaction Strategies and Algorithms<\/b><\/h2>\n

Compaction is the process of merging fragmented data files into a more efficient, ordered layout. This process is essential for maintaining read performance and reclaiming space in systems based on the Log-Structured Merge (LSM) tree architecture.<\/span>15<\/span> LSM-based systems buffer writes in memory (MemTables) and flush them to disk as Sorted String Tables (SSTables). As these files accumulate, compaction merges overlapping sorted runs to bound read amplification and remove deleted or superseded data.<\/span>15<\/span><\/p>\n

Size-Tiered vs. Leveled Compaction<\/b><\/h3>\n

The two primary historical compaction strategies are Size-Tiered Compaction Strategy (STCS) and Leveled Compaction Strategy (LCS), each representing a different point on the trade-off curve between write amplification, read amplification, and space amplification.<\/span>16<\/span><\/p>\n

    \n
  1. Size-Tiered Compaction (STCS):<\/b> This strategy groups SSTables into “tiers” based on their size. When a tier reaches a certain number of files (e.g., four), they are merged into a single larger file in the next tier.<\/span>16<\/span> STCS is optimized for write-heavy workloads because it copy-merges data fewer times than leveled strategies, resulting in lower write amplification. However, it suffers from high read amplification as query engines must search across multiple SSTables to find the latest version of a record.<\/span>16<\/span><\/li>\n
  2. Leveled Compaction (LCS):<\/b> This strategy organizes data into levels of exponentially increasing capacity. Each level (except L0) consists of non-overlapping files that cover the entire keyspace.<\/span>17<\/span> When a level reaches its capacity, its files are merged into the overlapping files of the next level. LCS significantly reduces read amplification by ensuring that most reads only need to check one file per level. This comes at the cost of high write amplification, as the same data is rewritten multiple times during the leveling process.<\/span>17<\/span><\/li>\n<\/ol>\n

    The Unified Compaction Strategy (UCS)<\/b><\/h3>\n

    The Unified Compaction Strategy (UCS) generalizes both tiered and leveled compaction by using a density-based sharding mechanism.<\/span>18<\/span> Density is defined as the size of an SSTable divided by the width of the token range it covers. UCS uses a scaling parameter <\/span> to determine its behavior: positive values of <\/span> simulate tiered compaction (low write cost, high read cost), while negative values simulate leveled compaction (high write cost, high read efficiency).<\/span>18<\/span><\/p>\n

    In the UCS framework, tiered compaction is triggered when <\/span>, while leveled compaction is represented by <\/span>. This mathematical unification allows for a stateless compaction process that can be parallelized across different shards of the keyspace, maximizing throughput on high-density storage nodes.<\/span>18<\/span><\/p>\n

    Advanced Maintenance in Open Table Formats<\/b><\/h2>\n

    The emergence of open table formats\u2014Apache Iceberg, Delta Lake, and Apache Hudi\u2014has revolutionized how compaction is managed in data lakes. Unlike traditional Hive tables, where compaction required manual ETL jobs and often resulted in “dirty reads” during the process, these formats use a metadata layer to provide ACID transactions and snapshot isolation.<\/span>20<\/span><\/p>\n

    Apache Iceberg: Metadata-Centric Optimization<\/b><\/h3>\n

    Iceberg manages table state through a hierarchy of metadata files, manifest lists, and manifest files.<\/span>23<\/span> This architecture allows Iceberg to perform “hidden partitioning,” where the engine identifies relevant files without relying on the physical directory structure.<\/span>20<\/span><\/p>\n

    The rewriteDataFiles Action<\/b><\/h4>\n

    Iceberg provides a powerful maintenance utility called rewriteDataFiles that supports three distinct compaction strategies:<\/span><\/p>\n