The modern data architecture landscape has undergone a paradigm shift, moving from the rigid schemas of enterprise data warehouses to the scalable but chaotic data swamps of early Hadoop, and finally arriving at the structured, transactional Data Lakehouse. This evolution has been driven by a singular necessity: the requirement to bring database-like guarantees\u2014specifically Atomicity, Consistency, Isolation, and Durability (ACID)\u2014to the scalable, cost-effective storage tier of object stores like Amazon S3, Azure ADLS, and Google Cloud Storage (GCS).<\/span>1<\/span><\/p>\n For nearly a decade, the Hive Metastore (HMS) served as the de facto standard for managing tabular data in lakes. However, HMS was fundamentally limited by its architecture, which tracked data at the folder level rather than the file level. This design choice introduced severe bottlenecks: changing a partition required expensive recursive directory listings, and the lack of atomic commits meant that readers could often see partial writes or inconsistent states. Furthermore, the eventual consistency models of early cloud object stores (e.g., S3 prior to 2020) exacerbated these issues, forcing engineers to rely on auxiliary consistency mechanisms like Netflix\u2019s S3Guard.<\/span>3<\/span><\/p>\n The resolution to these challenges emerged in the form of “Open Table Formats” (OTFs)\u2014specifically Apache Iceberg, Delta Lake, and Apache Hudi. These formats effectively moved the database management system (DBMS) logic out of the engine and into the application layer, interacting directly with files in object storage to guarantee transactional integrity. By 2025, the industry discourse has shifted from a “war” over which format to choose, to a complex engineering challenge: managing <\/span>governance<\/b> across disparate engines (Spark, Trino, Flink) and ensuring <\/span>interoperability<\/b> in a heterogeneous environment where no single format rules every workload.<\/span>5<\/span><\/p>\n This report provides an exhaustive analysis of the technical underpinnings of this new landscape. It dissects the concurrency control mechanisms that allow multiple engines to write to the same table simultaneously, the governance protocols that manage metadata at scale, and the translation layers that are beginning to dissolve the barriers between formats.<\/span><\/p>\n To understand the interoperability challenges inherent in cross-engine workloads, one must first deconstruct the distinct architectural philosophies of the three dominant formats. While they share a common goal\u2014metadata management over Parquet\/Avro files\u2014their internal implementations dictate their specific strengths, weaknesses, and compatibility limits.<\/span><\/p>\n Apache Iceberg was born at Netflix specifically to address the correctness issues of Hive on S3. Its core design philosophy is the complete isolation of table state into immutable snapshots.<\/span>2<\/span><\/p>\n Iceberg utilizes a sophisticated three-tier metadata tree that allows engines to plan queries without listing object storage directories\u2014a crucial optimization for performance on cloud storage.<\/span><\/p>\n A differentiating feature of Iceberg is “Hidden Partitioning.” Unlike Hive, which requires the user to create explicit partition columns (e.g., event_date) derived from the data, Iceberg defines partitions as transforms on existing columns (e.g., day(timestamp)). The metadata tracks the relationship between the raw column and the partition.<\/span><\/p>\n This architecture enables <\/span>Partition Evolution<\/b>: the partitioning scheme can be changed over time (e.g., from month to day) without rewriting old data. The metadata simply tracks which files belong to which partition spec version. This feature, while powerful, complicates interoperability, as translation layers like Apache XTable must map these logical transforms to physical columns in other formats.<\/span>9<\/span><\/p>\n Delta Lake, developed by Databricks, centers its architecture on a sequential transaction log, the _delta_log. This log serves as the single source of truth, providing a verifiable order of operations that facilitates the protocol’s reliance on file system atomic primitives.<\/span>7<\/span><\/p>\n The Delta Log consists of a sequence of JSON files (000000.json, 000001.json, etc.), each representing an atomic transaction.<\/span><\/p>\n Delta Lake manages compatibility through strict protocol versioning (minReaderVersion, minWriterVersion). Advanced features are gated behind these versions:<\/span><\/p>\n Apache Hudi (Hadoop Upsert Deletes and Incrementals) was designed by Uber with a “streaming-first” mindset. It treats the table not just as a state, but as a sequence of events on a timeline.<\/span>1<\/span><\/p>\n The core of Hudi is the .hoodie directory, which maintains a timeline of all actions performed on the table.<\/span><\/p>\n The ability for multiple distributed systems\u2014such as a Flink streaming job and a Spark compaction job\u2014to safely modify the same table concurrently is the “hard problem” of data lake engineering. The solution depends heavily on the consistency guarantees of the underlying storage and the locking mechanisms implemented by the table format.<\/span>12<\/span><\/p>\n While Amazon S3 now offers strong consistency (read-after-write), it does not support atomic rename operations or put-if-absent conditional writes for existing objects. This means that if two writers attempt to commit a transaction simultaneously, one might overwrite the other’s metadata file without realizing it, leading to a “lost update” and data corruption. Consequently, all three formats require an external locking provider or a specific catalog service to arbitrate commits on S3.<\/span>19<\/span><\/p>\n Iceberg employs Optimistic Concurrency Control. Writers assume they are the sole operator, prepare a new metadata file, and then attempt to atomically swap the table pointer to this new file.<\/span><\/p>\n On S3, the “Swap” operation is not atomic. Therefore, the Catalog serves as the synchronizer:<\/span><\/p>\n Delta Lake relies on the concept of a LogStore to abstract the file system specifics. The correctness of Delta relies on the storage system’s ability to fail a write if the file already exists (mutual exclusion).<\/span>25<\/span><\/p>\n Since S3 lacks “put-if-absent,” Open Source Delta Lake (OSS) cannot guarantee safety for concurrent writers from different clusters (e.g., Spark and Flink) out of the box.<\/span><\/p>\n It is important to note that within the Databricks platform, a proprietary commit service manages this concurrency, providing a seamless experience. The complexity of DynamoDB configuration is strictly a concern for open-source users managing their own infrastructure.<\/span>26<\/span><\/p>\n Hudi introduced multi-writer support via OCC in version 0.8.0. Like Iceberg, it separates the data write phase from the commit phase.<\/span>28<\/span><\/p>\n Hudi provides a pluggable locking interface. For S3 deployment, the DynamoDBBasedLockProvider is standard.<\/span><\/p>\n As data lakes scale to petabytes, the “file system as a catalog” model (referencing tables by path) has proven insufficient. The industry has standardized on Catalog Services that provide abstraction, security, and enhanced capabilities like branching.<\/span><\/p>\n The Apache Iceberg REST Catalog Specification is currently the most significant driver of interoperability. It decouples the engine from the catalog implementation, defining a standard OpenAPI contract for table operations.<\/span>23<\/span><\/p>\n Project Nessie extends the catalog concept to include Version Control System (VCS) semantics, enabling patterns like “Zero-Copy Isolation” and cross-table transactions.<\/span>30<\/span><\/p>\n In a standard catalog (e.g., Hive), operations are atomic only at the single-table level. Nessie tracks the state of the entire catalog as a commit hash.<\/span><\/p>\n Nessie exposes Git-like operations via SQL extensions in engines like Trino and Spark. This allows for rigorous DataOps workflows:<\/span><\/p>\n Unity Catalog (originally Databricks-proprietary) has moved towards openness, with its OSS version supporting the Iceberg REST API.<\/span><\/p>\n A primary requirement of the modern lakehouse is the ability to write data with one engine (e.g., Flink) and read it with another (e.g., Trino) regardless of the underlying format. This has led to two distinct approaches: translation layers (XTable) and native masquerading (UniForm).<\/span><\/p>\n Apache XTable serves as an omni-directional translator. It does not rewrite the data files (Parquet) but translates the metadata from one format to another.<\/span>6<\/span><\/p>\n XTable reads the source metadata (e.g., Hudi Timeline) and maps it to the target metadata structures (e.g., Delta Log and Iceberg Manifests).<\/span><\/p>\n UniForm is a uni-directional solution built directly into the Delta Lake writer. When enabled, the writer generates Iceberg metadata asynchronously alongside the Delta log.<\/span>41<\/span><\/p>\n2. The Internal Architecture of Open Table Formats<\/b><\/h2>\n
2.1 Apache Iceberg: The Snapshot Isolation Model<\/b><\/h3>\n
2.1.1 Hierarchical Metadata Structure<\/b><\/h4>\n
\n
2.1.2 Partition Evolution and Hidden Partitioning<\/b><\/h4>\n
2.2 Delta Lake: The Transaction Log Model<\/b><\/h3>\n
2.2.1 The Delta Log Protocol<\/b><\/h4>\n
\n
2.2.2 Protocol Versioning and Feature Flags<\/b><\/h4>\n
\n
2.3 Apache Hudi: The Timeline and Stream-Processing Model<\/b><\/h3>\n
2.3.1 The Timeline Architecture<\/b><\/h4>\n
\n
\n
3. Concurrency Control in Distributed Systems<\/b><\/h2>\n
3.1 The S3 Consistency Challenge<\/b><\/h3>\n
3.2 Apache Iceberg: Optimistic Concurrency Control (OCC)<\/b><\/h3>\n
3.2.1 The Atomic Swap Mechanism<\/b><\/h4>\n
\n
\n
3.2.2 The Role of the Catalog<\/b><\/h4>\n
\n
3.3 Delta Lake: LogStore Semantics and Multi-Cluster Writes<\/b><\/h3>\n
3.3.1 The S3 DynamoDB LogStore<\/b><\/h4>\n
\n
\n
3.3.2 Databricks Proprietary Commit Service<\/b><\/h4>\n
3.4 Apache Hudi: Multi-Writer Locking<\/b><\/h3>\n
3.4.1 Lock Providers<\/b><\/h4>\n
\n
4. Governance and Catalogs: The Shift to Services<\/b><\/h2>\n
4.1 The Iceberg REST Catalog Standard<\/b><\/h3>\n
4.1.1 Mechanism and Benefits<\/b><\/h4>\n
\n
4.2 Project Nessie: Git-Semantics for Data<\/b><\/h3>\n
4.2.1 The Commit Model<\/b><\/h4>\n
\n
4.2.2 Branching and Merging in SQL<\/b><\/h4>\n
\n
\n
4.3 Unity Catalog and Federation<\/b><\/h3>\n
\n
5. Cross-Engine Interoperability: Native vs. Translated<\/b><\/h2>\n
5.1 Apache XTable (formerly OneTable)<\/b><\/h3>\n
5.1.1 The Translation Process<\/b><\/h4>\n
\n
5.1.2 Limitations and Trade-offs<\/b><\/h4>\n
\n
5.2 Delta Lake UniForm (Universal Format)<\/b><\/h3>\n