jakub-zovi
Graphkasten
Blog
About

Search

Search IconIcon to open search

Apache Iceberg

Last updatedUpdated: by Jakub Žovák · 3 min read

Properties
tags cscs/data_eng
created 18.03.2026, 10:00
modified 26.07.2026, 13:19
published Empty
sources Apache Iceberg Docs · Apache Iceberg: An Architectural Look Under the Covers · Delta Lake vs Apache Iceberg
topics Table Format, Open Table Format, Data Lakehouse
authors Empty
ai-assisted Yes
  • Apache Iceberg is an open table format for huge analytic datasets. Originally developed at Netflix to solve large-scale data correctness and performance problems, it became a top-level Apache project in 2020.

Iceberg is a high-performance format for huge analytic tables. Iceberg brings the reliability and simplicity of SQL tables to big data, while making it possible for engines like Spark, Trino, Flink, Presto, Hive and Impala to safely work with the same tables, at the same time. ( Apache Iceberg Docs)

# Key Features

  • ACID transactions — serializable isolation with optimistic concurrency
  • Schema evolution — add, drop, rename, reorder columns without rewrites; type promotion supported
  • Hidden partitioning — partition transforms applied automatically; queries don’t require partition filter knowledge
  • Time travel — query any historical snapshot by timestamp or snapshot ID
  • Row-level operations — copy-on-write (CoW) and merge-on-read (MoR) strategies for deletes/updates/upserts
  • Multi-engine support — Spark, Flink, Trino, Presto, Hive, Impala, Dremio, StarRocks, and more
  • Partition evolution — change partitioning strategy without rewriting existing data

# Metadata Structure

Iceberg uses a layered metadata hierarchy to track all data files and changes:

1
2
3
4
5
6

Catalog
└── Table Metadata (JSON)
    └── Snapshot
        └── Manifest List (Avro)
            └── Manifest Files (Avro)
                └── Data Files (Parquet / ORC / Avro)
  • Table Metadata file (JSON) — current snapshot pointer, schema, partition spec, sort order
  • Snapshot — immutable point-in-time state of the table; each write creates a new snapshot
  • Manifest list — lists all manifest files for a snapshot with partition-level stats (enables partition pruning)
  • Manifest file — lists data files with column-level stats (enables column/row group pruning)
  • Data files — actual columnar data in Parquet, ORC, or Avro

This design means metadata operations (schema changes, time travel) are just pointer swaps — no data rewriting needed.

# vs Delta Lake

AspectApache IcebergDelta Lake
OriginNetflix → Apache (2020)Databricks (2019), open-sourced
Engine supportSpark, Flink, Trino, Presto, Hive, Impala, Dremio, …Spark-first; growing (Flink, Trino connectors exist)
Hidden partitioningYes — transforms applied at write; queries engine-agnosticNo — partition columns must be explicit in queries
Partition evolutionYes — change spec without rewriting old dataNo — requires full rewrite to repartition
Catalog flexibilityAny catalog (REST, Glue, Nessie, HMS)Tightly coupled to Databricks Unity Catalog or HMS
Metadata formatJSON (table metadata) + Avro (manifests)JSON transaction log (_delta_log/)
Row-level DML strategyCoW or MoR (configurable per table)CoW default; MoR (deletion vectors) added later
Ecosystem tractionCloud-agnostic; strong in AWS/GCP ecosystemsDominant in Databricks/Azure environments

# vs Delta Table

Delta Table is a Databricks-specific abstraction on top of Delta Lake — it is the managed/unmanaged table object registered in the Databricks catalog. Iceberg has no equivalent separation: an Iceberg table is a first-class object defined by its metadata files, not by a platform-specific catalog entry. This makes Iceberg inherently more portable across clouds and engines.

Backlinks