Apache Iceberg - Under the Hood
In this video, Dipankar breaks down how Apache Iceberg works under the hood - starting from the limitations of Hive-style tables to why Iceberg was built in the first place. He covers:
✅ Why Hive-based tables break at scale (Netflix example)
✅ How object storage changes the problem (S3 behavior, listing, throttling)
✅ Iceberg architecture (catalog, metadata, snapshots, manifests, data files)
✅ How query planning works step by step
✅ Why Iceberg is a specification — not an execution engine
Join the Cloudera Community to learn more! 👉https://community.cloudera.com
Explore the Full Series: 👉 https://www.youtube.com/playlist
- Get Started*
- 🔹 Watch Demos: https://www.cloudera.com/products/cloudera-data-platform/cdp-demos.html
- 🔹 Customer Success Stories: https://www.cloudera.com/customers.html
- 🔹 Read the Cloudera blog: https://www.cloudera.com/blog.html
- Resources:*
- Full Playlist: https://www.youtube.com/playlist
- Hear more from our customers: https://www.youtube.com/playlist
- Watch Demos: https://www.youtube.com/playlist
Chapters:
00:00 Introduction: Cloudera Developers & Learning Journey
00:45 What to Expect: Deep Dive into Apache Iceberg Internals
03:30 The Problem: Scaling Challenges & Expensive Updates
05:20 The Netflix Origin Story: Why Iceberg was Born
11:15 From Directories to Metadata: The Big Architectural Shift
13:10 Inside the Iceberg Architecture: The Catalog
15:35 Demo: Inspecting a Metadata File
19:30 Metadata Components
24:10 Summary: Turning Object Stores into Databases
- Connect with Cloudera*
- Subscribe to stay ahead of the curve with the latest in data strategy, open architectures, and enterprise AI innovations. https://www.cloudera.com
- LinkedIn ► https://www.linkedin.com/company/cloudera
- Facebook ► https://www.facebook.com/cloudera/
- X ► https://x.com/cloudera
- Spotify ► https://open.spotify.com/show/102S8zoZR6nmZV0HxZlxZu
#ApacheIceberg #DataEngineering #CloudComputing #OpenLakehouse #Cloudera #DataLakehouse #DataArchitecture #SoftwareEngineering #OpenSource #ApacheHive