DuckLake: SQL as a Lakehouse Format – DuckDB
TL;DR: DuckLake simplifies lakehouses by using a standard SQL database for all metadata, instead of complex file-based systems, while still storing data in open formats like Parquet. This makes it more reliable, faster, and easier to manage. The first part of the blog post is shared with the DuckLake manifesto. Jump to the DuckLake extension section to read the rest. Innovative data systems like BigQuery and Snowflake have shown that disconnecting storage and compute is a great idea in a time where storage is a virtualized commodity. That way, both storage and compute can scale independently and we don't have to buy expensive database machines just to store tables we will never read. At the same time, market forces have pushed people to insist that data systems use open formats like Parquet to avoid the all-too-common hostage taking of data by a single vendor. In this new world, lots of data systems happily frolic around a pristine “data lake” built on Parquet and S3 and all was well.
Mark Raasveldt and Hannes Mühleisen 2025-05-27 | 20 min TL;DR: DuckLake simplifies lakehouses by using a standard SQL database for all metadata, instead of complex file-based systems, while still storing data in open formats like Parquet. This makes it more reliable, faster, and easier to manage. The first part of the blog post is shared with the DuckLake manifesto. Jump to the DuckLake extension section to read the rest. Background Innovative data systems like BigQuery and Snowflake have shown that disconnecting storage and compute is a great idea in a time where storage is a…
related reading
- What Is DuckDB? Introduction, Use Cases & Architecture | DuckDB in Actionmotherduck.com
- From monolith to Lakebase to LTAP: rethinking the database from storage up | Databricks Blogdatabricks.com
- DuckDB goes distributed? DeepSeek’s smallpond takes on Big Datamehdio.substack.com
- Querying Postgres Tables Directly from DuckDB – DuckDBduckdb.org
- Building data-centric apps with a reactive relational databaseriffle.systems
- Separating Storage and Compute in DuckDBmotherduck.com
- Databricks + Tabular | Databricks Blogdatabricks.com
- Introducing Husky, Datadog’s third-generation event store | Datadogdatadoghq.com
- Databricks: Leading Data and AI Solutions for Enterprisesmooncake.dev
- Databases in 2024: A Year in Review // Blog // Andy Pavlo - Carnegie Mellon Universitycs.cmu.edu
- The Beginner's Guide to Databasestechnically.substack.com
- Security data lakehouse and modular designrippling.com