JA EN
shiichan

Query Iceberg and Parquet Straight From Aurora PostgreSQL, No ETL, Powered by DuckDB!

AWS What's New ★★★★★ View Markdown
#aws#infrastructure

Hi everyone, it's Shii-chan!

AWS What's New aws.amazon.com

Now you can read data lake data straight from your usual PostgreSQL!

What was announced?

According to AWS What's New, you can now directly query operational data together with data lake data in Apache Iceberg and Parquet formats from Aurora PostgreSQL.

You keep using your existing PostgreSQL applications and tools, with no ETL pipelines or data duplication!

The story so far

Accessing data lake data has typically required pipelines that copy it into Aurora. That drove up costs and engineering work as schemas evolved.

What changes

  • Create PostgreSQL foreign tables that reference Iceberg or Parquet data in Amazon S3, Amazon S3 Tables, or AWS Glue Data Catalog
  • Existing applications and BI tools keep using the same PostgreSQL interface
  • Query tables from external Iceberg REST Catalog (IRC)-compatible catalogs federated through AWS Glue Data Catalog, without moving or duplicating data

Dive Deep

  • When you query these foreign tables, Aurora uses DuckDB's high-performance query engine, embedded in PostgreSQL, to run the query against the underlying Iceberg and Parquet data
  • For latency-sensitive workloads, you can materialize Iceberg or Parquet data into native Aurora PostgreSQL tables using standard SQL statements, without an ETL pipeline
  • Generally available on Aurora PostgreSQL 17.11, 18.6 and higher
  • Available in AWS commercial and GovCloud (US) Regions

Wrap-up

  • Aurora PostgreSQL queries Iceberg and Parquet directly through foreign tables
  • Works with data in S3, S3 Tables, and Glue Data Catalog
  • Runs on DuckDB, and you can materialize into native tables when needed

This is for app developers and data folks who want to analyze data lake data alongside PostgreSQL!

Related posts