JA EN
shiichan

Query Iceberg and Parquet directly from Aurora PostgreSQL, no ETL!

AWS Blog ★★★★★ View Markdown
#aws#infrastructure#devops

Hi everyone, it's Shii-chan! Aurora can now read your data lake directly, and it's a big one.

AWS Blog aws.amazon.com

What was announced?

The AWS Blog announced that Amazon Aurora PostgreSQL can now directly query Apache Iceberg and Apache Parquet data in your data lake alongside your operational data. No ETL pipelines required!

The story so far

If you wanted to combine transactional data with historical data in the data lake in a single query, you had to build a pipeline to move the data. This feature removes that work.

What changes

Thanks to DuckDB embedded in Aurora, you can write a single query that joins transactional and historical data using familiar PostgreSQL syntax.

Dive Deep

How it works and what it supports:

  • A DuckDB engine embedded in Aurora PostgreSQL
  • Direct access to Iceberg and Parquet data in S3 and the AWS Glue Data Catalog, plus S3 Tables and Iceberg REST Catalog compatible catalogs
  • Predicate pushdown and column pruning optimizations
  • Caching that speeds up frequently accessed data

To get started:

  • Create an Aurora PostgreSQL cluster (version 17.11 or later, or 18.6 or later)
  • Enable the aurora_analytics extension
  • Grant the AuroraAnalytics permission to your IAM role
  • Create foreign tables that point to the S3 or Glue data

A SQL example:

CREATE EXTENSION aurora_analytics;
CREATE FOREIGN TABLE transaction_history ()
  SERVER aurora_analytics_server
  OPTIONS (location 's3://bucket/data.parquet', format 'parquet');

There is no additional charge; you pay for Aurora compute and S3 requests. It is available in all commercial AWS Regions and AWS GovCloud (US). For data that needs single-digit millisecond latency, the post recommends materializing it.

Wrap-up

  • Query Iceberg and Parquet directly from Aurora PostgreSQL
  • Built on embedded DuckDB, with Glue Data Catalog, S3, and S3 Tables support
  • Enable the aurora_analytics extension on 17.11+ or 18.6+
  • No additional charge, available in all commercial Regions and GovCloud (US)

This is for anyone who wants to analyze across operational databases and the data lake!

Related posts