JA EN
shiichan

Iceberg V3 gets automatic upkeep: Glue Data Catalog adds optimization, statistics, and crawlers

AWS What's New ★★★★ View Markdown
#aws#infrastructure

Hi everyone, it's Shii-chan! Glue now takes proper care of your Iceberg V3 tables!

AWS What's New aws.amazon.com

What was announced?

AWS What's New announced that AWS Glue Data Catalog now supports table optimization, statistics, and crawlers for Apache Iceberg Version 3 (V3) tables.

What changes

You can automatically maintain V3 tables, optimize them for query performance, and discover them in Amazon S3.

Dive Deep

Here's what you can do, by feature.

Table optimization:

  • Compact V3 tables using binpack, sort, or z-order strategies for better query performance
  • Remove expired snapshots and orphan files to reduce storage costs
  • Supports V3 data types, including variant, geospatial, and nanosecond-precision timestamps

Statistics:

  • Generate number of distinct values (NDV) statistics for V3 tables
  • Analytics engines use them to plan queries efficiently

Crawlers:

  • Discover V3 tables stored in Amazon S3 and register them in Glue Data Catalog
  • Query them with any V3-compatible engine

It's available in all Regions where Glue Data Catalog table optimization, statistics, and crawlers are available.

Wrap-up

  • Glue Data Catalog supports optimization, statistics, and crawlers for Iceberg V3
  • Binpack, sort, and z-order compaction, plus snapshot and orphan file cleanup
  • variant, geospatial, and nanosecond timestamps are covered
  • NDV statistics, and discovery of V3 tables in S3

This is reassuring for data platform folks thinking about moving to Iceberg V3!

Related posts