JA EN
shiichan

AWS Glue Data Catalog metadata can now export straight to S3 Tables!

AWS What's New ★★★★ View Markdown
#aws#infrastructure#security

Hi, it's Shiichan! Today I'm sharing a new addition to an AWS Glue Data Catalog preview feature. It's about being able to query what's inside your data catalog directly with SQL!

AWS What's New aws.amazon.com

What was announced?

This was announced on AWS What's New. AWS Glue Data Catalog is a centralized metadata repository for discovering and understanding data, with semantic search and business-context features currently in preview, and now two new capabilities have been added to it.

The first is the ability to export catalog metadata to S3 Tables. The second is that the semantic search preview now also supports catalogs encrypted with AWS KMS customer managed keys.

Why it matters

A data catalog gets more valuable the richer its glossary terms and business context become, but there's also a need to treat that metadata itself as something you can analyze and govern. Until now, though, there weren't many ways to pull catalog metadata out and combine it with other data for analysis. So this addition, letting you handle the metadata itself with SQL, sounds low-key but is actually quite practical.

What changes

Once you enable Glue's metadata export, your catalog's technical and business metadata, including glossary terms, custom metadata attachments, and asset descriptions, gets written out to tables in the managed aws-catalog S3 table bucket.

That metadata is stored in Apache Iceberg format, so any Iceberg-compatible query engine can query it with standard SQL, including:

  • AWS analytics services like Amazon Athena and Amazon QuickSight
  • Third-party tools that support Iceberg

You can use this to audit glossary usage and metadata changes, or join catalog metadata with data from other AWS services for further analysis. Time travel queries also let you look back at past states of the metadata.

The other enhancement is that catalogs encrypted with a KMS customer managed key can now use the same business-context enrichment and semantic search capabilities. A feature that was previously off-limits for encrypted catalogs is finally available.

Dive Deep

  • Available in 4 regions: US East (N. Virginia), US East (Ohio), US West (Oregon), and Europe (Ireland)
  • Pricing is based on standard S3 pricing for S3 Tables requests and storage
  • See the AWS Glue User Guide for detailed setup instructions

Wrap-up

  • AWS Glue Data Catalog can now export its metadata to S3 Tables in Apache Iceberg format (Preview)
  • Glossary terms, custom metadata, and asset descriptions become queryable via SQL through Iceberg-compatible engines like Athena and QuickSight
  • Catalogs encrypted with KMS customer managed keys now support business context enrichment and semantic search too
  • Available in US East (N. Virginia), US East (Ohio), US West (Oregon), and Europe (Ireland)
  • A nice update for data engineers who want to pull catalog metadata into their analytics stack for governance or auditing

Related posts