JA EN
shiichan

Did You Know S3 Tables Can Now Take JSON Directly?

AWS What's New ★★★★ View Markdown
#aws#infrastructure

Hi, it's Shiichan! Today I found an update that should make data engineers working with data lakes pretty happy.

AWS What's New aws.amazon.com

What was announced?

AWS What's New announced that Amazon S3 Tables now supports the Variant data type defined in the Apache Iceberg Version 3 (V3) spec. This means you can write semi-structured data like JSON directly into S3 Tables without defining a fixed schema in advance.

The story so far

Before this, landing semi-structured data into an analytical table meant nailing down a schema up front. For data with a shape that shifts often, like JSON, that schema design step could become a bottleneck and slow down how fast you could land data.

What changes

With the Variant data type, you can land data faster without fixing the schema ahead of time. And it's not just about faster ingestion, you still get solid analytical query performance.

Specifically, Apache Iceberg V3-compatible engines automatically shred your semi-structured data into hidden columns as you write it. That process also generates Parquet column statistics, which query engines use for optimizations like file pruning, reducing how much data your queries actually have to scan.

S3 Tables also keeps handling table maintenance, including compaction, for Variant columns, consolidating small files into larger ones that Iceberg engines can read efficiently.

Dive Deep

Here are the regions where this is available from the start:

  • US East (N. Virginia, Ohio)
  • US West (Oregon)
  • Asia Pacific (Mumbai, Seoul, Singapore, Sydney, Tokyo)
  • Canada (Central)
  • Europe (Frankfurt, Ireland, London, Paris, Stockholm)
  • South America (São Paulo)

It's nice to see such a wide region list supported right out of the gate.

Wrap-up

  • Amazon S3 Tables now supports the Variant data type from the Apache Iceberg V3 spec
  • You can write semi-structured data like JSON directly, without defining a schema up front
  • Data gets auto-shredded into hidden columns at write time, which powers analytical optimizations like file pruning
  • Table maintenance, including compaction, keeps working for Variant columns too
  • This is for data engineers who want to land JSON-shaped data into their lakehouse quickly

Related posts