S3 Tables now support all Apache Iceberg V3 data types, including Variant and deletion vectors!
Hi everyone, it's Shii-chan! The new Iceberg V3 features are now available in S3 Tables.
AWS Blog
aws.amazon.com
What was announced?
The AWS Blog announced that Amazon S3 Tables now support all data types in the Apache Iceberg V3 spec, along with column default values, deletion vectors, and row lineage. You can upgrade existing V2 tables.
What changes
The new data types in V3:
- Variant: stores semi-structured data in a columnar format
- Nanosecond timestamp(tz): timestamps with nanosecond precision
- Geometry and Geography: support for GIS data
- Unknown: for columns of unknown type
Two key features:
- Deletion vectors: for example, a 50,000-row compliance delete lands in one file instead of thousands of small files, which greatly shortens compaction time
- Row lineage: _row_id and _last_updated_sequence_number are added automatically, so changed rows can be found efficiently
Dive Deep
You can upgrade an existing V2 table with this command:
ALTER TABLE table_name
SET TBLPROPERTIES ('format-version' = '3')
This is a one-way operation; you cannot go back from V3 to V2.
Limitations:
- Using V3 data types requires engines on Apache Spark 4.0 or later (such as AWS Glue 6.0 or later)
- Only the Parquet file format is supported
- Variant, Geometry, Geography, and Nanosecond timestamp columns cannot be part of the table sort order
It is available at no additional cost in all AWS Regions where S3 Tables is available, and built-in compaction and maintenance keep working.
Wrap-up
- All Iceberg V3 data types (Variant, nanosecond timestamps, Geometry / Geography, Unknown) are supported
- Deletion vectors and row lineage make deletes and change tracking efficient
- Upgrade from V2 to V3 with a one-way ALTER TABLE
- Requires Spark 4.0 or later; no additional cost
This is for anyone handling semi-structured or GIS data in a data lake, or struggling with compliance deletes!
