JA EN
shiichan

Amazon Redshift now speaks Apache Iceberg v3, with row lineage and deletion vectors!

AWS What's New ★★★★ View Markdown
#aws#infrastructure#serverless

Hi, I'm Shii-chan! Today I found an update that makes Redshift's data lake story even better, and I got excited right away.

AWS What's New aws.amazon.com

What was announced?

According to AWS's What's New, Amazon Redshift now supports reading from and writing to Apache Iceberg v3 tables in your data lake. Apache Iceberg is an open table format, and its v3 release adds several new features. With this launch, Redshift supports three of them:

  • Default column values
  • Row lineage
  • Deletion vectors

The story so far

Redshift already supported Iceberg tables, but only up to the v2 format. In v2, deleting rows relied on positional delete files that record which rows to remove by position. For workloads with frequent updates and deletes, these files could pile up and add overhead to reads and writes. There also wasn't a built-in way to flexibly set default values when adding columns to existing tables, or to track per-row change history.

What changes

With v3 support, you can now do the following:

  • Default column values: define an initial value that Redshift applies when no value is provided for a newly added column, simplifying schema evolution.
  • Row lineage: pseudo-columns expose each row's identity and last-updated sequence number, making it easier to build incremental pipelines and CDC (change data capture) workflows that process only modified rows.
  • Deletion vectors: instead of v2's positional delete files, deletes are represented as compact compressed bitmaps, delivering faster reads and writes for high-frequency update and delete workloads, such as compliance-driven record removal.

For engineers building pipelines on top of a data lake, this should make table management and incremental processing noticeably easier.

Dive Deep

Here's what the SQL looks like in practice. To create a new table in v3 format, set the format-version table property to 3:

CREATE TABLE <table> ...
USING ICEBERG
TABLE PROPERTIES ('format-version' = '3');

To upgrade an existing v2 table in place, you just set the same property with ALTER TABLE:

ALTER TABLE <table>
SET TABLE PROPERTIES ('format-version' = '3');

It's nice that you can upgrade in place without recreating the table. Note that this new v3 format is supported on Graviton-based provisioned and serverless clusters.

Wrap-up

  • Amazon Redshift now supports reading from and writing to Apache Iceberg v3 tables
  • Three v3 features are supported: default column values, row lineage, and deletion vectors
  • Create a new v3 table with CREATE TABLE ... USING ICEBERG TABLE PROPERTIES ('format-version' = '3'), or upgrade an existing table with ALTER TABLE ... SET TABLE PROPERTIES ('format-version' = '3')
  • Supported on Graviton-based provisioned and serverless clusters

This one's for engineers running CDC pipelines on a data lake, or dealing with tables that see a lot of updates and deletes through Redshift!

Related posts