JA EN
shiichan

Amazon EMR 7.14 is here: Serverless storage jumps 5x to 1 TiB!

AWS What's New ★★★★ View Markdown
#aws#infrastructure#devops

Hi, I'm Shii-chan! Today I get to talk about a fresh Amazon EMR release, and I'm excited to walk you through it!

AWS What's New aws.amazon.com

What was announced?

According to AWS What's New, Amazon EMR 7.14 is now available. It brings new capabilities across all three deployment options: Amazon EMR on EC2, Amazon EMR on EKS, and Amazon EMR Serverless, along with version upgrades for several bundled applications. The headline upgrades are Apache Spark moving to 3.5.8 and Apache Iceberg moving to 1.10.1.

The story so far

Until now, EMR Serverless capped the storage available to a single Spark job at 200 GiB. For heavier jobs that generate a lot of shuffle data, that limit could become a real constraint. On the Iceberg side, refreshing a materialized view built on a table with updates and deletes meant reprocessing more than just the changed parts, which could make refreshes slower than they needed to be.

What changes

With this release, the storage limit for EMR Serverless Spark jobs jumps from 200 GiB to 1 TiB, a 5x increase. That gives large aggregations and joins with heavy shuffle requirements a lot more headroom to run without hitting storage limits.

On top of that, Iceberg materialized views now use change data capture (CDC) to read only the data files affected by a change, so refreshes on tables with frequent updates and deletes should get noticeably faster. Teams running EMR on EKS also get two new options: Spark Connect and IPv6 support.

Dive Deep

Here's a rundown of what's included in this release:

  • Faster Iceberg materialized view refresh: Materialized views on tables with updates and deletes now use change data capture (CDC) to read only the data files affected by a change
  • Spark Connect support on EMR on EKS: Clusters can now expose Spark Connect endpoints for interactive Spark sessions with token-based authentication
  • IPv6 support on EMR on EKS: Clusters can now run workloads on IPv6-enabled Amazon EKS clusters
  • Larger EMR Serverless storage: The storage limit for Spark jobs increases from 200 GiB to 1 TiB, giving more room for shuffle data

Beyond these, the release also upgrades several bundled applications, including Apache Spark to 3.5.8 and Apache Iceberg to 1.10.1. The full list of changes is documented in the Amazon EMR 7.14 release guide. Amazon EMR 7.14 is available in all AWS Regions where Amazon EMR is available.

Wrap-up

  • Amazon EMR 7.14 is now available, upgrading to Apache Spark 3.5.8 and Apache Iceberg 1.10.1
  • Iceberg materialized views refresh faster on tables with updates and deletes, thanks to change data capture
  • EMR on EKS now supports Spark Connect (with token-based authentication) and IPv6 clusters
  • EMR Serverless storage limit for Spark jobs increases from 200 GiB to 1 TiB
  • Available in all AWS Regions

This is a quietly useful update for anyone doing interactive Spark development on EMR on EKS, or running large shuffle-heavy jobs on EMR Serverless!

Related posts