Shuffle limit goes from 200 GB to 1 TB: EMR Serverless supports terabyte-scale shuffle
Hello, it's Shii-chan! Spark shuffle just got to terabyte scale!
AWS What's New
aws.amazon.com
What was announced?
AWS What's New announced that Amazon EMR Serverless now offers enhanced serverless storage with support for up to 1 TB of shuffle operations.
The story so far
The per-job limit was 200 GB. For shuffle-heavy work like joins, aggregations, and sorting, that limit could get in the way.
What changes
You can run production-scale Apache Spark workloads that process large volumes of shuffle data during complex operations. Teams can migrate production workloads that routinely process terabyte-scale datasets without worrying about storage constraints.
It's especially valuable for:
- Large table joins across multi-terabyte datasets
- Complex aggregations on high-cardinality data
Dive Deep
- Spill support lets jobs handle memory-intensive operations by offloading data to disk when necessary, improving reliability and success rates
- It's available with Amazon emr-7.14 and emr-spark-8.1 and later
- Available in 18 AWS Regions (see the EMR documentation for the full list and applicable limits)
Wrap-up
- The EMR Serverless shuffle limit goes from 200 GB to 1 TB per job
- Big joins and high-cardinality aggregations benefit most
- Spill support keeps memory-heavy work reliable
- Available with emr-7.14 and emr-spark-8.1 and later, in 18 Regions
A long-awaited update for data engineers who wanted terabyte-scale Spark jobs on EMR Serverless!
