EMR Serverless Finally Speaks IPv6: Run Spark in IPv6-Only Subnets!
Hi everyone, it's Shii-chan! Today I have some great networking news for EMR Serverless.
AWS What's New
aws.amazon.com
What was announced?
In AWS What's New, Amazon EMR Serverless announced support for IPv6 and dual-stack connectivity.
You can now run Apache Spark and Apache Hive jobs in your Amazon VPC by supplying dual-stack (IPv4 and IPv6) or IPv6-only subnets.
The story so far
Until now, EMR Serverless supported IPv4 only.
Spark and Hive jobs can scale out to many workers, so the IPv4 address space in your VPCs and subnets can become a constraint. A growing number of organizations also face compliance mandates that call for IPv6.
What changes
There are two connectivity modes now:
- Dual-stack: your applications accept both IPv4 and IPv6. It is backward compatible with existing IPv4 workloads, so you can adopt IPv6 gradually.
- IPv6-only subnets: no IPv4 addresses are needed at all. This helps with IPv4 exhaustion and with standards or compliance requirements that call for IPv6, and your jobs can reach IPv6 endpoints and services both inside and outside the VPC.
Dive Deep
Here are the conditions from the announcement:
- No additional cost
- Available in all AWS Regions where EMR Serverless is available
- IPv6 and dual-stack subnets are supported on all EMR Serverless release versions
The setup steps are in the EMR Serverless User Guide under "Configure IPv6 subnets for EMR Serverless applications".
Wrap-up
- EMR Serverless now supports IPv6 and dual-stack
- Spark and Hive jobs can run in IPv6-only subnets
- Existing IPv4 workloads keep working
- No extra cost, all release versions, all supported Regions
This one is for data platform teams dealing with IPv4 exhaustion or IPv6 requirements!
