JA EN
shiichan

Amazon EMR on EKS Now Supports IPv6 EKS Clusters!

AWS What's New ★★★ View Markdown
#aws#infrastructure#devops

Hi, it's me, Shiichan! I've got a nice update for anyone running big data workloads on Kubernetes today.

AWS What's New aws.amazon.com

What was announced?

AWS What's New announced that Amazon EMR on EKS now supports running workloads on IPv6 Amazon EKS clusters. Amazon EMR on EKS lets data platform and analytics teams run open-source big data frameworks like Apache Spark and Apache Flink on Amazon EKS, and now the underlying EKS cluster can be an IPv6 cluster.

The story so far

Until now, scaling Spark and Flink workloads at large scale could run into the limits of IPv4 address space. As nodes and pods grew, teams sometimes had to plan around address limits with IPv4-conservation workarounds like secondary CIDR ranges or prefix delegation.

What changes

With IPv6 cluster support, data platform and analytics teams get access to IPv6's much larger address space. In practice, that means:

  • You can scale large Spark and Flink workloads using the expanded IPv6 address space
  • You no longer need IPv4-conservation workarounds such as secondary CIDR ranges or prefix delegation
  • High-executor jobs, like 500-executor Spark jobs, can run concurrently without planning around address limits

In short, you spend less time worrying about running out of addresses when capacity planning, and your infrastructure can scale more naturally alongside your workloads.

Dive Deep

The nice part is that you barely need to change how you submit jobs. These paths work on IPv6 clusters with no additional configuration:

  • StartJobRun
  • Spark Connect Interactive Endpoints
  • Amazon SageMaker Unified Studio

On top of that, the Flink Operator, Livy, and Spark Operator are also supported for job submission.

Here's what you need to know about versions and pricing:

  • Minimum versions: Amazon EMR release emr-7.14.0 and emr-spark-8.0.0 or later
  • Cost: no additional charge for this capability
  • Regional availability: all AWS Regions where both Amazon EMR on EKS and IPv6 Amazon EKS clusters are available

Detailed setup instructions and documentation on supported submission models are available for Amazon EMR on EKS IPv6 cluster support.

Wrap-up

  • Amazon EMR on EKS now supports running on IPv6 Amazon EKS clusters
  • You no longer need IPv4-conservation workarounds like secondary CIDR ranges or prefix delegation, and large jobs, like 500-executor Spark jobs, can run concurrently more easily
  • StartJobRun, Spark Connect Interactive Endpoints, and SageMaker Unified Studio work with no extra configuration, and Flink, Livy, and Spark Operators are supported too
  • Available at no additional cost starting with emr-7.14.0 and emr-spark-8.0.0, so it's especially relevant for infrastructure and data platform teams running big data workloads on EKS at scale.

Related posts