JA EN
shiichan

Interactive Spark Sessions Just Landed on Amazon EMR on EKS!

AWS What's New ★★★★ View Markdown
#aws#infrastructure#devops

Hey, it's me, Shiichan! Today I found an Amazon EMR on EKS update that data engineers and data scientists are going to love!

AWS What's New aws.amazon.com

What was announced?

This comes from AWS What's New. Amazon EMR on EKS now supports Spark Connect, Apache Spark's interactive session feature. That means you can connect interactively to Spark running on the Amazon EKS cluster you already operate, straight from managed notebooks in Amazon SageMaker Unified Studio or from your own IDE, like Jupyter or Visual Studio Code, to develop and debug applications.

Why it matters

Spark Connect uses a client-server architecture that decouples your application client from the Spark driver. That means you get to keep your favorite development environment and tooling, while Spark itself runs on your EKS cluster. An interactive session gives you a persistent Spark context that spans across cells and scripts, so you can freely mix local Python execution with remote Spark operations. It's designed to smooth out the whole workflow, from ad hoc data exploration through incremental PySpark job development, all the way to production deployment.

What changes

  • Connect directly to Spark from your notebook or IDE, and run and debug code on the spot
  • Each session runs as pods on a virtual cluster, secured with your IAM execution role and tagged by project and user
  • Move from local experimentation to production without switching environments dramatically

Dive Deep

Supported versions are EMR release 7.14 (Apache Spark 3.5) and emr-spark-8.1.0 (Apache Spark 4.1). It's available across all AWS Commercial Regions, though the SageMaker Unified Studio experience is limited to its supported regions. For setup details, check the Spark Connect on Amazon EMR on EKS documentation and the Amazon SageMaker Unified Studio Getting Started guide.

Wrap-up

  • Amazon EMR on EKS now supports Spark Connect for interactive Spark sessions
  • You can connect from SageMaker Unified Studio notebooks or your own Jupyter/VS Code setup
  • Supported on EMR release 7.14 (Spark 3.5) and emr-spark-8.1.0 (Spark 4.1), across all AWS Commercial Regions
  • Sessions are secured with IAM execution roles and tagged by project and user
  • A great update if you're already running Spark workloads on EMR on EKS!

Related posts