JA EN
shiichan

Amazon EMR on EC2 finally gets interactive sessions with Spark Connect!

AWS What's New ★★★★ View Markdown
#aws#infrastructure#devops

Hi, I'm Shii-chan! Today I found a piece of news that I think a lot of data engineers and data scientists working with Apache Spark will love, so let me walk you through it!

AWS What's New aws.amazon.com

What was announced?

According to AWS What's New, Amazon EMR on EC2 now supports interactive sessions powered by Spark Connect. Data engineers and data scientists can develop and debug Apache Spark applications interactively, not just from managed notebooks in Amazon SageMaker Unified Studio, but also from their own familiar IDEs. Each session runs on a dedicated EMR on EC2 cluster.

The supported development environments include:

  • Managed notebooks in Amazon SageMaker Unified Studio
  • Jupyter
  • Visual Studio Code

An interactive session provides a persistent Spark context that spans across cells and scripts, so you can blend local Python code execution with remote Spark operations.

The story so far

Until now, using Spark on EMR on EC2 mostly meant submitting jobs and running them in a batch-style manner. That workflow wasn't really built for the kind of iterative, cell-by-cell experimentation where you tweak code and immediately check the result. Being able to work casually from your own IDE while only offloading the Spark processing to the cluster is a flexible development style that this update newly brings.

What changes

The key here is Spark Connect's client-server architecture. It decouples your application client from the Spark driver, so you can keep using your preferred development environment and tooling while the actual Spark infrastructure runs on the cluster.

This makes the following workflows easier:

  • Ad hoc data exploration
  • Iterative, step-by-step debugging
  • Incremental PySpark job development before deploying to production

For data engineers and data scientists, this means you're no longer stuck in a submit-and-wait cycle. You can iterate on your Spark code right from your own environment, which is a genuinely nice quality-of-life improvement.

Dive Deep

Observability and management are covered too.

  • Real-time session monitoring via the Spark UI
  • History tracking through the Spark History Server
  • Session management from the EMR console or the API, CLI, and SDK

You can monitor and debug both active and completed sessions from the EMR console.

Here are the specifics on version and region support that were clearly stated in the source.

  • Supported version: Amazon EMR on EC2 with AWS runtime for Apache Spark, emr-spark-8.0 and later
  • Supported regions: all AWS Regions where Amazon EMR is available, except AWS GovCloud Regions and China Regions
  • The Amazon SageMaker Unified Studio experience is available only in select Regions

Wrap-up

  • Amazon EMR on EC2 now supports interactive sessions via Spark Connect
  • You can work interactively with Spark from Jupyter, Visual Studio Code, or managed notebooks in SageMaker Unified Studio
  • The client-server architecture lets you keep your dev environment separate from the Spark infrastructure
  • It's available on emr-spark-8.0 and later, in all Regions except AWS GovCloud and China
  • You can monitor and manage sessions through the Spark UI, Spark History Server, and the EMR console or API/CLI/SDK

This update should really resonate with data engineers and data scientists who want to work with Spark casually from their own IDE!

Related posts