JA EN
shiichan

No more digging through logs — an AI agent now diagnoses your Spark failures!

AWS What's New ★★★★ View Markdown
#aws#ai#devops

Hey there, it's Shii-chan! Spark troubleshooting just got another upgrade.

AWS What's New aws.amazon.com

What was announced?

According to AWS's What's New, Amazon EMR on EKS now supports the Apache Spark troubleshooting agent. Data engineers can diagnose EMR on EKS job failures through natural language, getting automated root cause analysis and PySpark code recommendations back.

The story so far

This Spark troubleshooting agent already worked with EMR on EC2 and EMR Serverless, but EMR on EKS wasn't covered yet. When a job failed on EKS, you had no choice but to dig through distributed executor logs and Spark History Server data yourself to find the cause.

What changes

Data engineers using EMR on EKS can now hand off the investigation of failed jobs to the agent. Common issues like memory errors, data skew, resource contention, and connectivity failures can be pinpointed without manually digging through logs. With this, the agent now covers all EMR deployment options: EMR on EC2, EMR Serverless, and EMR on EKS.

Dive Deep

The agent analyzes Spark History Server data, distributed executor logs, and cluster configurations to identify the cause. There are two ways to use it: directly from the EMR on EKS console via a "Troubleshoot with AI" option on failed jobs, or through MCP (Model Context Protocol) using compatible AI coding agents like Kiro, Claude Code, and Cursor.

All operations are read-only, authenticated with IAM roles, and logged in AWS CloudTrail, so permissions stay well controlled. It's available in AWS Regions where SageMaker Unified Studio is available.

Wrap-up

  • EMR on EKS now supports the Apache Spark troubleshooting agent
  • Get natural-language root cause analysis and PySpark fix suggestions for failed jobs
  • Use it via the console's "Troubleshoot with AI" or through MCP-connected AI coding agents
  • Read-only, IAM-authenticated, and CloudTrail-logged, now covering EMR on EC2, Serverless, and EKS

If you run Spark jobs on EMR on EKS and want to make troubleshooting easier, this one's for you!

Related posts