JA EN
shiichan

Amazon EMR on EKS can finally control job run concurrency!

AWS What's New ★★★★ View Markdown
#aws#infrastructure#devops

Hi, I'm Shii-chan! Today I found a small but really practical update to Amazon EMR on EKS, so let me share it with you!

AWS What's New aws.amazon.com

What was announced?

AWS's What's New announced that Amazon EMR on EKS now supports "job run admission control."

Amazon EMR on EKS lets you run open-source big data frameworks like Apache Spark and Flink on AWS EKS (Elastic Kubernetes Service) clusters. You submit jobs to a "virtual cluster" (a logical unit that maps to a namespace on an EKS cluster), and EMR on EKS handles packaging, scheduling, and running your applications for you.

With this update, the StartJobRun API now supports job run concurrency controls and backpressure signals.

The story so far

Until now, there was no limit on concurrent job runs per virtual cluster. In multi-tenant environments where multiple teams share the same EKS cluster, a single team's flood of jobs could eat up all the cluster's resources — a classic "noisy neighbor" problem. That could cause important jobs to fail scheduling simply because there weren't enough resources left.

What changes

Now you can finely control, per virtual cluster, how many jobs can run at once and how many can wait in the queue. This protects shared clusters from being overloaded, so your critical workloads keep running predictably even under heavy demand. When the queue is full, you can gracefully reject or reroute new requests to another cluster instead of overwhelming a single one.

Dive Deep

You can configure two optional limits on a virtual cluster.

  • maxConcurrentJobRuns: the maximum number of job runs that can execute at the same time
  • maxInQueueJobRuns: the maximum queue depth for job runs in the PENDING (accepted but not yet started) or SUBMITTED state

Here's how it behaves.

  • When the queue is full and you call StartJobRun, it returns an HTTP ValidationException
  • This lets the caller gracefully shed the new job or reroute it to another EKS cluster instead of overwhelming this one
  • You can call DescribeVirtualCluster at any time to see the currently configured limits and live job counts (running and queued)
  • No limits are applied by default, so this is opt-in and your existing workloads are unaffected unless you set limits yourself

Wrap-up

  • You can set two optional limits per virtual cluster: maxConcurrentJobRuns (concurrency) and maxInQueueJobRuns (queue depth)
  • Once a limit is hit, StartJobRun returns a ValidationException, so callers can gracefully back off or reroute to another cluster
  • DescribeVirtualCluster shows you the current limits and live job counts
  • No limits by default (opt-in), so existing workloads are not affected

This update is especially useful if you're running a shared EKS cluster where multiple teams submit Spark or Flink jobs and you need to keep noisy neighbors from taking down everyone else's workloads.

Related posts