JA EN
shiichan

Data Quality Checks Right Inside Spark Jobs: Amazon EMR Data Quality!

AWS What's New ★★★★ View Markdown
#aws#infrastructure#devops

Hi everyone, it's Shii-chan! This one is for data engineers who want to stop bad data before it moves downstream!

AWS What's New aws.amazon.com

What was announced?

According to AWS What's New, Amazon EMR can now validate data quality natively within Apache Spark jobs. Amazon EMR Data Quality is powered by AWS Glue, so you don't need to build or maintain a separate validation infrastructure. It comes at no additional cost.

What changes

  • Author rules directly in your Spark scripts
  • Reference rules stored in the AWS Glue Data Catalog
  • Apply checks at any stage: after reading, during transformation, or after processing

Dive Deep

It uses the same rules language (Data Quality Definition Language, DQDL) and APIs as AWS Glue, giving a consistent experience across both engines.

  • 30+ rule types
  • Dynamic rules that compare against historical runs
  • ML-based anomaly detection for tracking drift without fixed thresholds

Based on the evaluation results, you can fail the pipeline, filter invalid records, or route them to quarantine. Results are produced at the dataset, rule, and row level, so it's easy to see which records or rules drive the failures.

Support covers:

  • Table formats: Apache Iceberg, Apache Hudi, Delta Lake
  • Deployment modes: EC2, EKS, Serverless
  • Runtime: AWS runtime for Apache Spark (emr-spark-8.1.0)

Wrap-up

  • Native data quality validation inside EMR Spark jobs
  • Same DQDL and APIs as Glue, no additional cost
  • 30+ rule types plus ML anomaly detection
  • Choose to fail, filter, or quarantine

Great for data platform teams running Spark on EMR who kept quality checks in a separate system!

Related posts