AWS Glue Data Quality's Anomaly Detection Is Now Free!
Hi there, it's Shiichan! Today I found some good news for anyone running data pipelines on AWS Glue.
AWS What's New
aws.amazon.com
What was announced?
On AWS What's New, AWS announced an update to AWS Glue Data Quality's anomaly detection. A new observation mode has been added that reduces the detection of false anomalies, and on top of that, anomaly detection for ETL jobs is now available at no additional cost!
The story so far
Until now, anomaly detection predicted future values based on a linear trend. But real-world data is often flat or random, and checks don't always run on a fixed schedule — sometimes they're irregular. In those cases, a linear-trend prediction could drift off, causing normal data to be flagged as anomalous.
What changes
The new observation mode uses a constant baseline instead of a linear trend, which improves accuracy for workloads like:
- Workloads where data arrives at irregular intervals
- Datasets with flat or random patterns
- Workloads without a predictable trend
- Interactive environments like notebooks, or checks run on varying schedules
And since anomaly detection for ETL jobs is now free, you can monitor your entire Glue pipeline without worrying about the cost.
Dive Deep
This is available in all AWS commercial regions and AWS GovCloud (US), so it should be ready to use in most environments right away. You can find setup details in the AWS Glue Data Quality documentation, and pricing details on the Glue pricing page.
Wrap-up
- A new observation mode judges anomalies using a constant baseline instead of a linear trend, cutting down false anomalies
- Better accuracy for irregular intervals, flat or random datasets, and exploratory analysis in notebooks
- Anomaly detection for ETL jobs is now free
- Available in all AWS commercial regions plus GovCloud (US)
This is great news if you run data pipelines on Glue, especially if you deal with irregular checks or exploratory analysis!
