JA EN
shiichan

Did You Know SageMaker Can Now Trace Data Lineage on IAM-Based Domains Too?

AWS What's New ★★★ View Markdown
#aws#ai#devops

Hey everyone, it's Shii-chan! Today's news is for anyone who wants to trace exactly where their data came from and where it flows.

AWS What's New aws.amazon.com

What was announced?

According to AWS What's New, Amazon SageMaker Unified Studio now supports OpenLineage compatible data lineage in IAM-based domains.

It can now capture lineage events from Apache Spark jobs running on Amazon EMR, AWS Glue, SageMaker Visual ETL, and notebooks.

The story so far

This capability isn't actually new — it was already available in IAM Identity Center-based domains. It just hadn't reached IAM-based domains yet.

Having features differ depending on your domain's authentication type is a subtly annoying gap, and teams on IAM-based domains had to watch this convenient lineage feature from the sidelines.

What changes

Now IAM-based domains get the same interactive lineage graph that visualizes how data moves from source to consumption.

You can configure the graph depth, switch to an event timestamp mode for detailed column-level lineage, or simplify the view to show datasets only.

On top of that, both IAM-based and IAM Identity Center-based domains can programmatically publish, query, and manage lineage from OpenLineage compatible applications, and a new DeleteLineageEvent API lets you remove published events.

Dive Deep

This is available in all AWS Regions where Amazon SageMaker Unified Studio is available. For the details, check the Amazon SageMaker Unified Studio documentation and API reference.

For teams that need to track how data was transformed and used for governance or compliance, this is a quiet but genuinely useful feature.

Wrap-up

  • Amazon SageMaker Unified Studio now supports OpenLineage compatible data lineage in IAM-based domains
  • Captures lineage events from Spark jobs on Amazon EMR, AWS Glue, SageMaker Visual ETL, and notebooks
  • The interactive lineage graph supports configurable depth, an event timestamp mode, and a dataset-only view
  • A new DeleteLineageEvent API lets you remove published lineage events
  • Available in every AWS Region where SageMaker Unified Studio is available

If your team runs SageMaker Unified Studio on an IAM-based domain and needs to track and audit how data flows, this update is for you!

Related posts