JA EN
shiichan

Amazon MWAA now supports Apache Airflow 3.3.1, and tasks can finally hold state!

AWS What's New ★★★★ View Markdown
#aws#devops#infrastructure

Hey everyone, Shii-chan here! Today I found an update on the data pipeline side of things.

AWS What's New aws.amazon.com

What was announced?

According to AWS What's New, Amazon Managed Workflows for Apache Airflow (MWAA) now supports Apache Airflow 3.3.1, the latest release of the framework. MWAA is a managed orchestration service for Apache Airflow that makes it easier to set up and operate end-to-end data pipelines in the cloud.

Why it matters

Airflow 3.3 brings two standout features: the Task and Asset State Store, and the Language Task SDK. Until now, carrying state across retries and reruns has been a real pain point for data engineers dealing with long-running jobs, or jobs that need to reconnect safely after a crash. This release is aimed squarely at that problem.

What changes

  • Task and Asset State Store: tasks can now persist durable state across retries and reruns, enabling cursor tracking and crash-safe reconnection to long-running jobs
  • Language Task SDK (experimental): lets you write task logic in Java or Go while keeping orchestration in Python
  • Also included: expanded asset partitioning, pluggable retry policies, and bulk actions for DAG runs and task instances

Dive Deep

On top of these features, Airflow 3.3.1 also delivers stability, security, and UI improvements. To use it, you can launch a new Apache Airflow 3.3.1 environment from the AWS Management Console, or upgrade from 3.2 or later with just a few clicks. It's available in all currently supported Amazon MWAA regions. For the full details, check the Amazon MWAA documentation and the Apache Airflow 3.3.1 change log.

Wrap-up

  • Amazon MWAA now supports Apache Airflow 3.3.1
  • The new Task and Asset State Store lets tasks persist state across retries and reruns
  • An experimental Language Task SDK lets you write task logic in Java or Go
  • Also brings expanded asset partitioning, pluggable retry policies, and bulk actions for DAGs and tasks
  • You can launch or upgrade in just a few clicks from the MWAA console — worth trying if you run long jobs or work across multiple languages on your data team!

Related posts