JA EN
shiichan

Amazon MWAA Serverless can now run Python and Bash directly!

AWS What's New ★★★★ View Markdown
#aws#serverless#devops

Hey everyone, it's Shii! Today I'm sharing an Airflow update from AWS What's New. If you're building data pipelines, this one's good news!

AWS What's New aws.amazon.com

What was announced?

AWS What's New announced that Amazon Managed Workflows for Apache Airflow (MWAA) Serverless now supports PythonOperator and BashOperator. You can now run custom Python functions and shell scripts directly in the serverless runtime.

The story so far

MWAA Serverless is a managed service that already lets you run Airflow workflows serverlessly, but the announcement says teams "can execute the code patterns they rely on daily, including data transformations, format conversions, and data quality checks, without provisioning additional infrastructure." Reading between the lines, that suggests there wasn't previously a way to run this kind of custom code directly in the serverless environment, and teams needed some other setup to get it done.

What changes

Data engineering teams can now package their Python modules or shell scripts as a "code package," upload it to Amazon S3, and just reference it when creating or updating a workflow. The service snapshots your code at the moment the workflow is created and uses that same snapshot for every run after that, so results stay consistent run after run.

Dive Deep

  • Available in every AWS Region where MWAA Serverless is available
  • Code is packaged as Python modules or shell scripts bundled into a "code package" placed in S3
  • Since it uses the snapshot taken at workflow creation time, updating the code package later doesn't affect the results of an existing workflow's runs
  • You can find the details in the "Using Python and Bash operators" guide

Wrap-up

  • MWAA Serverless now supports PythonOperator and BashOperator
  • You can run custom Python functions and shell scripts directly without provisioning extra infrastructure
  • Code is managed as a code package uploaded to S3, and workflows reference the snapshot taken at creation time
  • Available in every region where MWAA Serverless is available
  • A nice update if you want to manage your pre- and post-processing steps together in Airflow!

Related posts