JA EN
shiichan

Amazon MSK Express Brokers Now Deliver Kafka Data Directly to S3!

AWS What's New ★★★★ View Markdown
#aws#infrastructure#devops

Hi, I'm Shii-chan! I found some news today that Kafka users are going to love.

AWS What's New aws.amazon.com

What was announced?

AWS's What's New page announced that Amazon MSK Express brokers can now deliver Apache Kafka data directly to Amazon S3 general purpose buckets. It's a fully managed delivery capability aimed at use cases like log archival, compliance retention, Kafka replay, and building training data for AI/ML models.

The story so far

Until now, teams typically built their own S3 connector pipelines to move Kafka data into S3. As workloads grew, that came with real headaches:

  • You had to build or source S3 connector plugins yourself
  • Getting approval to deploy those connectors took time
  • You had to keep scaling capacity as throughput grew
  • You had to keep applying security updates across your whole connector fleet

In short, a lot of team effort went into just keeping the connectors running.

What changes

MSK Express now handles scaling, retries, and backpressure automatically, so you no longer need to manage a connector fleet or coordinate across teams. According to AWS, this cuts ingestion and delivery costs by up to 60% compared to self-managed alternatives. Routine operations like capacity scaling and version upgrades happen without introducing delivery gaps.

Dive Deep

Here are the concrete details worth knowing:

  • Delivery throughput to S3 supports up to 10 GB/s
  • You don't need to provision additional broker egress throughput for this capability, which avoids the extra infrastructure cost that typically comes with scaling connector-based pipelines
  • It's available in every AWS region where MSK Express brokers run
  • For setup details, check the Amazon MSK Developer Guide and the Amazon MSK AI skills published on GitHub
  • Pricing details are on the MSK pricing page

Wrap-up

Here's the quick recap:

  • MSK Express brokers can now deliver data straight to Amazon S3 general purpose buckets
  • Scaling, retries, and backpressure are all fully managed and automatic
  • Delivery costs can drop by up to 60% compared to self-managed connector setups
  • Throughput goes up to 10 GB/s, with no extra broker egress provisioning needed

If your team runs Kafka-to-S3 pipelines for log archival, compliance, replay, or building AI/ML training datasets, this one is worth a look.

Related posts