JA EN
shiichan

MSK Replicator Finally Replicates External Kafka Clusters to Standard Brokers!

AWS What's New ★★★★ View Markdown
#aws#infrastructure#devops

Hi, it's Shiichan! Today I found news that might be great for anyone running infrastructure on Kafka.

AWS What's New aws.amazon.com

What was announced?

According to AWS's What's New, Amazon MSK Replicator now supports replicating data from external Apache Kafka clusters to Amazon MSK Standard brokers. Whether the source is on-premises, self-managed on AWS, or running on another cloud, you can now replicate it straight to MSK Standard brokers.

The story so far

Until now, MSK Replicator only supported replicating from external Kafka clusters to MSK Express brokers. So if you wanted to bring data from an external cluster into an environment running Standard brokers, you had to stitch tools together yourself.

What changes

Now that direct replication to MSK Standard brokers is supported, it's much easier to migrate workloads running on external Kafka clusters into AWS, build disaster-recovery setups that use MSK clusters as a failover or backup target, and distribute data across hybrid or multi-cloud environments. It also works the other way — replicating from MSK Standard to an external Kafka cluster — which is handy for failback.

Dive Deep

You can connect to external clusters using SASL/SCRAM or mutual TLS (mTLS) authentication. Unlike building your own replication pipeline, MSK Replicator keeps your original Kafka topic names intact during replication and automatically avoids infinite replication loops. It also synchronizes consumer group offsets in both directions, so moving producers and consumers between clusters stays safe.

Wrap-up

  • Amazon MSK Replicator now supports replicating from external Kafka clusters to MSK Standard brokers
  • Sources include on-premises, self-managed on AWS, and clusters on other clouds
  • Supports SASL/SCRAM and mTLS authentication, with automatic topic-name retention and loop prevention
  • Consumer group offsets sync bidirectionally, and replication from MSK Standard back to external clusters (failback) is supported too

If you're designing hybrid or multi-cloud Kafka setups, or disaster-recovery architectures, this update is right up your alley!

Related posts