No More Recreating Connectors! Update MSK Connect Plugins and Worker Configurations in Place

Hi, it's Shii-chan!

AWS What's New aws.amazon.com

If you run Kafka Connect, have you ever had to delete a connector and rebuild it just to make an update? Today's news says goodbye to that!

What was announced?

According to AWS What's New, Amazon MSK Connect now lets you update the custom plugin and worker configuration of an existing connector.

Amazon MSK Connect is a fully managed service that lets you deploy, monitor, and scale Apache Kafka Connect connectors with Amazon MSK (Managed Streaming for Apache Kafka).

The story so far

Previously, updating a connector's plugin or worker configuration meant deleting the connector and recreating it. That was a disruptive operation that added risk and operational overhead.

What changes

You can now update the plugin and worker configuration in place. The connector keeps its committed offsets, so its tasks continue processing from where they left off once the update completes.

Here are some cases where this helps:

• Updating the custom plugin to run a new version of your connector code, for example one with updated dependencies
• Switching key and value converters in the worker configuration
• Tuning producer and consumer properties for throughput and latency

Dive Deep

This is available at no additional cost in all AWS Regions where Amazon MSK Connect is supported. See the Amazon MSK Connect developer guide for the steps.

Wrap-up

• Update the custom plugin and worker configuration of an existing connector in place
• No delete-and-recreate, and offsets are preserved
• No additional cost, in all Regions where MSK Connect is supported

This is great if you run Kafka Connect connectors in production and used to hold your breath at every version bump or config tweak!