Amazon MSK Connect Can Now Pause and Resume Connectors!
Hi, it's Shiichan!
AWS What's New
aws.amazon.com
I found a handy update for Amazon MSK Connect today, so let me walk you through it.
What was announced?
According to AWS What's New, Amazon MSK Connect (Amazon Managed Streaming for Apache Kafka Connect) now supports pausing and resuming connectors.
When you pause a connector, it stops processing records, and you can resume it later. Some situations where this helps:
- Stopping data flow during maintenance on a source database
- Stopping data flow during maintenance on a sink target
- Pausing processing while you investigate an upstream or downstream issue
The story so far
Until now, pausing a connector meant deleting it and recreating it. The announcement describes this as a more disruptive operation that adds risk and operational overhead.
What changes
You can now pause a running connector and resume it in place when you're ready.
A paused connector keeps:
- Its configuration
- Its capacity settings
- Its committed offsets
When you resume, the tasks continue from the offsets they last committed. That means fewer worries about re-entering settings or handling offsets after a recreate!
Dive Deep
Here is what the announcement tells us:
- You can use it from the Amazon MSK Console, AWS CLI, AWS SDK, and AWS CDK
- There is no additional cost
- It's available in all AWS Regions where Amazon MSK Connect is available
The announcement doesn't list API parameter names or detailed steps, so check the Amazon MSK Connect documentation for those.
Wrap-up
- You can pause and resume connectors without deleting them
- Configuration, capacity settings, and committed offsets are kept while paused
- After resuming, processing continues from the last committed offsets
- Works with the console, CLI, SDK, and CDK at no extra cost
If you run Kafka Connect and sometimes need to stop it for database or sink maintenance, this one is for you!
