DynamoDB Streams can now start reading from any timestamp!
Hi, it's Shii-chan! Now you can say "resume from that moment" in stream processing.
AWS What's New
aws.amazon.com
What was announced?
AWS What's New says Amazon DynamoDB Streams now supports timestamp-based stream processing, so you can begin reading stream records from a specific point in time.
The story so far
Previously you had only two choices:
- Start from the oldest retained record (up to 24 hours old)
- Start from new records written after you begin consuming
What changes
Supply a timestamp with the AT_TIMESTAMP shard iterator type and processing begins at the first record written at or after that time. You avoid reprocessing entire retained streams or missing records when resuming.
Dive Deep
The use cases called out:
- Recovery after an interruption
- Data replay for debugging and auditing by reading from a specific point in the past
- Skipping a backlog by resuming from a recent timestamp near the latest records
- Synchronizing after a restore by starting at the export timestamp following a point-in-time recovery (PITR) export
Other details:
- Works with the Amazon Kinesis Client Library (KCL) through the DynamoDB Streams Kinesis adapter
- Available in all commercial AWS Regions, the AWS GovCloud (US) Regions, and the China Regions
- No additional charge beyond standard GetRecords API request charges
See the DynamoDB Streams timestamp documentation in the developer guide.
Wrap-up
AT_TIMESTAMPstarts from the first record at or after a chosen time- Before, it was the oldest record (up to 24 hours) or new records only
- Handy for recovery, replay, skipping backlogs, and syncing after PITR exports
- Works with KCL via the Kinesis adapter, no extra charge
Perfect if you run DynamoDB Streams consumers!
