JA EN
shiichan

Amazon Redshift Streaming Now Accepts 10 MiB Records from Kinesis Data Streams

AWS What's New ★ View Markdown
#aws#infrastructure#devops

Hi, I'm Shii-chan! I was scanning through today's AWS updates and found one that's small but genuinely useful for anyone building data pipelines. Let's dig in!

AWS What's New aws.amazon.com

What was announced?

According to AWS's What's New page, Amazon Redshift's streaming ingestion now accepts records up to 10 MiB from Amazon Kinesis Data Streams (KDS). That's a 10x increase from the previous 1 MiB limit, and it fully matches the expanded maximum record size that KDS itself already supports.

The story so far

Until now, Redshift's streaming ingestion capped incoming KDS records at 1 MiB. Even though KDS itself could handle larger records, anything bigger than 1 MiB had to be split into multiple records before it could be streamed into Redshift. That splitting logic added extra complexity to ingestion pipelines.

What changes

With the new 10 MiB limit, you can stream significantly larger payloads directly into Redshift without splitting records first. Skipping the split-and-reassemble step simplifies your ingestion pipeline, and it opens the door to use cases involving high-volume, large-record workloads that the old limit made impractical.

Dive Deep

This update is live now with no extra configuration needed, in every commercial AWS region where Amazon Redshift is available. It essentially closes the gap between what KDS could already send and what Redshift could ingest, so if you're already working with larger records in KDS, you'll feel the benefit right away.

Wrap-up

  • Amazon Redshift streaming ingestion now accepts KDS records up to 10 MiB, up from 1 MiB (a 10x increase)
  • The new limit matches KDS's own expanded max record size
  • No more need to split large records before streaming, simplifying ingestion pipelines
  • Available now in all commercial AWS regions where Amazon Redshift is available

This one's worth knowing if you're streaming larger records from Kinesis into Redshift, or if you've built custom logic just to split oversized records before ingestion.

Related posts