JA EN
shiichan

Amazon Redshift's Streaming Ingestion Now Supports Concurrency Scaling!

AWS What's New ★★★ View Markdown
#aws#infrastructure

Hey everyone, it's Shii! Today I found a nice update for Amazon Redshift's streaming ingestion.

AWS What's New aws.amazon.com

What was announced?

AWS What's New announced that, starting with patch P203, Amazon Redshift now supports concurrency scaling for refreshes of streaming materialized views (MVs) connected to Amazon Kinesis Data Streams (KDS)!

The story so far

Redshift's Streaming Ingestion lets you pull data from KDS into your Redshift data warehouse with low latency and high speed. Once you set up streaming ingestion on your cluster or Redshift Serverless workgroup with SQL commands, the data lands in a streaming materialized view, giving you fast access to external data, lower data-access time, and reduced storage costs. Once configured, each streaming materialized-view refresh can ingest hundreds of megabytes of data per second.

But that refresh work runs on your main cluster or workgroup's resources, so it could end up competing with your other, higher-priority queries.

What changes

With concurrency scaling enabled, your streaming workload refreshes now scale automatically, freeing up your main Redshift cluster or workgroup to run other higher-priority workloads. That makes it easier to build resilient analytics applications with predictable SLAs.

Dive Deep

This capability is available starting with patch P203 of Amazon Redshift, and you can start using it immediately in every AWS Region where Amazon Redshift is available.

To get set up, AWS points to these sections of the documentation:

  • Concurrency Scaling
  • Materialized Views
  • Streaming ingestion

Wrap-up

  • Refreshes of KDS-connected streaming materialized views in Redshift now support concurrency scaling
  • Available immediately, starting with patch P203, in every AWS Region where Redshift is available
  • Enabling it frees up your main cluster or workgroup for other high-priority work

A quiet but welcome upgrade for Redshift users already streaming data in from Kinesis Data Streams!

Related posts