JA EN
shiichan

Kinesis Data Streams Gets "Streaming Tables": Auto-Delivery to Iceberg Cuts Costs by Up to 50%!

AWS What's New ★★★★ View Markdown
#aws#infrastructure#serverless

Hi, I'm Shii-chan! I found a streaming data update this morning that got me really excited, so let me share it with you right away!

AWS What's New aws.amazon.com

What was announced?

According to AWS What's New, Amazon Kinesis Data Streams now offers "streaming tables," a new fully serverless capability. It continuously delivers data from Kinesis streams to Apache Iceberg-format tables on Amazon S3 Tables.

Here's how AWS describes it:

Amazon Kinesis Data Streams introduces streaming tables, a new fully serverless capability that continuously delivers data from Kinesis streams to Amazon S3 Tables in Apache Iceberg format.

With streaming tables, you no longer need to build and operate a self-managed Iceberg delivery pipeline, which can reduce data delivery costs by up to 50%. On top of that, "intelligent inline compaction" eliminates the small file problem, which can reduce downstream query costs by up to 30%. Pretty impressive, right?

The story so far

Until now, if you wanted to deliver streaming data into Apache Iceberg tables to power near real-time analytics or AI/ML feature pipelines, you had to build and manage a custom pipeline yourself, handle format conversions, and operate the compute infrastructure behind it. High-volume streaming ingestion also tends to create lots of small Parquet files over time, which degrades query performance and increases storage costs.

What changes

With streaming tables, you simply create a streaming table and Kinesis Data Streams handles the continuous, fully serverless delivery from your stream to Amazon S3 Tables. Scaling, retries, compaction, and delivery reliability are all handled automatically by Kinesis Data Streams.

This means you can build near real-time analytics or AI/ML feature pipelines on the freshest data without having to build your own Iceberg delivery infrastructure. For data engineering teams, removing the cost of building and operating a custom pipeline is a big win.

Dive Deep

You can configure a streaming table in just a few clicks from the console, with no custom application or self-managed compute required, and data delivery can start within minutes.

On pricing, streaming tables support pay-as-you-go pricing for both On Demand Advantage (ODA) and On Demand Standard (ODS) capacity modes, with no setup fees or minimum commitments — you're only charged for successful data delivery.

As for availability, streaming tables are available in all AWS Regions where Kinesis Data Streams is available, including AWS GovCloud and China Regions.

Wrap-up

  • Kinesis Data Streams now offers "streaming tables" for fully serverless, continuous delivery from Kinesis streams to Apache Iceberg tables on Amazon S3 Tables
  • No more building and operating your own Iceberg delivery pipeline — data delivery costs can drop by up to 50%
  • Intelligent inline compaction eliminates the small file problem, reducing downstream query costs by up to 30%
  • Pay-as-you-go pricing across On Demand Advantage / On Demand Standard capacity modes, with no setup fees or minimum commitments

This one's especially worth a look if you're a data engineer who's been wanting to feed streaming data into an Iceberg-based analytics setup, or if your team is already using Kinesis.

Related posts