JA EN
shiichan

Lambda Can Now Consume Kafka Like a Queue, Beyond Partition Count!

AWS What's New ★★★★★ View Markdown
#aws#serverless#infrastructure

Hi, it's Shii-chan! Lambda and Kafka just got a new feature that tackles the parallelism problem!

AWS What's New aws.amazon.com

What was announced?

In AWS What's New, AWS announced that AWS Lambda now supports queues for Apache Kafka in self-managed Kafka event source mappings (ESMs). It works for Kafka clusters you run yourself and for managed offerings such as Confluent Cloud.

The story so far

Kafka ESMs used the Kafka consumer group protocol, which assigns each partition to exactly one consumer within a consumer group.

So to get more parallelism, you had to over-partition topics or move traffic to a separate queuing system. That meant more infrastructure to operate and more producer code to maintain.

What changes

Queue mode lets you scale processing beyond a topic's partition count. You get higher throughput during traffic spikes and more resilient processing, because a slow or failed event no longer blocks other events.

No changes to producers, topic configuration, or function code are needed.

It suits workloads where each event is an independent unit of work and processing speed matters more than strict ordering. Examples from the post:

  • Image and video encoding
  • Fraud detection
  • Notification delivery
  • Telemetry ingestion

Dive Deep

Here are the mechanics and caveats.

  • It's built on Kafka share groups, which are generally available in Apache Kafka 4.2
  • Multiple event pollers can consume concurrently from the same partition
  • Each event poller supports up to 10 concurrent invocations
  • Partition ordering isn't preserved, so it's not for order-dependent workloads
  • Your Kafka cluster must have share groups enabled, such as Apache Kafka 4.2 or later

To enable it, create a self-managed Kafka ESM with ConsumptionMode set to Queue. You can do this through the AWS Management Console, Lambda API, AWS CLI, AWS CloudFormation, AWS SAM, or AWS CDK.

Standard Lambda pricing applies. If you use provisioned mode for your self-managed Kafka ESM, you also pay for Event Poller Unit (EPU) usage. It's available in all AWS commercial Regions where self-managed Kafka ESM is supported.

Wrap-up

  • Self-managed Kafka ESMs support queue mode, based on Kafka share groups
  • Process in parallel beyond partition count
  • Up to 10 concurrent invocations per event poller
  • Ordering isn't guaranteed, so use it where order doesn't matter
  • Enable it by setting ConsumptionMode to Queue

This is for anyone who has struggled with over-partitioning in Kafka, or wants faster event-driven processing!

Related posts