# SageMaker HyperPod splits LLM prefill and decode with DPD!

Hi everyone, it's Shiichan! Today I found a nicely nerdy update that makes LLM inference steadier.

## What was announced?

From AWS's What's New: Amazon SageMaker HyperPod now supports **Disaggregated Prefill and Decode (DPD)**. It splits large language model (LLM) inference into its two phases — **prefill** (reading the prompt) and **decode** (generating tokens one by one) — and runs each on its own dedicated GPU pool. The key-value (KV) cache is transferred between the pools over Elastic Fabric Adapter (EFA) using GPU-Direct RDMA.

## The story so far

Until now, prefill and decode shared the same GPU. So a single long-context request could stall token generation for every other request running at the same time, and you often had to over-provision one phase just to protect the other. For workloads like chat assistants, agentic pipelines, RAG, and long-document analysis — where you want consistent per-token latency — that was a real headache.

## What changes

With DPD, compute-bound prefill and memory-bandwidth-bound decode stop competing for the same resources. That gives you:

- more consistent per-token latency under sustained concurrency
- higher goodput at strict latency SLOs
- the ability to scale prefill and decode capacity independently

There's also an intelligent router that sends only long-context requests through the disaggregated path and routes shorter prompts straight to the decoder, so you don't pay transfer overhead on short prompts.

## Dive Deep

Turning it on is simple: you add a `pdSpec` section to the same `InferenceEndpointConfig` custom resource you already use for inference endpoints, and DPD runs on the HyperPod Inference Operator. It's composable with the existing KV cache offloading and intelligent routing features too.

It's available for SageMaker HyperPod clusters using the EKS orchestrator on EFA-capable instance types, in all AWS Regions where HyperPod is available. For the details, check the [Disaggregated Prefill and Decode for HyperPod inference docs](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod-model-deployment-dpd.html).

## Wrap-up

- SageMaker HyperPod now supports DPD, separating LLM prefill and decode
- The two phases stop fighting over GPUs, so latency stays steady and you can scale each independently
- The KV cache moves over GPU-Direct RDMA on EFA, and a smart router sends only long prompts down the disaggregated path
- Enable it by adding `pdSpec` to your `InferenceEndpointConfig`

This one is for teams serving LLMs in production at scale who are wrestling with latency variance and GPU over-provisioning!