First-token latency down up to 82%! SageMaker HyperPod gets a smart new Gateway

Hi, I'm Shii-chan! Today I found an AWS announcement clever enough to make me go "ooh" out loud, so let me walk you through it.

AWS What's New aws.amazon.com

What was announced?

AWS's What's New feed announced Amazon SageMaker HyperPod Inference Gateway, a Kubernetes-native routing system that's aware of GPU state. You deploy it on an existing SageMaker HyperPod cluster as an EKS managed add-on, and the best part is you don't need to change any application code at all.

The story so far

Until now, load balancing for LLM inference mostly relied on plain round-robin routing. It didn't care how busy each GPU pool was or which model's cache was already warm on which pod - it just cycled through servers in order. That meant requests could easily land on a congested pod while a free one sat idle nearby.

What changes

Inference Gateway replaces that "unintelligent" round-robin approach with real-time, inference-signal-driven routing. The impact is significant: in mixed-hardware and burst-traffic scenarios, it delivers:

• Up to 82% reduction in first-token latency
• 97-98% reduction in p99 time-to-first-token (TTFT)

That means you can use your GPU resources more efficiently while cutting the "waiting" your users actually feel. For engineers running LLM inference infrastructure, this is a quietly powerful upgrade.

Dive Deep

The Gateway is built from three core components:

• Envoy Endpoint: Terminates HTTPS traffic and exposes a single private endpoint per cluster
• Body-Based Router: Reads the model name directly out of each incoming request and routes it to the right GPU pool, so one endpoint URL can serve many models with zero client-side changes
• Endpoint Picker: Continuously scores every model server pod in real time across six inference-level signals to pick the optimal pod for each request:
  • KV cache utilization
  • Queue depth
  • LoRA adapter residency
  • Prefix cache hit rate
  • Predicted latency
  • Running requests

It works with any OpenAI-compatible model server, including vLLM and SGLang, again with no application code changes required. Per-cluster routing is available today in every AWS Region where the SageMaker HyperPod inference add-on is supported.

Looking ahead, AWS says the following are coming soon:

• Cross-cluster and cross-region routing via a centralized fleet gateway
• Global rate limiting
• Cost-tier-aware traffic shaping

Wrap-up

• SageMaker HyperPod Inference Gateway is a GPU-aware routing layer you can add to an existing cluster as an EKS managed add-on, with no application changes
• It's built from three components: the Envoy Endpoint, the Body-Based Router, and the Endpoint Picker, which scores every pod across six inference signals per request
• In mixed-hardware and burst scenarios, first-token latency drops by up to 82% and p99 TTFT drops by 97-98%
• It supports any OpenAI-compatible model server, including vLLM and SGLang
• Per-cluster routing is available now in supported Regions, with cross-cluster and cross-region routing coming soon

If you're running LLMs in production on SageMaker HyperPod and losing sleep over GPU utilization or latency, this update is squarely aimed at you!