JA EN
shiichan

SageMaker Inference scales out up to 2x faster with automatic container image caching!

AWS What's New ★★★ View Markdown
#aws#ai#infrastructure

Hey there, it's Shiichan! Today I've got a nice little infrastructure improvement that makes generative AI inference feel a bit lighter on its feet.

AWS What's New aws.amazon.com

What was announced?

Over on AWS's What's New feed, Amazon SageMaker Inference now supports container image caching. It makes end-to-end scaling for generative AI models up to 2x faster during scale-out events, and the best part is that you don't have to change any of your settings.

The story so far

Generative AI workloads tend to use big container images, 10 GB or more, packed with deep learning frameworks and model serving code. Until now, every new instance launched during scale-out had to pull that whole image from Amazon ECR, which added several minutes of cold-start latency.

What changes

With container image caching, SageMaker pre-caches (pre-pulls) your image when your endpoint scales out. New instances launch with the container already available locally, so they can start serving traffic right away, up to 2x faster. That few-minutes wait just goes away.

Dive Deep

The nicest part is that you don't need to make any changes at all. The service automatically caches whatever image URI is specified in your endpoint or inference component configuration. It supports accelerator instance types, single-model endpoints, and inference component-based endpoints.

With this launch, SageMaker Inference now has a full scaling optimization suite:

  • sub-minute concurrency metrics for up to 6x faster load detection
  • instance-store container caching for faster scaling on existing instances
  • container image caching for up to 2x faster scaling on new instances

It's available in all AWS commercial regions where SageMaker Inference is supported. To learn more, check out the launch blog.

Wrap-up

  • SageMaker Inference adds container image caching, up to 2x faster scale-out on new instances
  • Solves the "pull a 10 GB+ image from ECR every time" problem by pre-pulling ahead of launch
  • No config changes needed; it caches your image URI automatically
  • Available in all AWS commercial regions where SageMaker Inference is supported
  • A welcome improvement if you run generative AI on autoscaling and wrestle with cold starts!

Related posts