SageMaker AI Inference Adds GPU Partitioning: Up to 7 Models on One GPU

Hi everyone, it's Shii-chan! Here's a way to waste fewer GPUs!

AWS What's New aws.amazon.com

What was announced?

AWS What's New announced that Amazon SageMaker AI inference now supports GPU partitioning for real-time endpoints.

Using NVIDIA Multi-Instance GPU (MIG) technology, it divides a GPU into up to seven isolated partitions, each with its own memory and compute.

The story so far

Even for small and mid-size models such as fine-tuned variants, embedding models, rerankers and classifiers, each model needed a whole GPU. An instance hosted no more models than it had GPUs, and you paid for GPU capacity those models could never use.

What changes

You choose how to divide the GPUs behind your endpoint and place each model on a partition sized to fit it.

• Several models share one GPU
• Hardware-enforced isolation keeps a busy model from taking memory or throughput from its neighbors
• Right-sizing opens up newer instance generations that would be too big for a single model

You still invoke each model by name, so your application code doesn't change.

Dive Deep

• Setup: configure GPU partitioning when you create an endpoint that hosts inference components
• Instances: P4d, P5, G7 and G7e
• Regions:
  • US East (N. Virginia)
  • US East (Ohio)
  • US West (Oregon)
  • Asia Pacific (Singapore)
  • Asia Pacific (Tokyo)
  • Europe (Frankfurt)
  • Europe (Ireland)

See the accelerator partitioning section of the SageMaker AI Developer Guide for details.

Wrap-up

• MIG splits one GPU into up to seven partitions
• Small and mid-size models can share a GPU with hardware isolation
• No application code changes
• Available on P4d / P5 / G7 / G7e in seven Regions

Great if you run many small models and want to cut GPU costs!