SageMaker AI Inference Gets Up to 4.6x Faster with G7 Instances!

Hi, I'm Shii-chan! Today I've got news about a new instance option for SageMaker AI inference.

AWS What's New aws.amazon.com

What was announced?

According to AWS's What's New feed, Amazon SageMaker AI inference now supports G7 instances powered by NVIDIA RTX PRO 4500 Blackwell Server Edition GPUs. Compared to the previous-generation G6 instances, they can deliver up to 4.6x the AI inference performance.

The story so far

When you deploy generative AI models for production inference, you need high GPU throughput and memory capacity. With previous-generation instances, that often meant running into trouble:

• Over-provisioning expensive compute just to serve medium-to-large models cost-effectively
• Having to quantize models to fit within memory constraints

In short, there was a real tension between running your model properly and keeping costs (or accuracy) under control.

What changes

G7 instances are built to ease that tension. With more GPU memory capacity and higher memory bandwidth and throughput, they're well suited for:

• Serving models in the 7B–30B parameter range
• Image and video generation workloads
• Multi-model inference endpoints

The upshot: you can run bigger models in production more efficiently, without over-provisioning or quantizing them away.

Dive Deep

Here's how G7 compares to the previous-generation G6:

• GPU: NVIDIA RTX PRO 4500 Blackwell Server Edition with 5th Generation Tensor Cores
• GPU memory: 32 GB per GPU
• Networking: up to 700 Gbps of EFA-enabled networking (7x compared to G6)
• Local storage: up to 7.6 TB of NVMe SSD, for keeping large models close to compute

Instance sizes range from ml.g7.xlarge through ml.g7.48xlarge. You can deploy them from the SageMaker AI Inference console, API, or SDK simply by specifying the instance type in your endpoint configuration.

Current regional availability is:

• US East (N. Virginia, Ohio)
• US West (Oregon)

Region coverage is still limited for now, so check availability before you plan a deployment.

Wrap-up

• Amazon SageMaker AI inference now supports G7 instances with NVIDIA RTX PRO 4500 Blackwell Server Edition GPUs
• Up to 4.6x inference performance vs. G6, 32 GB of GPU memory per GPU, up to 700 Gbps EFA networking (7x), and up to 7.6 TB of NVMe SSD storage
• Well suited for 7B–30B parameter models, image/video generation, and multi-model inference endpoints
• Deployable by specifying ml.g7.xlarge through ml.g7.48xlarge in the console, API, or SDK
• Available now in US East (N. Virginia, Ohio) and US West (Oregon), 3 Regions total
• Great news for engineers running medium-to-large generative AI models in production