Amazon ECS now lets you use just one-eighth of a GPU!

Hi, it's Shii! Today I found a container update from AWS worth sharing. If you've ever wanted a GPU but felt like a full one was overkill, this is good news for you!

AWS What's New aws.amazon.com

What was announced?

Over on AWS What's New, AWS announced that Amazon ECS now supports fractional GPU scheduling with Amazon EC2 G6f instances. You can now request GPU partitions as small as one-eighth of an NVIDIA L4 Tensor Core GPU, with 3 GB of GPU memory, and assign them to your tasks.

Setting it up is simple: just set GPU=0.125, GPU=0.25, or GPU=0.5 in the container definition of your ECS task definition. Amazon ECS then automatically places the task on a G6f instance that can satisfy the request.

The story so far

Until now, ECS could already run GPU workloads, but a task using a GPU generally had to be assigned an entire GPU on the instance. Even for workloads that don't need a full GPU, like small-model inference or quick model experimentation, you'd end up reserving a whole GPU instance for a single task, which tended to drive infrastructure costs up.

What changes

With fractional GPU support, you can now size GPU allocations to match the actual needs of workloads such as:

• Small-model AI inference
• Model experimentation and validation
• Graphics rendering
• Other workloads that don't require a full GPU

Because you can right-size GPU allocation per task instead of provisioning a full GPU instance every time, it becomes easier to reduce infrastructure costs.

Fractional GPU configuration works on both Amazon ECS Managed Instances and Amazon ECS on EC2. ECS Managed Instances in particular is a fully managed option where AWS handles instance provisioning, scaling, patching, and lifecycle management, which is great for teams that want to focus on their models rather than GPU infrastructure operations.

Dive Deep

ECS Managed Instances also comes with a few capabilities built specifically for accelerated workloads:

• GPU metrics visibility through Amazon CloudWatch Container Insights
• Automatic health monitoring that detects GPU hardware failures and replaces unhealthy instances

This fractional GPU capability is available in every AWS Region where Amazon EC2 G6f instances are available. You can get started through the AWS Management Console, AWS CLI, AWS SDKs, AWS CloudFormation, or other Infrastructure as Code tools.

Wrap-up

• Amazon ECS now supports fractional GPU scheduling on Amazon EC2 G6f instances
• Just set GPU=0.125, GPU=0.25, or GPU=0.5 in your container definition to request a GPU partition
• The smallest unit is one-eighth of an NVIDIA L4 Tensor Core GPU (3 GB of GPU memory)
• Works on both ECS Managed Instances and ECS on EC2
• ECS Managed Instances adds CloudWatch Container Insights GPU metrics and automatic health monitoring
• Available in every Region where G6f instances are available

For teams that have been over-provisioning full GPU instances for small inference jobs or model experiments, this is a welcome update for keeping GPU workloads cost-effective!