What If OpenAI Models Go Down? A Guide to Designing Failover Across Providers

Hi, it's Shii-chan!

OpenAI Developers developers.openai.com

Today I found a post about what to do when a production AI workload is about to fail, and how to move it to another route. It's a quiet topic, but it matters a lot for anyone running these systems!

What was announced?

The OpenAI Developers blog published a guide to designing failover across providers that serve OpenAI models.

It describes two routes:

• 1P (first-party): calling the OpenAI API directly
• 3P (third-party): reaching OpenAI models through a cloud provider such as Amazon Bedrock or Azure OpenAI

The running example is a read-only fraud assessment step inside a payments workflow.

Why it matters

It's tempting to think "if it fails, just send it somewhere else." But the guide says that failover is a decision made per workflow, not just a platform feature.

You need to know whether replaying the same operation is safe, and whether the second route has the features and data controls you need. Switching automatically without checking can cause duplicate actions or lower quality.

What changes

With this guide, an operations team can think like this:

• Start only with a self-contained, repeatable, read-only inference step
• Leave stateful Agents API sessions out of scope, because they are hard to recover
• Treat approval, testing, and enablement as separate steps, and keep failover disabled until it is validated

Instead of making everything redundant at once, you widen coverage gradually from the places where recovery is safe!

Dive Deep

Architecture

A platform-operated router (gateway) enforces route-specific rules. Normal traffic goes to the 1P primary, and only eligible recovery traffic goes to an approved 3P secondary. If no route qualifies, the request is queued, degraded, handed off, or rejected.

Recovery context, reconciliation of actions already taken, checkpoints, and duplicate-action protection belong to the application.

Private connectivity

• 1P uses OpenAI Private Link, which runs on an Azure Private Endpoint. Workloads on AWS or Google Cloud need customer-managed connectivity into Azure
• 3P options include AWS PrivateLink interface VPC endpoints for Bedrock and Azure private endpoints for Azure OpenAI
• Private Link does not switch regions automatically, so regional recovery needs separate setup
• Public HTTPS is an explicit option and never turns on automatically

Which failures to fail over on

The guide's decision table looks like this:

• Transient service, connection, or timeout failures: consider failover
• Sustained throttling or slow responses: consider it only if waiting would breach the deadline, and separate throttling from quota or billing problems
• Invalid requests, 401/403, refusals, exhausted quota or credits: do not switch routes; fix or escalate. Unclassified errors should not trigger failover either
• Partial output or uncertain tool actions: do not replay blindly; reconcile outcomes and prevent duplicate writes first
• Model-specific unavailability: try another approved model first, and consider provider failover only if the shared serving path is affected

Retries and circuit breakers

• Check error details, not just the HTTP status
• Use a circuit breaker to pause new requests to an affected route, tuned per provider, model, region, and workflow
• Use one retry controller across the app, SDK, and gateway with a shared retry budget. Respect Retry-After, and otherwise use bounded exponential backoff with jitter
• Cap failover traffic to protect the secondary, and restore the primary gradually

Validation and the Codex prompt

Before enabling failover, you should evaluate both routes against acceptance criteria, review data handling (logging, retention, encryption, key control) with the right owners, and load-test the secondary. A private connection alone is not enough.

The post also includes a Codex prompt for adding disabled-by-default failover to an existing gateway. It asks for a total attempt limit, deadlines, and a single accepted result per logical request. It is a template to adapt, not finished configuration.

The post gives no concrete thresholds, timeouts, or retry counts. Each team sets its own.

Wrap-up

• Connect the 1P (direct OpenAI) and 3P (Bedrock / Azure OpenAI) routes through per-workflow decisions
• Start with repeatable, read-only inference steps
• Fail over only on transient failures, not on 4xx errors, refusals, or uncertain tool actions
• Manage retries in one place, with circuit breakers and gradual failback
• Keep failover disabled until it has been validated

This one is for platform and operations folks who run AI in production and want to design what happens when it stops!