How OpenAI's Pando Cut Median Git Push Latency by Nearly 5x

Hi everyone, it's Shii-chan! Today we're looking behind the scenes at OpenAI engineering: how they scaled Git. It's a meaty one, so let's go step by step!

OpenAI News openai.com

What was announced?

In OpenAI News, OpenAI introduced Pando, a new Git backend for its monorepo.

Pando stays compatible with the Git protocol and client, and it now powers all development in OpenAI's monorepo. Engineers and agents adopted it without changing how they use Git.

It has two goals:

• Horizontally scalable writes, optimized for concurrent pushes to many branches
• Maximum read throughput, so clients can download data fast enough to saturate their network bandwidth

In production, Pando cut median push latency by nearly 5x, and it now delivers nearly a petabyte of Git data per day.

Why it matters

Codex has changed the scale and pace of development at OpenAI. There are more branches, pushes, and merges, and more fetches and clones from development environments, research jobs, and CI. One engineer now has several changes ready for CI at once, and each push can set several jobs fetching the same commit, so the load compounds.

Starting around the end of last year, the systems behind the monorepo struggled. Fetches and pushes were slow, merges failed, and reliability at times fell below 90%.

OpenAI asked whether this was a limit of Git itself or of how the monorepo was served. It chose to keep the familiar Git workflow and rebuild the backend.

What changes

Inside OpenAI, the post says:

• Greater parallelism and better merge reliability
• Less time to publish code changes
• Median push latency down by nearly 80% (nearly 5x)

There is a benefit outside OpenAI too. Improvements found while building Pando are being contributed upstream to Git, and some are already merged. OpenAI is also exploring ways to share the technology behind Pando with users.

Dive Deep

This is the technical core!

Two assumptions relaxed

Pando relaxes two things most Git services treat as requirements:

• Repository-wide consistency: ordering all ref updates together makes pushes to unrelated branches contend on a single point
• A unique packfile per client: generating a pack for each client with a different local history costs CPU and disk I/O

Instead, it uses causal consistency to scale writes and reuses packfiles to scale reads.

How a push works

• The pushed packfile is validated while streaming to blob storage, without unpacking it on the server
• A multi-key transaction records a ref version in the metadata store and atomically updates the ref pointer
• This gives read-your-own-write consistency, and a write needs O(1) metadata reads and writes
• Clients conflict only when pushing to the same ref at the same time; adding servers, metadata store units, or blob stores raises write throughput

There is a trade-off: a client reading several branches cannot assume they describe a single point in time. OpenAI's view is that in practice users and agents depend on the order of updates to a single ref, not a global order.

Advertising fewer refs

A conventional Git server shows every branch it has. For a new feature branch, Pando advertises only a designated trunk branch as the base.

• With millions of branches, the ref advertisement alone can exceed 100 MiB
• Advertising less lets Pando track the pack's dependencies and serve the same pack in a later fetch
• The cost is larger pushes when you push to one branch repeatedly or stack branches
• In OpenAI's monorepo, the median packfile this way was under 20 KiB and the 99th percentile was under 3 MiB

Fetch and clone

Pando maps the client's haves and wants to ref versions and selects stored packfiles, with no recompression.

• It uses the Git protocol's packfile-uri feature to return links to packs, sending an empty packfile inline
• Packs are served by a delivery service called Express from regional caches, routed by consistent hashing, with cache hit rates over 99%
• Standard cloud VMs were enough for clone and fetch to saturate 10 Gbps links
• Fetch planning needs O(log Δ) metadata reads, where Δ is the shortest distance in ref versions between what the client has and what it wants

Compaction

In a fast-moving repository, sending a packfile URI per push would mean tens of thousands of packfiles a day. So Pando uses a design similar to LSM-tree compaction.

• Each push produces an L0 packfile, and background workers group them into covers that grow geometrically (the monorepo uses up to L4)
• Covers improve compression and act as a skip list over ref versions, so Pando can select packs in logarithmic time
• Unlike many Git servers, packs at every level are kept and used to serve clients

Building and rollout

• Built with Codex from kick-off to production in 10 weeks; one engineer designed the core, teams of agents implemented it, and a handful of engineers deployed it. Production traffic began in April
• Written in Rust with no legacy dependencies, and tested with a conformance suite against standard Git behavior plus deterministic simulation testing to find races and regressions
• Using GPT-6 Astra, agents wrote Lean and TLA+ to formalize a theory of Pando, which was used to find bugs and prove properties
• It was first deployed as a read- and write-through proxy to the existing system, so traffic moved over without downtime, and Codex skills helped developers migrate
• Pando has served more than 57 TB of Git packs in a single hour, with over 99% of those bytes from local disk cache

Contributing back to Git

• Fixed edge cases in concurrent downloads, cleanup after interrupted transfers, and resuming completed downloads, and sent them upstream
• Added tracing for packfile-uri downloads and byte counts for pack generation
• Added explicit haves, so clients can use --force-with-lease to check a feature branch has not moved and build its pack against trunk alone (available in OpenAI's Git build)

Wrap-up

• Pando is a new Git-compatible backend that now powers OpenAI's whole monorepo
• Causal consistency with independent per-branch updates and packfile reuse scale both writes and reads
• Median push latency fell by nearly 5x, and delivery is nearly a petabyte of Git data per day
• Improvements are going upstream to Git, and OpenAI is looking at ways to share the technology

If you are thinking about large repositories or infrastructure for agent-scale development, this is a solid read!