OpenAI's Custom Chip Jalapeño Hits Up to 1.9x the Throughput per Watt!

Hi everyone, it's Shii-chan! Today I want to share some numbers from OpenAI that genuinely surprised me — their own inference chip is now running, and the first real results are in.

OpenAI News openai.com

What was announced?

This comes from OpenAI's News. OpenAI shared the first real test results for Jalapeño, the custom inference chip it designed in-house. Normally, hardware makes you trade off throughput against latency, but Jalapeño delivers both at once with a single architecture.

The tests covered three models: GPT-OSS 120B, DeepSeek R1 670B, and Kimi K2.5 1T — including models built outside OpenAI, not just its own.

Why it matters

Power and latency are the two things that matter most for AI inference infrastructure. Agents especially chain together many steps, so any delay in a single step compounds across the whole task.

This is OpenAI showing off its full-stack advantage: designing models, products, serving software, chips, memory, networking, and systems all together. Jalapeño is the first piece of that story to come with real, measured evidence behind it.

What changes

Across all three models, Jalapeño delivered:

• 1.5 to 1.9x more throughput per watt at peak
• 1.7 to 3.6x lower end-to-end latency
• 2.1 to 4.1x higher performance on highly interactive workloads

In practice, that means faster responses, snappier agents, and more reliable access as demand grows. For OpenAI, it also means getting more useful work out of the same power and hardware, which helps keep costs down as capable AI reaches more people.

Dive Deep

The results come from InferenceX, a public benchmark from SemiAnalysis that measures the entire process of serving a request, not just raw chip specs. OpenAI also compares performance per watt rather than per chip. Jalapeño is rated at 700 watts, but measured sustained power stayed at or below 550 watts on the tested workloads.

Here's the model-by-model breakdown:

• GPT-OSS 120B (vs. GB200 at 1,200 W): about 1.9x peak throughput per kW (85,448 vs. 44,960 mixed/kW), about 1.7x lower end-to-end latency (1.03s vs. 1.80s)
• DeepSeek R1 670B (vs. GB300 at 1,400 W): about 1.7x peak throughput per kW (19,641 vs. 11,781 mixed/kW), about 3.6x lower latency (1.65s vs. 5.99s)
• Kimi K2.5 1T (vs. GB300 at 1,400 W): about 1.5x peak throughput per kW (18,195 vs. 11,862 mixed/kW), about 3.4x lower latency (1.56s vs. 5.31s)

The architecture is the interesting part. Prefill (processing the prompt) is compute-bound, while decode (generating tokens one at a time) is bound by memory bandwidth — two very different bottlenecks. Jalapeño's chip, memory, network, software, and rack-scale system were all designed together to excel at both, keeping the KV cache and other model state explicitly placed and local to minimize data movement and communication delays.

AI helped build the chip itself, taking it from initial design to tapeout in just nine months. Jalapeño was also designed to be easy for AI to program: engineers describe work as local tensors, explicit communication, and predictable synchronization, and AI optimizes how that work gets mapped, placed, and scheduled. Using Codex with GPT-Astra, the team brought three open-weight models that weren't part of the original plan to full performance in two months. For selected GPT-OSS attention and mixture-of-experts blocks, AI-generated implementations ran 1.5 to 1.8x faster than the existing human-written code (that figure applies to those specific blocks, not the whole model).

OpenAI plans to start deploying Jalapeño within its own infrastructure by the end of the year. This is generation one of a multigenerational roadmap — generation two is already deep in development, and generation three is taking shape. OpenAI says it will keep deploying NVIDIA and other partners' accelerators widely too; this isn't a full replacement.

Wrap-up

• OpenAI shared the first measured results for its custom inference chip, Jalapeño
• Across GPT-OSS 120B, DeepSeek R1 670B, and Kimi K2.5 1T, it hit 1.5 to 1.9x more throughput per watt and 1.7 to 3.6x lower latency
• The architecture is built to excel at both prefill and decode, minimizing data movement across the whole stack
• AI helped design the chip itself, cutting design-to-tapeout time to nine months
• Deployment inside OpenAI's own infrastructure starts by year's end, as generation one of a multi-generation roadmap

If you're curious how AI infrastructure scales and gets more efficient behind the scenes, this one's worth a read.