We analyzed core dumps like an epidemiologist and found an 18-year-old libunwind bug that had been hiding all along!

Hey there, it's Shii! Today I want to share a debugging story that OpenAI published, and it reads just like a mystery novel. The title might look a little plain, but once I read it I found it super exciting, so I couldn't wait to bring it to you!

OpenAI News openai.com

What was announced?

On June 30, 2026, OpenAI published an article that walks through how they tracked down and fixed a long-standing, mysterious crash in their internal data infrastructure. The setting is "Rockset," the real-time data platform that OpenAI acquired in 2024. It's a pretty important system that powers things like ChatGPT's search and data plugin features behind the scenes.

This crash had an eerie symptom: a C++ program would, for some reason, jump execution to a corrupted address or a NULL address while running. On top of that, it happened very rarely, so they couldn't reproduce it on demand. At first, the engineers apparently treated it as "one mystery bug."

So the team decided not to chase each individual crash one at a time. Instead, much like an epidemiologist investigating an outbreak of a disease, they built an automated analysis pipeline that looked at every core dump Rockset had generated over the past year. And here's a very OpenAI-like detail: part of that pipeline was actually written by ChatGPT itself.

The story so far

Up to now, infrastructure debugging has basically followed the style of "when a crash happens, a human reads through that one core dump." But with a bug like this one that happens rarely and can't be reproduced, going through them one by one makes it hard to tell whether you're looking at "different variations of the same bug" or "several unrelated bugs." In fact, in this case people initially assumed there was a single cause, and that turned out to be one reason the investigation dragged on.

What changes

They built a pipeline to download, analyze, and classify a huge number of core dumps all at once, extracting things like register information and looking at them statistically as a population. And it turned out that "the one bug they thought they had" was actually two separate clusters of crashes caused by two unrelated things. A correlation that was invisible when chased individually popped out clearly the moment they looked at the whole population. That's the highlight of this story.

As a result, they could now properly separate the causes of Rockset's crashes and deal with each one. And it didn't stop there: the bug in libunwind, an open-source library widely used across the industry that was one of the causes, wasn't kept inside OpenAI. They fed it back upstream (to the original project). That could benefit other high-throughput C++ services that use the same library, too.

Dive Deep

From here, let me dig in properly for all of you tech lovers.

There were two causes they found:

1. A hardware failure: A hard-to-detect, silent hardware corruption occurring on one specific physical Azure host. It seems to have been the kind of failure where the CPU's computations themselves went wrong.
2. A race condition in GNU libunwind: They found a race condition in "GNU libunwind," an open-source library widely used across the industry to unwind the stack during C++ exception handling. The race condition had existed for over 18 years, all the way back to the earliest version that supported C++ exception unwinding for x86_64.

According to several international outlets, this race condition works like this: there's a tiny gap between the instruction that updates the stack pointer and the instruction that reads the return instruction pointer, and if a signal interrupt lands in exactly that window, the instruction pointer that gets restored ends up corrupted. There's also an explanation that, because OpenAI's environment used a signal-based mechanism to measure CPU usage, this extremely rare race was more likely to surface. As a workaround, they switched Rockset's unwinder to a different implementation (libgcc's unwinder), and it's reported that they fed reproduction steps and a proposed fix back to the libunwind project upstream.

The reason they could dig this deep was precisely because they built an automated pipeline to download, analyze, and classify a full year's worth of core dumps as one large population, instead of chasing crashes one at a time. Filtering out known false positives (noise) to get a clean dataset, and then seeing the correlation snap into focus, is an approach with a lot of general applicability as a debugging method for large-scale infrastructure, I think.

Wrap-up

Today I shared the story of how OpenAI tracked down Rockset's mysterious crashes through an "epidemiology"-style population analysis of core dumps. The fact that what looked like one bug was actually two totally unrelated causes, "a hardware failure on a single Azure host" and "an 18-year-old race condition in libunwind," honestly surprised me too. And I felt it was really sincere of them to properly give that fix back to the upstream of an industry-standard library. Infrastructure debugging stories might look plain on the surface, but the way a single shift in perspective can crack a years-old mystery is truly full of romance, isn't it?

Here's the original article: Core dump epidemiology: fixing an 18-year-old bug