When DNSSEC Broke and All of .de Went Dark: How 1.1.1.1 Held the Line!

Hi everyone, Shii-chan here! Today I've got a story where the very foundation of the Internet gave a real wobble. Germany's .de domain briefly became unreachable across the board, and I want to peek behind the scenes with you!

Cloudflare Blog blog.cloudflare.com

What happened?

On May 5, 2026, at roughly 19:30 UTC, DENIC, the registry that runs Germany's .de ccTLD, published broken DNSSEC signatures for the .de zone. Any resolver that validates DNSSEC is required by the spec to reject responses like that and return SERVFAIL, and Cloudflare's public resolver 1.1.1.1 was no exception.

.de is one of the largest TLDs on the Internet, so an outage this high up the DNS hierarchy can make millions of domains unreachable at once. This is an incident report from Cloudflare's Blog, walking through what they saw that day and the temporary mitigations they applied until DENIC fixed things.

So what is DNSSEC?

DNSSEC attaches a digital signature called an RRSIG to each set of records, cryptographically proving that an answer hasn't been tampered with. It's about integrity, not encryption, and because the signatures travel alongside the records, a response stays verifiable no matter how many caches it passes through.

DNSSEC is built on a chain of trust: root trusts .de, .de trusts example.de, with each parent handing trust to its child through DS records. If any single link breaks, everything below it fails validation, which is exactly why a misconfigured TLD like .de takes all of its domains down with it. This time the trigger was a key rollover that ended up publishing signatures resolvers couldn't verify.

How did users feel it?

SERVFAIL spiked at 19:30 and then climbed for about three hours as cached records slowly expired. Because failing clients tend to retry three or more times, the raw numbers look scarier than the real user impact.

The interesting part is that the NOERROR rate stayed fairly stable. That's thanks to "serve stale," formalized in RFC 8767: when upstream resolution fails, a resolver can keep serving expired cache entries past their TTL. Plenty of .de records cached before the incident quietly kept users online.

Dive Deep

The main lever was the Negative Trust Anchor (NTA). Defined in RFC 7646, an NTA tells the resolver to treat a specific zone as unsigned and bypass validation. When a TLD ships broken signatures, every domain underneath returns SERVFAIL even though those domains are blameless, and once the cause is public and being fixed, returning SERVFAIL adds no security value. RFC 7646 even names TLD misconfiguration as its primary use case.

Behind 1.1.1.1 sits Cloudflare's own resolver, Big Pineapple, which also powers 1.1.1.1 for Families, Gateway DNS, DNS Firewall, and more. It has no native NTA yet, so the team used an existing override mechanism to mark .de as an insecure zone for the same effect. Turning off DNSSEC leaves .de open to genuine attacks for a while, but that was acceptable here because the signing failure was widespread, publicly confirmed, and hit every validating resolver equally. As it was put in the internal incident room:

There is no user of 1.1.1.1 resolving a .de name right now who would prefer a SERVFAIL over an unvalidated response.

The mitigation rolled out at 22:17 UTC, ending the impact for 1.1.1.1, and they shared it with fellow operators in the DNS-OARC Mattermost. They applied the same NTA on the internal resolver used for CDN origin resolution too, restoring connectivity for customers with .de origins.

There was one honest miss as well. Each SERVFAIL carries an Extended DNS Error (EDE) code from RFC 8914, and while it should have returned EDE 6 (DNSSEC Bogus), 1.1.1.1 returned EDE 22 (No Reachable Authority). It's a bug where the Bogus code from the trust-chain verifier never makes it into the response, and they've said they'll fix it.

DENIC also posted a short note explaining that a routine, scheduled key rollover generated and distributed non-validatable signatures, and that future rollovers are suspended until the exact cause is found.

Wrap-up

• DENIC's broken DNSSEC signatures made .de fail broadly starting 19:30 UTC on May 5
• 1.1.1.1 absorbed much of the pain with serve stale (RFC 8767) and recovered at 22:17 UTC via an NTA-equivalent that marked .de insecure
• The root cause was a TLD-side key rollover, not a defeat for DNSSEC itself: any misconfigured technology can break
• Cloudflare openly shared its own EDE-code mix-up bug too, which is very on-brand

If you run DNS resolvers or authoritative servers, work with DNSSEC in production, or just love seeing the real decision-making inside an incident, this one is for you!