Packet Path/

Akamai, July 22, 2021: A Safety Fallback That Silenced DNS

On July 22, 2021, nothing was attacked and no customer server failed. A load-balancing configuration directive — a routine file, updated roughly weekly — contained one formatting error. Akamai's load-balancing component did the safe-looking thing: it threw the bad file away and fell back to a minimal configuration. That fallback quietly lacked the data needed to answer DNS lookups for a large set of sites, so Akamai's authoritative DNS answered SERVFAIL instead. For about an hour, healthy websites were unreachable to anyone whose connection was not already open. Akamai's own detailed report is unusually honest about the second failure too: its deployment safety checks did raise alerts — they just were not configured to stop anything.

Case Studies · Intermediate · 12 min · November 21, 2026

Editorial illustration on a dark teal background: a large glowing DNS signpost in the centre with one broken direction arrow, healthy website servers glowing green behind it that nobody can reach, and small orange SERVFAIL warning tags hanging from the signpost.

The short answer

At 15:45 UTC on July 22, 2021, a software configuration update set off a disruption in Akamai's Secure Edge Content Delivery Network that made some customer websites unavailable for up to an hour. Akamai confirmed the same day that this was not a cyberattack. Its detailed report, updated on July 24, gave the real mechanism: a load-balancing configuration directive — load balancing is the job of spreading incoming work across servers so no one server is overloaded — contained a formatting error. As a safety measure, the load-balancing component discarded the bad directive and fell back to a minimal configuration. That fallback did not support load balancing for Enhanced TLS slots greater than 6145. With that data missing, Akamai's authoritative DNS system for the akamaiedge.net zone — DNS is the internet's phone book, and an authoritative server is the official source for a name's address — had no directive for how to answer queries for many of those slots, so it answered SERVFAIL, an error meaning "I cannot answer this." Rolling back the change, completed at 16:44 UTC, restored normal operations.

The company: what Akamai did

Akamai runs one of the world's largest content delivery networks, or CDNs — a web of servers spread around the world that sits in front of customer websites, delivering their pages from a nearby server and absorbing traffic the customer's own servers could never handle alone. If a big site loads quickly wherever you are, there is a good chance a CDN like Akamai's is the reason.

Two parts of that machine matter in this story. The first is the mapping and load-balancing system: the part that decides, for each incoming visitor, which Akamai server should handle them. It is driven by configuration directives — instruction files that tell the components how to behave — and Akamai's report says updates to the directive in this story were routinely made approximately weekly, as preparation work for a forthcoming product.

The second part is DNS. Before a browser can ask any Akamai server for a page, it must look up the name and get an address. Akamai's authoritative DNS for the akamaiedge.net zone answers those lookups — and it can only answer using directives the mapping system gives it. Why this matters: the servers holding the websites were healthy all along. The failure lived one step earlier, in the instructions for finding them.

The problem: a routine file, a quiet fallback, and checks that could not stop anything

Three quiet facts lined up before July 22.

First, the directive was ordinary. The load-balancing configuration directive that caused the outage was not an emergency change or an experiment. It was sent as preparation for independent load-balancing control of a forthcoming product, on a channel updated roughly weekly. Routine channels are exactly where a formatting error — the file written in a shape the receiving component does not accept — is most likely to slip through, because nobody treats a weekly file as dangerous.

Second, the safety fallback was incomplete. When the load-balancing component saw the improper configuration, it did what it was designed to do: it disregarded it and fell back to a minimal, VIP-only configuration. A VIP here is a virtual IP address — a shared front-door address that stands in for a group of servers. The problem was what the minimal state left out: it did not support load balancing for Enhanced TLS slots greater than 6145. TLS is the encryption behind the padlock in a browser; an Enhanced TLS slot is, in plain terms, one of the numbered serving slots Akamai used for customers' encrypted sites. Every slot above that number lost its load-balancing data the moment the fallback engaged — and nobody was told in a way that stopped the deployment.

Third, the safety checks alerted but did not act. Akamai's zoning process — the staged process that deploys configuration changes across its network — includes an alert check for potential issues, and that check did raise alerts during this deployment. But, in Akamai's own words, due to how that particular safety check was configured, the alerts did not prevent the configuration from continuing to propagate and did not result in escalation to engineering SMEs — the subject-matter experts who owned the component. A second check failed quietly too: the input safety check on the load-balancing component did not automatically roll back the change when it detected the error. An analogy: a smoke alarm wired to a light in an empty room. It works. It even turns on. It just never wakes anyone, and it cannot turn the stove off.

The mechanism: one bad file, one incomplete fallback STEP 1 — THE DIRECTIVE A weekly load-balancing directive contains a formatting error STEP 2 — THE FALLBACK Component discards it and falls back to a minimal VIP-only configuration STEP 3 — THE GAP No load balancing for Enhanced TLS slots greater than 6145 DNS GETS NO DIRECTIVE — AND ANSWERS SERVFAIL Visitors cannot look up healthy websites, so the sites look down Nothing was attacked and no website failed. The instructions for finding them did.

What this shows: the failure chain, animated. The fallback was a safety feature — it just silently dropped the data DNS needed, turning a bad file into an hour of failed lookups.

What happened on July 22, 2021

All times are UTC, from Akamai's blog summary of July 22 and its detailed incident report updated July 24, 2021.

Thursday, July 22, 2021 — about one hour 15:45 UTC — the disruption begins A configuration update triggers the failure; new connections to affected sites fail 16:44 UTC — rollback completed The mapping-component change is rolled back; the platform stabilises 18:30 UTC — Akamai confirms resolution Status update: not a cyberattack; DNS services back to normal operations Jul 23–24 — correction, then root cause Scope narrowed Jul 23; Jul 24 report names the formatting error and fallback and the alerts that raised but did not block

What this shows: the day's spine in Akamai's own times — under an hour of disruption, then two days of increasingly precise public explanation.

15:45 UTC — the update lands. A software configuration update triggered what Akamai's first-day summary called a bug in its Secure Edge CDN, impacting that network's DNS system. In the fuller July 24 account, the malformed directive propagated, the load-balancing component fell back to its minimal configuration, and authoritative DNS for akamaiedge.net began answering SERVFAIL for queries it now had no directive for.

Who felt it — and who did not. Akamai's status update is precise here: affected customer sites were significantly impacted for connections that were not established before the incident began. If your browser already had its connection open, it kept working — it no longer needed a fresh lookup. Everyone arriving new hit the missing phone-book entry first. That split is the signature of a DNS failure, and it is why reports in the first minutes of such incidents always sound contradictory: the site is simultaneously up and down, depending on when you arrived.

Why SERVFAIL, on purpose. The authoritative system answers SERVFAIL when it has no directive by design: during a small, localised failure, a SERVFAIL tells the asking resolver to retry an alternate authority, which may have the answer. That design is sensible — when the failure is local. On July 22 the missing directive was not local, so the polite "ask someone else" answer was given everywhere, and there was no someone else with the missing data.

The response was slowed by the incident itself. Akamai lists three contributing factors beyond the checks that did not block. The alert and procedure indicating widespread SERVFAILs possibly due to mapping-system issues did not lead to an appropriately urgent and timely response. The internal alert that was escalated pointed first at a separate component — the Edge DNS system rather than the mapping system — delaying troubleshooting of the component that had actually changed. And the Enhanced TLS impact affected Akamai staff access to internal tools and websites, delaying escalation, troubleshooting and, in Akamai's words, especially the initiation of the incident process. The outage had hold of the fire alarm panel as well as the building.

16:44 UTC — rollback completed. Teams identified that a change in a mapping component was causing the issue and rolled it back. Immediately following the rollback, Akamai reported, the platform stabilised and DNS services resumed normal operations. Total duration: approximately one hour.

Two checks fired. Neither one could stop anything. CHECK 1 — ZONING ALERTS Alerts DID raise during deployment But they did not block the change from continuing to propagate And did not escalate to the SMEs CHECK 2 — INPUT SAFETY CHECK The component detected the improper configuration But it did not automatically roll the change back DETECTION WITHOUT AUTHORITY IS NOT A CONTROL A check earns its name only if it can halt, roll back, or wake the owner Both facts are Akamai's own, from its July 24 root-cause update.

What this shows: the second failure inside the first. The monitoring worked as configured — the configuration of the monitoring is what failed.

The damage

The damage was availability, and only availability — but at CDN scale, availability is the product. For about an hour, some customer websites could not be reached by new visitors, through no fault of those customers' own systems. Akamai published no count of affected sites and no dollar figure in either primary source, and this case prints neither rather than borrowing numbers from coverage Akamai did not write.

Two honest boundaries belong in this section. First, the scope correction: Akamai's blog initially stated there was an impact to its DNS service, and its July 23 update corrected that — after further investigation, the impact was determined to be isolated to a DNS component of its secure edge content delivery network. This case uses the corrected scope throughout. Second, the mirror-image lesson of the Dyn case that ran yesterday in this series: in 2016 an attacker flooded a DNS provider from outside; in 2021 no attacker existed at all. A shared dependency does not care why it stops answering.

How they recovered

Step one: find the changed component. Despite the first escalated alert pointing at Edge DNS rather than the mapping system, Akamai's teams identified that a change made in a mapping component was causing the issue. That identification — which component changed, not just which service was failing — is what made a clean recovery possible.

Step two: roll the change back. The rollback of the configuration change was completed at 16:44 UTC, about an hour after the disruption began. Akamai's status update records the result plainly: immediately following the rollback, the platform stabilised and DNS services resumed normal operations, and the incident was resolved with monitoring continuing to ensure traffic remained stable.

Step three: freeze the channel. Akamai blocked any further changes to the involved configuration channel, and said other related channels were being reviewed and might be blocked the same way — channels would be unblocked only after additional safety measures were assessed and implemented where needed. The weekly file stopped being weekly until it could be trusted again.

Step four: correct the record in public, twice. On July 23, Akamai narrowed its own first-day scope claim. On July 24, it published the mechanism in detail — formatting error, fallback, the 6145 threshold, SERVFAIL by design, and the safety checks that alerted without blocking — including facts unflattering to itself. That sequence is why this incident can be taught from primary sources at all.

The fix: technology and controls that came out of it

Akamai's report lists its corrective actions directly, and they map one-to-one onto the failures above:

Make deployment safety checks able to block. First on Akamai's in-progress list: validate and strengthen the safety checks for the configuration deployment zoning process. The July 22 checks raised alerts that changed nothing; the fix is a check whose alert halts propagation and escalates to the component's own experts — authority, not just visibility.

Treat a SERVFAIL surge as a page, not a statistic. Akamai committed to increase the sensitivity and priority of alerting for high rates of SERVFAILs. Its own contributing-factor list admits the existing SERVFAIL alert and procedure did not produce an appropriately urgent response. A sudden rise in "we cannot answer" from an authoritative DNS service is a customer-facing outage in progress, and the alerting now had to say so.

Strengthen input checks on the components themselves. Longer term, Akamai said it was reviewing and improving input safety checks for mapping components — the layer that accepted a malformed directive, fell back silently, and never rolled back. A component that detects a bad input needs a defined next move: reject loudly, roll back automatically, or both — not a quiet downgrade to a configuration missing whole ranges of customers.

Audit the watchers. Finally, Akamai committed to auditing critical systems to identify gaps in monitoring and alerting, then closing unacceptable gaps. The deepest gap July 22 exposed was structural: the alert that escalated named the wrong component, and the incident degraded the internal tools the responders needed. An audit that only asks "do we monitor this?" misses both; the useful question is "when this fires at 3 a.m., who is woken, what can they stop, and will the tools they need still work?"

Keep incident tooling off the blast radius. Akamai does not list this as a separate action item, but its own contributing factors state the fact: staff access to internal tools and websites was hit by the same failure they were trying to fix, delaying the incident process itself. Any control plane, status tool or runbook that shares fate with the service it exists to rescue is a dependency to engineer around — with an out-of-band path arranged before it is needed.

What this means for you

Enterprise network engineers

Your version of this file is every routine change ticket marked low-risk because it is done weekly: the firewall rule push, the load-balancer config sync, the DNS zone transfer. July 22 asks two questions of each one. If the receiving system rejects the file, what exactly does it fall back to — and have you ever read that fallback's limits, the way slot 6145 was sitting in Akamai's? And if your deployment pipeline raises an alert mid-push, can that alert actually stop the push and wake the owner, or does it just get logged next to the outage it predicted? Test both answers before a formatting error does.

Hyperscaler engineers

You build the zoning processes and fallbacks in this story, so take the design lessons directly. A minimal fallback configuration is a product decision, not an error path: whatever it omits becomes an outage shape, so fallback coverage deserves the same review as the primary configuration — including an explicit list of what it cannot serve. Safety checks need teeth proportional to blast radius: a check guarding a channel that can silence DNS for many customers should block by default and require a human to override, not alert by default and require a human to notice. And route component alerts to the component that changed: Akamai's first escalated alert named the victim (Edge DNS), not the cause (the mapping system), and that misdirection cost troubleshooting time inside a one-hour incident.

Try it yourself

Akamai's customers learned the same lesson Dyn's customers learned in 2016: your availability is not your number alone — it is your number multiplied by every shared service you depend on. You can feel that maths with Packet Path's free SLA and Availability Calculator: enter 99.9% for your own service, then chain it behind a second 99.9% service it depends on, and watch the combined figure — and the monthly downtime budget — fall. One hour of a dependency's bad day is your bad day too, whether the cause was an attack, a bug, or one formatting error in a routine weekly file.

Sources

Primary source — Akamai, "Akamai Summarizes Service Disruption (RESOLVED)," Mani Sundaram, Executive Vice President and General Manager, Security Technology Group (July 22, 2021; updated July 23, 2021): the 15:45 UTC start, the up-to-an-hour duration, rollback restoring normal operations, the not-a-cyberattack confirmation, and the July 23 correction narrowing the impact to a DNS component of Akamai's secure edge content delivery network: https://www.akamai.com/blog/news/akamai-summarizes-service-disruption-resolved

Akamai detailed incident report for Edge DNS / GTM — status update of 18:30 UTC, July 22, 2021 (disruption began 15:45 UTC, lasted approximately one hour, established connections unaffected, mapping-component change rolled back at approximately 16:44 UTC, platform stabilised) and root-cause update of 07:35 UTC, July 24, 2021 (the formatting error, the minimal VIP-only fallback, Enhanced TLS slots greater than 6145, SERVFAIL by design, the zoning and input safety-check failures, the contributing factors, and the corrective actions). The live status page no longer carries the 2021 incident; the full text of both Akamai updates was read for this case in the NANOG mailing-list archive, which reproduces them verbatim: https://seclists.org/nanog/2021/Jul/207

Scope and attribution note: Akamai's primary sources name no affected customer, publish no count of affected sites and no dollar damage figure — this case claims none of those. Duration is given as Akamai stated it (up to / approximately one hour, 15:45 to 16:44 UTC); longer durations in secondary coverage are not used.

Key takeaways

Keep reading