Fastly, June 8, 2021: One Valid Setting Woke a Sleeping Bug
On May 12, 2021, Fastly began deploying software that carried a hidden bug. For almost four weeks nothing happened. Early on June 8, one customer pushed a completely valid configuration change — the kind customers are supposed to be able to push — and it happened to include the exact circumstances the bug needed. In minutes, 85 percent of Fastly's network was returning errors. Fastly detected the disruption within one minute, found the triggering configuration in 40 minutes, and had 95 percent of its network operating as normal within 49 minutes. This is the documented story, from Fastly's own summary published the same day.
The short answer
On June 8, 2021, Fastly — a company whose computers sit between websites and their visitors to make those sites fast — suffered a global outage (a failure across its whole worldwide network at once). The cause, in Fastly's own words in its summary published that day, was an undiscovered software bug that surfaced when it was triggered by a valid customer configuration change. The bug had been introduced earlier, in a software deployment that began on May 12. It could only be triggered by a specific customer configuration under specific circumstances — so it waited, hidden, for almost four weeks. Early on June 8, a customer pushed a configuration change that included those circumstances. The result was that 85 percent of Fastly's network returned errors. Fastly detected the disruption within one minute, identified and isolated the cause, and disabled the configuration. Within 49 minutes, 95 percent of its network was operating as normal. A permanent fix for the bug began deploying later that day, at 17:25 UTC. Fastly's closing line in the summary is the honest one: even though specific conditions triggered the outage, it should have anticipated it.
The company: what Fastly did
Fastly runs an edge cloud platform. In its own description, that platform helps popular digital businesses keep up with what their customers expect by delivering fast, secure, and scalable online experiences. Two words in that description do most of the work, so here they are in plain English.
An edge platform puts copies of a website's content and logic on computers spread around the world, close to visitors — at the edge of the network, near the user — instead of in one distant data centre. When you open a site that uses a service like Fastly, you are often talking to a nearby edge computer, not to the site's home servers. That closeness is what makes pages load quickly, and it is why the industry word for this kind of service is a content delivery network, or CDN: a network whose job is to deliver content from nearby.
A configuration is the set of settings that tells that edge network how to handle a particular customer's traffic: what to cache (keep a local copy of), where to fetch fresh copies from, and what rules to apply on the way through. Fastly customers can change their own configuration — that self-service speed is a feature, not a flaw. A customer pushing a new configuration is ordinary, expected, everyday work.
Why this matters: when an edge network fails, it does not fail for one website in one building. Its customers' visitors feel it in many places at once, because the edge is the front door for all of them. Fastly calls the June 8 event, in its first paragraph, broad and severe — and the figure that follows explains why: 85 percent of the network returning errors is not a single broken machine. It is most of the front doors, failing together.
The problem: a bug that needed one exact key
The setup for this outage has three parts, and Fastly's summary gives all three plainly. The lesson is in how they fit together.
First, the bug was shipped weeks earlier, and nobody knew it was there. On May 12, Fastly began a software deployment — installing a new version of its own software across its network, the routine way platforms improve themselves. That deployment introduced a bug: a flaw in the code. Fastly calls it undiscovered, which is the important word. It was not a known risk accepted for a while. It was present, and testing had not found it.
Second, the bug could not fire on its own. It needed a trigger: a specific customer configuration, under specific circumstances. Think of it as a lock that only one unusual key shape can open. For almost four weeks, customers pushed ordinary configurations — ordinary keys — and nothing happened. The software ran. The network ran. Every normal day was also, unknowingly, a day the bug survived another test it was never given on purpose.
Third, the trigger, when it came, was completely legitimate. Early on June 8, a customer pushed a valid configuration change — Fastly uses that word itself, and repeats the point at the end of its summary. The customer was using the system as designed. The change simply happened to include the specific circumstances the bug needed. Fastly does not name the customer in its summary, and it does not describe the specific configuration — this page does not guess at either. What the summary does say, in its conclusion, settles the blame question: even though there were specific conditions that triggered this outage, we should have anticipated it.
An analogy: imagine a hotel where a maintenance crew, in May, quietly installs a new kind of door lock on most floors — and one lock has a hidden flaw that makes it jam if a guest uses a perfectly ordinary key card in one exact way. For weeks, hundreds of guests use their cards in other ways, and the flaw sleeps. Then one morning a guest uses their valid card in that one exact way, and the flaw does not jam one door. Because the locks are connected, it jams most doors in the building at once. The guest did nothing wrong. The flaw was installed by the hotel, weeks earlier, and the hotel's job was to have caught it before a guest could find it by accident.
What this shows: the three-part setup from Fastly's summary, in order. The outage needed both halves — a shipped bug and a valid trigger — and only one of those halves was Fastly's customer.
What happened on June 8, 2021
All times are UTC, as given in Fastly's timeline. Keep two clocks in mind as you read: the fast clock of bulk recovery, and the slower clock of full resolution. Fastly's summary reports both, and this page keeps both, because collapsing them into one number would misstate the day in either direction.
What this shows: the full day on one line — a one-minute detection, a 40-minute hunt for the trigger, a 49-minute bulk recovery, and a longer tail to resolution and to the permanent fix.
09:47 — onset. The disruption begins globally. Behind it, as Fastly established, is the chain above: the May 12 bug, woken by the valid configuration change pushed early that day, causing 85 percent of the network to return errors. An error, here, means the edge answers a visitor's request with a failure instead of the site they asked for.
09:48 — detected within one minute. Fastly's monitoring identifies the global disruption one minute after onset. This is the strongest number in the whole summary, and it matters for the lessons: the failure was not slow detection. The alarms worked. What took time was the next step — diagnosis.
09:58 — the first status post. Eleven minutes after onset, Fastly publishes its first status update. In a global outage, that public clock matters: customers cannot route around a problem they cannot see confirmed, and a status post is the signal that the provider, not the customer's own systems, owns the incident.
10:27 — the trigger is found. Forty minutes after onset, Fastly Engineering identifies the customer configuration that triggered the bug. Note what this step actually is: among all the valid configuration changes customers push as normal work, engineers have to find the one change whose specific circumstances fit the bug — while the network is failing. Once identified, the cause could be isolated and the configuration disabled, which is the move that stops the bleeding: remove the key, and the faulty lock stops jamming new doors.
10:36 — recovery begins. Impacted services begin to recover, nine minutes after the configuration was identified. This is also the moment behind the headline figure: measured from the 09:47 onset, 95 percent of the network operating as normal within 49 minutes lands at about 10:36.
What this shows: why one customer's change could affect most of the network. Configurations are not kept in one place — they are distributed to the edge, which is the platform's strength in normal times and its blast radius in this one.
11:00 — majority recovered. By 11:00, the majority of services have recovered. The shape here is familiar in large incidents and worth naming plainly: most of the recovery happens in a rush once the trigger is disabled, and then a tail remains — the last services, the last locations, the checks that it is really over.
12:35 and 12:44 — mitigated, then resolved. The incident is mitigated at 12:35 and the status post is marked resolved at 12:44 — two hours and fifty-seven minutes after onset. Both numbers belong in the record alongside the 49-minute headline. The 49 minutes describes how fast the bulk of the network came back, which is genuinely fast. The 12:44 describes when Fastly could call the incident finished. A reader who is only given the first number has been given half the day.
17:25 — the permanent fix starts. Once the immediate effects were mitigated, Fastly turned to fixing the bug itself and communicating with customers. It created a permanent fix and began deploying it at 17:25. Note the two-layer recovery: disabling the configuration treated the symptom (the trigger), deploying the fix treated the cause (the bug). Both were needed — the first to restore service today, the second to make tomorrow's valid configurations safe again.
The damage
Fastly's summary is deliberately short on damage accounting, and this page follows it exactly rather than borrowing numbers from press coverage. What the primary source establishes is this.
The scale, in the source's own figure: 85 percent of the network returned errors. That is a share of Fastly's network — not, as some retellings put it, a share of the internet or of all websites. Precision matters here because the honest figure is already large: a platform whose job is to be the front door for many businesses had most of that network answering requests with errors at the same time.
The character, in the source's own words: broad and severe. Fastly opens its summary by saying the outage was broad and severe, and that it was truly sorry for the impact to its customers and everyone who relies on them. That last phrase — everyone who relies on them — is the real damage shape of an edge outage. The direct customers are the websites and services on the platform. Behind each of them are their own visitors and users, who experience the failure as a site that will not load, with no idea that a third company's bug, shipped in May, is the reason.
The duration, in both of the source's clocks. For most of the network, the practical outage was under an hour: 95 percent operating as normal within 49 minutes. For the incident as a whole, the clock ran from 09:47 to mitigation at 12:35 and resolution at 12:44. Both belong in any honest damage account, for opposite reasons: the first shows how effective the response was once the trigger was found, and the second shows that a global platform does not get to declare victory at 95 percent.
What the source does not do is put a dollar figure on the outage, name the affected customers, or name the triggering customer — and neither does this page. The measurable damage in the primary source is reach (85 percent of the network), severity (Fastly's own description), and time (49 minutes to bulk recovery, almost three hours to resolution).
How they recovered
Step one: detect, in one minute. Monitoring identified the disruption at 09:48, one minute after onset. Whatever else this case criticises, the detection layer did its job at a speed most organisations would accept on their best day.
Step two: communicate, in eleven minutes. The first status post was published at 09:58. Early, factual status communication does not shorten the technical recovery by a minute — and it is still part of recovery, because it lets every affected customer stop debugging their own systems and start planning around a known platform incident.
Step three: find the trigger, in forty minutes. By 10:27, engineering had identified the specific customer configuration that triggered the bug. This was the diagnostic heart of the incident: the fault was not a failed machine or a cut cable, but an interaction between code shipped in May and a valid input that arrived in June. Finding it meant connecting those two events across almost four weeks.
Step four: disable the trigger, and let the network come back. With the cause isolated, Fastly disabled the configuration. Services began recovering at 10:36, 95 percent of the network was operating as normal within 49 minutes of onset, and the majority of services had recovered by 11:00.
Step five: finish the tail, then fix the cause. Mitigation at 12:35 and resolution at 12:44 closed the incident. The permanent bug fix — created after the immediate effects were mitigated — began deploying at 17:25, the same day. Recovery, in other words, had a service phase (disable, recover, resolve) and a cause phase (fix the bug, deploy the fix), and Fastly's summary keeps them in that order.
The fix: technology and controls that came out of it
Fastly's summary lists its next steps under the heading "Where do we go from here?" They are commitments made on the day itself, and this page reports them as such — as the programme Fastly set itself, not as a later audit of whether each one landed.
Deploy the permanent fix, quickly and safely. The short-term commitment was to deploy the bug fix across the network as quickly and safely as possible. The tension in that phrase is real and worth noticing: after an outage caused by shipped code, the fix is also shipped code — so speed and safety have to be held together, with the fix itself passing the scrutiny the original deployment missed.
A complete post-mortem of process, not just of code. Fastly committed to a complete post-mortem (a structured review after an incident) of the processes and practices it followed during the incident. The distinction matters: the bug explains what broke, but only a process review explains how a bug that a valid input could trigger reached production and stayed there for almost four weeks.
Answer the testing question directly. In Fastly's words, it would figure out why it did not detect the bug during its software quality assurance and testing processes. Quality assurance is the testing work meant to catch flaws before customers do. A latent, input-triggered bug is precisely the class that ordinary testing misses, because the test inputs and the triggering input differ — so the useful question is not "did we test?" but "what would a test have had to include to catch this, and how do we generate such tests for configurations we have never seen?"
Evaluate remediation time. Fastly also committed to evaluating ways to improve its remediation time — the time from knowing to fixing. Its own timeline shows where the minutes lived: one minute to detect, forty minutes to identify the trigger, nine minutes from identification to recovery beginning. The forty-minute diagnosis is the obvious target, and it is a tooling and observability problem: the faster an engineer can correlate a global failure with the specific recent change that fits it, the shorter the next incident of this shape will be.
Build resiliency into the platform itself, with isolation. The longer-term commitment in the summary is architectural. Fastly said it had been — and would continue — investing in fundamental changes to the safety of its underlying platforms, broadly by fully leveraging the isolation capabilities of WebAssembly and its Compute@Edge platform to build greater resiliency from the ground up. In plain English: isolation means keeping one piece of running code from being able to bring down its neighbours — walls between compartments, so a fault floods one compartment instead of the ship. A bug triggered by one customer's configuration is exactly the failure isolation is meant to shrink: the ideal end state is not that such bugs never ship, but that when one fires, its blast radius is one customer, not 85 percent of the network.
What this means for you
Hyperscaler engineers
This is your case. Three checks fall straight out of Fastly's timeline. First: treat every customer-reachable configuration surface as a code surface. A valid input that can crash the platform is a platform bug, full stop — which means configurations deserve the same fuzzing (feeding a system large numbers of varied, unexpected inputs to see what breaks), validation, and staged rollout discipline as code deployments. Second: measure your change-to-failure correlation time honestly. Fastly needed 40 minutes to connect a global failure to one valid change among many. If your tooling cannot answer "which recent changes, across all tenants, match this failure's start time?" in minutes, that gap — not detection — is your remediation budget. Third: design for the latent bug, not just the bad change. The May 12 code passed whatever gates existed and then waited. Canary deployments (rolling a change to a small slice first) only help if the canary period exercises the inputs that matter — and a trigger that arrives monthly will sail through a canary that lasts a day.
Enterprise network engineers
Your version of this story is the firewall policy push, the load-balancer configuration change, or the SD-WAN template update that is valid, approved, and wrong for one device in a way nobody tested. The transfers are direct. Keep a fast, rehearsed way to answer "what changed, everywhere, in the last hour?" — Fastly's 40 minutes was spent on exactly that question, at global scale, under pressure. Make disabling a change as fast as pushing it: Fastly recovered when it could disable one configuration centrally, and your equivalent is a rollback path you have actually practised, not a change ticket in reverse. And borrow the isolation lesson at your scale: a template or policy that applies to every site at once has an 85-percent-style blast radius by design, so stage it — one site, a pause, then the fleet — even when the change is, on paper, completely valid. Especially then.
HFT / low-latency engineers
Your configurations are strategy parameters, risk limits, and feed-handler settings — pushed fast, often by the people closest to the market, and treated as safe because they are "just settings." Fastly's day is the counter-argument: a setting is an input to code, and code with a latent bug turns a valid setting into an outage. The controls that map are pre-trade in spirit: validate parameter changes against bounds and combinations the code is proven to handle, not just against the field being changed; log every change with a timestamp your monitoring can correlate against a failure in seconds; and keep a single-action way to revert the last configuration state of a running system without restarting it. Fastly disabled one configuration and recovered in nine minutes. If reverting your last parameter push takes a redeploy, your remediation time is already written.
Try it yourself
Two free Packet Path tools build the habits this case rewards. Start with the Network Config Verifier: paste configurations side by side and it flags mismatches and inconsistencies — the discipline of treating a configuration as something to be checked, compared, and proven before it is trusted, which is the habit that shrinks the "valid but dangerous" category. Then map a set of device configs with the Config to Topology Mapper: it draws the network your configs actually describe and surfaces mismatches, so you can see the blast radius of a shared configuration — what else one change touches — before the change does it for you. Fastly's lesson is that the map and the check are cheapest before June 8, not during it.
Sources
Primary source — Fastly, Summary of June 8 outage, published June 8, 2021. Every fact, time, figure and commitment on this page is taken from this summary, including: the cause (an undiscovered software bug triggered by a valid customer configuration change); the origin (a software deployment that began May 12, 2021, introducing a bug triggerable by a specific customer configuration under specific circumstances); the impact figure (85% of the network returning errors); the detection and recovery figures (detected within one minute; 95% of the network operating as normal within 49 minutes); the full UTC timeline (09:47 onset, 09:48 identification, 09:58 status post, 10:27 configuration identified, 10:36 recovery began, 11:00 majority recovered, 12:35 mitigated, 12:44 resolved, 17:25 bug-fix deployment began); and the follow-up commitments (deploy the fix, a complete post-mortem, why quality assurance and testing missed the bug, improving remediation time, and leveraging WebAssembly and Compute@Edge isolation for resiliency). Full text: https://www.fastly.com/blog/summary-of-june-8-outage
No other sources are used for facts on this page. The summary does not name the triggering customer or describe the specific configuration, and this page does not guess at either. Lists of affected websites that appear in press coverage are not in Fastly's summary and are not stated here on its authority.
- The customer did nothing wrong: Fastly's summary stresses the configuration change was valid — the fault was a bug Fastly had shipped on May 12 that should have anticipated it.
- A latent bug is a timed risk: the code sat in production for almost four weeks before the one input that could trigger it arrived, which is why testing that only covers expected inputs misses this class of failure.
- Speed of detection was not the problem: monitoring identified the disruption within one minute (09:47 to 09:48 UTC) — the 40 minutes to 10:27 was finding which configuration, among all valid changes, was the trigger.
- Bulk recovery and full recovery are different clocks: 95% of the network was normal within 49 minutes, but the incident was not mitigated until 12:35 and not resolved until 12:44 — almost three hours after onset.