Meta / Facebook: The Day Facebook Deleted Itself From the Internet
On October 4, 2021, a routine maintenance command took down every connection in Facebook's global backbone — the private network linking its data centers. Its own DNS servers then did exactly what they were designed to do: when they could no longer reach the data centers, they withdrew the routes that tell the internet where Facebook lives. Facebook, Instagram and WhatsApp vanished for hours, even though the servers themselves were still running. This is the documented story, from Meta's own post-mortems and Cloudflare's outside view, of how it happened, why recovery was so slow, and what Meta said it would change.
The short answer
On October 4, 2021, Facebook — the company now called Meta — disappeared from the internet for hours. It was not a hack and it was not a failed server. During routine maintenance, an engineer issued a command whose job was only to assess the availability of global backbone capacity — in plain English, to check how much spare capacity the private network linking Facebook's data centers had. Instead, the command took down all the connections in that backbone network, disconnecting Facebook's data centers globally. A safety system designed to audit commands like this and stop dangerous ones failed to stop it, because of a bug in the audit tool itself. Then a second, designed behaviour made everything worse: Facebook's DNS servers — DNS is the internet's address book, turning names like facebook.com into numbers computers can reach — are built to stop advertising themselves if they cannot talk to the data centers. When the backbone vanished, every one of them decided it was unhealthy and withdrew its BGP advertisements (BGP is the protocol networks use to announce "you can reach these addresses through me"). The servers were still operational. Nobody could find them. Facebook, Instagram and WhatsApp stayed unreachable until engineers physically reached the data centers, restored the backbone, and carefully brought services back without crashing them a second time.
The company: what Facebook did
Facebook in 2021 ran some of the most-used services on the internet: the Facebook app and site itself, plus Instagram and WhatsApp, which it owned. People and businesses around the world relied on those platforms every day to stay connected — which is why, when they vanished, Meta's first public note opened with an apology "to all the people and businesses around the world who depend on us."
Underneath the apps sits the infrastructure this story is about. Meta describes its backbone as the network it built to connect all of its computing facilities together: tens of thousands of miles of fiber-optic cables crossing the globe and linking all of its data centers. Those facilities come in two kinds, and the difference matters. Some are massive buildings housing millions of machines that store data and run the heavy computing behind the platforms. Others are smaller facilities that connect the backbone to the broader internet and to the people using the platforms.
Here is the normal journey of a tap, in Meta's own description. When you open an app and load your feed or messages, the request travels from your device to the nearest facility, which communicates directly over the backbone to a larger data center. That is where the information your app needs is retrieved and processed, and sent back over the network to your phone. Routers — the traffic directors of a network — manage the data traffic between all of these facilities, deciding where to send incoming and outgoing data. Break the backbone, and the small facilities at the edge have nothing to fetch. Break the address book as well, and the rest of the internet cannot even ask.
The problem: a check that disconnected everything, and a safety net with a hole in it
Three facts, all from Meta's own post-mortem of October 5, 2021, set the trap.
First, taking part of the backbone offline was normal work. Meta says its engineers often need to take part of the backbone offline for maintenance — repairing a fiber line, adding more capacity, or updating the software on a router itself. Commands that probe or change backbone capacity were part of day-to-day operations, not an unusual experiment. That ordinariness is the risk: a command that runs routinely is a command nobody fears, right up until it does something it was never supposed to do.
Second, the guardrail existed — and had a bug. In Meta's words: "Our systems are designed to audit commands like these to prevent mistakes like this, but a bug in that audit tool prevented it from properly stopping the command." Read that slowly, because it is the whole lesson in one sentence. Facebook had anticipated exactly this class of mistake and built a checker for it. The checker itself was faulty. A safety control you have never watched fail is not a proven safety control — it is an assumption with a user interface.
Third, the DNS safety behaviour was correct locally and catastrophic globally. Meta's smaller facilities answer DNS queries. The authoritative name servers behind them sit at well-known IP addresses, and those addresses are advertised to the rest of the internet via BGP. To ensure reliable operation, Meta built in a rule: if a DNS server cannot speak to the data centers, it disables its BGP advertisements, because that inability is a sign of an unhealthy connection. For a single sick location, that is good design — traffic stops flowing to a place that cannot serve it. But the rule had an unstated assumption: that only some locations would lose contact at once. When the entire backbone was removed from operation, every location made the same correct local decision at the same time. Meta's summary is precise: "The end result was that our DNS servers became unreachable even though they were still operational. This made it impossible for the rest of the internet to find our servers."
An analogy: imagine every branch of a delivery company follows the same sensible rule — "if you cannot reach head office, take your listing out of the phone book so customers do not waste trips." One branch with a cut phone line follows the rule and nobody notices. Then head office's phone system itself fails, every branch follows the rule on the same morning, and the company no longer exists in any phone book anywhere — while every warehouse sits full, staffed, and waiting.
What this shows: the outage was a chain, not a single break. Each step caused the next — including step 4, where the DNS servers hid Facebook from the internet by doing exactly what they were designed to do.
What happened on October 4, 2021
All times are UTC. The inside account is Meta's; the clock times for what the outside world saw are Cloudflare's, from its own network measurements published the same day.
During routine maintenance (October 4) — A command is issued with the intention to assess the availability of global backbone capacity. It unintentionally takes down all the connections in the backbone network, effectively disconnecting Facebook's data centers globally. The audit system that should have stopped it does not, because of the bug described above. Meta stresses the speed: "All of this happened very fast."
Around 15:40 UTC — Watching from outside, Cloudflare sees a peak of routing changes from Facebook. In its words: "That's when the trouble began." Routes are withdrawn, and Facebook effectively disconnects itself from the internet.
15:51 UTC — Cloudflare opens an internal incident titled "Facebook DNS lookup returning SERVFAIL," initially worried that something is wrong with its own resolver, 1.1.1.1. SERVFAIL is the error a resolver returns when it cannot get an answer. Engineers quickly confirm the larger truth: Facebook, WhatsApp and Instagram are all down, their DNS names have stopped resolving, and their infrastructure addresses are unreachable — "as if someone had pulled the cables from their data centers all at once."
Around 15:50–15:58 UTC — Cloudflare records that facebook.com stops being available on 1.1.1.1 at around 15:50, and at 15:58 it confirms Facebook has stopped announcing the routes to its DNS prefixes. With no BGP route to the name servers, Cloudflare's resolver — and Google's 8.8.8.8 and every other major public resolver — has no way to connect to them, and resolvers worldwide start issuing and caching SERVFAIL responses. Human and app behaviour then multiplies the load: apps retry, people reload and relaunch, and Cloudflare reports handling 30 times more DNS queries than usual on 1.1.1.1. Inside Facebook, the same chain is unfolding in Meta's telling: the DNS locations have declared themselves unhealthy and withdrawn their advertisements, and the data centers are cut off from the internet and from each other.
What this shows: Facebook's DNS addresses were announced from many sites at once, so users always had a nearby door. On October 4 every site withdrew at the same time — one shared address, gone everywhere at once.
The long afternoon — Meta's engineers face what its post-mortem calls two large obstacles. First, it is not possible to access the data centers through normal means, because their networks are down. Second, the total loss of DNS breaks many of the internal tools the company would normally use to investigate and resolve an outage like this. The first public note, on October 4, says the same thing more briefly: the underlying cause also impacted many of the internal tools and systems used in day-to-day operations, complicating attempts to diagnose and resolve the problem. Even the backup route in fails: "Our primary and out-of-band network access was down." Out-of-band access is the separate, emergency path engineers keep for exactly the day the main network dies. On this day, it was down too.
Onsite, in person — With remote access gone, Meta sends engineers onsite to the data centers to debug the issue and restart the systems. That takes time by design. The facilities are built with high levels of physical and system security: they are hard to get into, and once inside, the hardware and routers are designed to be difficult to modify even with physical access. It takes extra time to activate the secure access protocols needed to get people onsite and able to work on the servers. Only then can the team confirm the issue and bring the backbone back online. Meta is explicit that this slowness was a trade-off it accepts — see The fix, below.
Around 21:00–21:28 UTC — Cloudflare sees renewed BGP activity from Facebook's network, peaking at 21:17. facebook.com is available on 1.1.1.1 again at 21:20, and by 21:28 Facebook appears reconnected to the global internet with DNS working again. On Cloudflare's resolver clock, the name was unavailable for about five and a half hours; Meta's platforms then took further time to come fully back, as the next section explains.
The damage
Start with what did not happen, because Meta stated both points clearly and they bound the damage. There was no malicious activity behind the outage — its root cause was a faulty configuration change of Facebook's own making — and Meta said it had no evidence that user data was compromised as a result of the downtime. No attacker got in, and nothing was stolen. This was a self-inflicted availability failure, total in reach and limited in kind: everything stopped, nothing leaked.
The availability damage itself was as broad as an outage can be. Every major platform the company ran — Facebook, Instagram and WhatsApp — was unreachable worldwide at the same time, because all of them depended on the same backbone and the same DNS behaviour. There was no degraded mode and no partial region left standing to carry a smaller service. For the hours the names would not resolve, the services did not merely run slowly; from a user's point of view they did not exist.
The second-order damage landed on everyone connected to the platforms and on the internet around them. Meta apologised specifically to the people and businesses around the world who depend on its platforms — businesses that take bookings, messages and orders through those apps lost their channel for the duration. Cloudflare documented the spillover onto shared infrastructure: resolvers worldwide fielding 30 times the usual query volume as apps and people retried, with most queries still answered quickly but a small fraction seeing increased response times. An outage at this scale is never contained inside one company's walls, even when its cause is.
One honesty note belongs in any case study of this event: Meta's two post-mortems give no dollar figure for the loss, and this page invents none. The documented damage is duration, breadth and dependency — hours in which the world's largest social platforms were unreachable, the internal tooling needed to fix them was broken by the same event, and the recovery itself had to be done in person, at walking pace, inside deliberately hard-to-enter buildings.
How they recovered
Step one: get people to the machines. Because primary and out-of-band network access were both down, recovery began with travel, not commands. Engineers went onsite to the data centers, worked through the secure access protocols those facilities require, confirmed the issue in person, and restarted the systems. Meta does not publish a minute-by-minute onsite log, and this page does not invent one — the documented point is the sequence: physical access first, diagnosis second, backbone third.
Step two: restore the backbone. Once backbone connectivity was restored across the data center regions, Meta says, everything came back up with it — the edge facilities could reach the data centers again, the reason the DNS servers had declared themselves unhealthy disappeared, and the platforms could return. The outside view matches: BGP announcements resumed around 21:00 UTC, DNS began resolving again, and the services followed.
Step three: refuse the easy finish. The problem was not over when connectivity returned, and Meta knew it. Flipping every service back on at once could have caused a new round of crashes from the surge in traffic as the world's users returned together. There was a physical risk too: individual data centers were reporting dips in power usage in the range of tens of megawatts — idle machines draw far less power — and suddenly reversing a dip that large could put everything from electrical systems to caches at risk. So the return was managed carefully, load by load, rather than thrown open.
Why step three worked: rehearsal. Meta credits its long-running "storm" drills. In a storm exercise, the company simulates a major failure by taking a service, a data center, or an entire region offline, stress-testing the infrastructure and software involved. Experience from those drills gave the team the confidence and experience to bring things back online while carefully managing the increasing loads. In the end, Meta says, services came back up relatively quickly without any further systemwide failures. The honest caveat follows immediately in the same post: the company had never previously run a storm that simulated the global backbone being taken offline.
The fix: technology and controls that came out of it
Meta's post-mortem is a learning document, not a product announcement, and it names directions rather than a shipped checklist. Those stated directions — plus the control failures the document itself exposes — are the fix this case leaves behind.
Test the guardrail, not just the system. The audit tool was the control, and the bug was in the control. Meta's stated job after the outage was "to strengthen our testing, drills, and overall resilience." Applied concretely to this failure: a command-auditing system needs its own adversarial testing — a library of known-dangerous commands that must be blocked, run against the auditor continuously — because an untested blocker fails silently on exactly the day it is needed. Any engineer reading this case should ask of their own change guards: when did we last watch ours refuse something real?
Drill the failure you have not imagined yet. Meta says it will be looking for ways to simulate events like this — a global backbone loss — in its storm drills going forward. The general lesson: a drill programme that only rehearses failures already survived rehearses the past. The value of the October 4 event, used properly, is a new drill scenario whose defining feature is that the recovery tools themselves are inside the blast radius.
Design recovery paths that do not depend on the thing that broke. This outage's cruellest property was circular dependency: fixing the network required network access, diagnosing the outage required DNS, and DNS was the outage. Meta documents that both primary and out-of-band access were down and that internal tools broke with DNS. A recovery path is only independent if it shares nothing — no DNS, no backbone, no identity system — with the production path it replaces, and if that independence is tested, not diagrammed.
Make correlated safety behaviour a design question. The DNS withdrawal rule was right for one location and wrong for all locations at once. Controls that make sense per-node need a second question at fleet scale: what happens when every node does this simultaneously, on the same signal? Rate limits, staggered action, or a floor below which the fleet will not withdraw are the usual shapes of the answer — the case's contribution is proving the question is not theoretical.
Keep the security trade-off, eyes open. Meta states plainly that its hardening against unauthorised access slowed recovery from an error of its own making — and that it believes the trade-off is worth it: greatly increased day-to-day security against a slower recovery from a hopefully rare event. That is a defensible call, made explicitly. The lesson for everyone else is to make the same call explicitly in advance: know how long physical, secure access takes at your own facilities, and count that time in your recovery estimates instead of discovering it mid-incident.
Review extensively, every time. Meta notes that after every issue, small and large, it runs an extensive review process to understand how its systems can be made more resilient — and that the process for this event was already underway when the post-mortem was published. The two public posts themselves, published within about a day and naming the audit bug in plain language, are part of that fix: the next company to build a command auditor now knows to test it.
What this means for you
Hyperscaler engineers
This is your case study, at your scale, with your failure shapes: a global backbone, a fleet-wide automated behaviour, and a safety tool whose bug mattered more than the command it failed to stop. Three checks follow directly. First, inventory your correlated behaviours — every place a fleet of nodes makes the same health decision on the same signal — and ask what the all-at-once outcome is. Second, treat guardrail systems (command auditors, change blockers, blast-radius limiters) as production systems with their own tests, drills and failure reviews; October 4 is what a guardrail bug costs. Third, prove your out-of-band path shares no dependency with production — then take production away in a drill and use it for real.
Enterprise network engineers
Your backbone is smaller, but the pattern fits in a single building: one maintenance command on the core, one automation script with broader rights than its author realised, one DNS or identity service that every recovery tool quietly depends on. The portable lessons are the circular-dependency hunt — can you still reach, log into, and change your core devices if DNS and your usual jump host are down? — and the guardrail question: whatever reviews your changes, human or software, when did it last say no, and did anyone notice whether it was right? Meta had the tool and the process; the tool had a bug and nobody knew. A second pair of eyes that has never caught anything is the same risk at human scale.
HFT / low-latency engineers
Your version runs in seconds, not hours, which makes the same three properties more dangerous, not less: a capacity or routing command with fleet-wide reach, a pre-trade or pre-deploy checker that is assumed rather than proven, and recovery tooling that rides the same network and name services as the trading path. Meta could afford a recovery measured in travel time; a trading firm cannot. The transferable discipline is identical — independent, tested kill and recovery paths that share nothing with the path they rescue — with the addition that yours must work at machine speed, by a human who may be watching the failure through tools the failure has already blinded.
Try it yourself
This outage turned on BGP — routes withdrawn, names unresolvable, a network present but unfindable. You can build the reading skill for exactly that view of the internet with Packet Path's free BGP Toolkit: look up prefixes and autonomous systems, check how routes are announced, and practise seeing a network the way the rest of the internet sees it. Pair it with the free Network Config Verifier to compare device configurations side by side — the habit that catches the one change, or the one checker, that does not match the rest of the fleet before the fleet finds out for you.
Sources
Primary source — Meta (Facebook) Engineering, Santosh Janardhan, "More details about the October 4 outage" (October 5, 2021): the backbone and its tens of thousands of miles of fiber, the two kinds of facilities, the maintenance command intended to assess backbone capacity and the audit-tool bug, the DNS/BGP withdrawal mechanism, the loss of normal and out-of-band access, engineers sent onsite and the secure-access delay, the tens-of-megawatts power dips, the storm drills, and the stated work on testing, drills and resilience. https://engineering.fb.com/2021/10/05/networking-traffic/outage-details/
Primary source — Meta (Facebook) Engineering, Santosh Janardhan, "Update about the October 4th outage" (October 4, 2021; updated October 5, 2021): configuration changes on the backbone routers as the cause, the cascading halt to services, the impact on internal tools and day-to-day systems, no malicious activity, and no evidence that user data was compromised. https://engineering.fb.com/2021/10/04/networking-traffic/outage/
External observation — Cloudflare, "Understanding how Facebook disappeared from the Internet" (October 4, 2021): the outside clock and measurements — the peak of routing changes at around 15:40 UTC, the internal incident opened at 15:51 UTC, Facebook's DNS prefixes no longer announced at 15:58 UTC, facebook.com unavailable on 1.1.1.1 from around 15:50 UTC and returning at 21:20 UTC, renewed BGP activity around 21:00 UTC peaking at 21:17 UTC, reconnection by 21:28 UTC, and the 30x surge in DNS queries. Cloudflare observed the outage from outside Facebook; Meta's posts above are the source for everything inside it. https://blog.cloudflare.com/october-2021-facebook-outage/
- The trigger was routine: a command meant only to assess backbone capacity disconnected the whole backbone, and a bug in the audit tool that should have blocked it let it through.
- The servers never crashed. Facebook's DNS servers withdrew their BGP advertisements by design when they lost contact with the data centers — so the internet could no longer find services that were still running.
- Recovery needed the thing that was broken: normal and out-of-band network access were both down, and the total loss of DNS broke the internal tools engineers normally use to diagnose an outage.
- Meta's stated response was not a single patch: stronger testing, storm drills that simulate losing the whole backbone, and an extensive resilience review after the event.