Packet Path/

Cloudflare: One Regex That Silenced the Internet's Front Door

On July 2, 2019, Cloudflare deployed a minor rule change to its Web Application Firewall — the kind of change its team made every few hours. One pattern in one rule made the matching engine retry combination after combination on every request, and within minutes every processor core serving web traffic across more than 180 cities was exhausted. Visitors worldwide got 502 errors and traffic fell 82% at its worst. The fix required switching the firewall off everywhere at once — through a control panel the outage itself had locked the engineers out of. This is the documented story, from Cloudflare's own same-day post and full post-mortem, of how one pattern took down the internet's front door for 27 minutes, and the eleven converging causes Cloudflare listed afterwards.

Case Studies · Intermediate · 11 min · November 7, 2026

Editorial illustration on a dark teal background: a globe wrapped in glowing network lines converging on one overloaded server whose gauges are pinned at maximum, with small warning signs flickering across the continents it serves.

The short answer

On July 2, 2019, at 13:42 UTC, Cloudflare automatically deployed a minor change to its WAF Managed Rules — the shared set of protective patterns its Web Application Firewall runs on web traffic worldwide. (A web application firewall, or WAF, is a filter that inspects incoming web requests for attack patterns before they reach a website.) One rule, written to catch cross-site scripting (XSS) attacks — a common way of smuggling malicious scripts into web pages — contained a regular expression (a pattern-matching recipe, shortened to regex) that backtracked enormously: the matching engine kept retrying combination after combination instead of finishing. The rule shipped in "simulate" mode, where nothing gets blocked — but the rule still executed on real traffic. Within minutes, every processor core serving HTTP/HTTPS traffic across Cloudflare's network was exhausted, front-line servers returned 502 errors (the "bad gateway" message a middleman shows when it cannot get an answer from behind it), and traffic dropped 82% at its worst, per Cloudflare's same-day post. Engineers identified the WAF by 14:00, executed a global termination of the WAF at 14:07, and traffic and CPU were back to normal by 14:09. The WAF itself returned at 14:52, after careful single-city testing. Cloudflare's post-mortem puts the customer-facing outage at 27 minutes.

The company: what Cloudflare did

Cloudflare runs a global network of points of presence (PoPs) — data-center locations — in more than 180 cities. Websites put Cloudflare in front of their own servers: a visitor's request reaches Cloudflare first, and Cloudflare provides the core proxying (passing requests through to the site), a CDN (content delivery network — keeping copies of content near visitors so pages load fast), and the WAF that inspects traffic for attacks. That middleman position is why this outage was everyone's outage: when Cloudflare's front door jammed, every site sitting behind it showed errors, even though those sites' own servers were fine.

The WAF Managed Rules are the shared rulebook of that filter, and they change constantly, because new attacks appear constantly and the speed of the response is the product. In the 60 days before this incident, Cloudflare handled 476 change requests for the WAF Managed Rules — an average of one every three hours. In May 2019, that same speed had let Cloudflare push a rule protecting customers against a serious SharePoint vulnerability as exploits spiked. And the machinery underneath was built for speed too: a configuration system, described below, that makes changes live everywhere in seconds. The post-mortem notes, with some pride dented, that this was Cloudflare's first global outage in six years.

The problem: a fast road with no guardrails

Nothing about the morning of July 2 was unusual, and that is the point. Three facts — each sensible on its own — lined up to make one bad pattern a global outage.

First, WAF rules skipped the careful road by design. Every other software change at Cloudflare rolled out in stages: first to DOG, an internal PoP used only by employees (dogfooding — using your own product first), then to PIG (as in guinea pig), a PoP carrying a small subset of non-paying customer traffic, then to three Canary PoPs around the world carrying a final mix of traffic, and only then to the world — a process that can take hours or days, and that frequently catches problems early. WAF rules did not use that road, because a rule answering a live attack cannot wait hours. Instead, rules rode Quicksilver, Cloudflare's distributed key-value store (a system for holding configuration values and copying them everywhere), which pushes a change to every machine worldwide with a p99 of 2.29 seconds — meaning 99% of changes land everywhere within about two and a third seconds — at an average rate of about 350 changes per second. Speed like that is normally the feature customers love. On July 2 it meant a bad rule was global before anyone's pager went off.

Two roads to production NORMAL SOFTWARE — STAGED ROLLOUT, HOURS OR DAYS DOG Employees only dogfooding PoP PIG Small subset of non-paying customers CANARIES Three PoPs final check GLOBAL Worldwide — only after every check A problem found at any stage stops the rollout before most customers ever see it. WAF RULES ON JULY 2, 2019 — STRAIGHT TO GLOBAL ONE RULE CHANGE XSS rule — simulate mode still runs on real traffic QUICKSILVER Distributed config store replicated to 180+ cities GLOBAL IN SECONDS p99: 2.29 seconds about 350 changes a second Built for speed against live attacks — but no canary step stood between one bad rule and the world. Same company, same day: the careful road for code, the fast road for rules. The outage took the fast road.

What this shows: ordinary Cloudflare software earned its way to the world through three trial stages. WAF rules, by design, went straight from one change to every city in seconds — the design choice that turned one pattern into a global outage.

Second, the guard that would have caught the cost was gone. Cloudflare's post-mortem lists the converging causes plainly. A protection that would have helped prevent excessive CPU use by a regular expression had been removed by mistake weeks earlier, during a refactoring (a restructuring of code) that was itself part of making the WAF use less CPU. Underneath, the WAF was written in Lua and matched patterns with the PCRE engine, which uses backtracking and — in Cloudflare's words — has no complexity guarantees and no mechanism to protect against a runaway expression. And the safety checks looked the wrong way: the test suite verified that rules blocked attacks they should block and passed traffic they should pass, but it had no way of identifying excessive CPU consumption — and the logs of previous builds showed no increase in test-suite run time that might have warned anyone. Every step up to deployment was done "correctly": a pull request (a proposed code change) was raised and approved, the build system built it and ran the tests, a change request was filed with a rollout and rollback plan. The process was followed. The process could not see this failure.

Third, "simulate" mode still runs the code. The change was deployed in simulate mode, where real customer traffic passes through the rule but nothing is blocked — a way to measure false positives before a rule goes live. But a rule in simulate mode still has to execute to be measured. For anything the code consumes — CPU, memory, time — a test mode that runs the code is a production mode.

Now the match itself. The heart of the pattern, Cloudflare explains, can be treated as .*.*=.* — "match anything, then anything, then an equals sign, then anything." The dot means any single character; the star means "zero or more of the previous thing, greedily" — grab as much as possible first. When a greedy grab leaves nothing for the rest of the pattern, the engine backtracks: it gives a little back and tries again, and again, through every combination. On a short string that is merely wasteful. As the string grows, the combinations explode. Cloudflare's appendix counts the steps exactly: matching the three characters x=x takes 23 steps. Matching x= followed by 20 x characters takes 555 steps. Twenty x characters with no equals sign at all takes 4,067 steps — just to conclude there is no match. A close variant of the pattern ending in a semicolon takes 5,353 steps on that same input. Real web requests are far longer than 20 characters, and every one of them had to be checked.

Backtracking: a little more input, a flood of work EDGE CPU: ~100% Steps the PCRE engine needed to test one short string against the pattern — Cloudflare's appendix x=x — matches 23 steps x= plus 20 more x characters — matches 555 steps 20 x characters, no equals sign — no match 4,067 steps — just to fail Same input, pattern variant ending in a semicolon — no match 5,353 steps — just to fail Bar lengths are compressed so all four fit on one chart — read the step counts, not the bars. On July 2, every core serving web traffic was doing this kind of work, on real requests, all at once.

What this shows: the same pattern is cheap on a tiny input and ruinous on a slightly bigger one — 23 steps to match three characters, thousands of steps just to fail on twenty. Multiply that by real web traffic, on every core, in every city, and the CPUs had nothing left for serving pages.

What happened on July 2, 2019

All times are UTC, from Cloudflare's full post-mortem, with the traffic figure from its same-day post.

July 2, 2019 — all times UTC 13:42 — The rule goes global One XSS rule deploys worldwide in simulate mode — CPUs spike toward 100% 13:45 — First pages fire A synthetic WAF test fails first, then traffic-drop, 502 and CPU alerts; a P0 is declared 14:00 — The WAF is identified Live CPU profiling and strace point at the WAF; an attack is ruled out 14:02 — Global terminate is decided The team agrees to disable the WAF worldwide — if they can reach the controls 14:07 — Termination executed Reached through a rarely used bypass, because Cloudflare Access was down too 14:09 — Traffic and CPU back to normal The rest of Cloudflare's protections kept running throughout 14:52 — The WAF returns Re-enabled globally only after single-city negative and positive testing

13:31 — An engineer on the firewall team merges a pull request containing the rule change, after approval. 13:37 — The build system (TeamCity) builds the rules and runs the test suite, and gives it the green light. Nothing in the suite measures runaway CPU, and nothing looks slow.

13:42 — The automatic process deploys the change globally: a minor change to the rules for XSS detection, in simulate mode. Quicksilver does what it is built to do, and the rule is live everywhere in seconds. CPUs dedicated to serving HTTP/HTTPS traffic spike to nearly 100% across the network.

About 13:45 — Three minutes later, the first PagerDuty page goes out: a synthetic test (an automated pretend-visitor that checks the WAF from outside, one of hundreds of such tests) reports a fault with the WAF. It is rapidly followed by pages for many other failing end-to-end tests, a global traffic-drop alert, widespread 502 errors, and reports from PoPs in cities worldwide of CPU exhaustion. The site reliability engineering (SRE) team — the engineers on call for the health of the service — immediately declares a P0 incident, its most serious level, and escalates to engineering leadership. In the first moments there is speculation that this is an attack of a type never seen before. One leader reports having lost 80% of traffic. The London engineering team, gathered for an internal talk, abandons it and assembles in a conference room, with others dialing in.

14:00 — The WAF is identified as the component causing the problem, and an attack is dismissed. The Performance Team pulls live CPU data from a machine that clearly shows the WAF is responsible; another team member confirms it with strace (a tool that watches what a running program is doing); error logs agree.

14:02 — It is proposed that the team use the global terminate: a mechanism built into Cloudflare to disable a single component worldwide. The same-day post records the decision the same way: at 14:02 they understood what was happening and decided to issue a global termination on the WAF Managed Rulesets. Deciding was the easy part.

The locked door — Cloudflare uses its own products, and its Access service — the gatekeeper that authenticates staff to internal systems — was down with everything else, so the team could not authenticate to its own internal control panel. Other internal services, like the ticket system and the build system, were hard to reach too. The way in was a bypass mechanism that was not frequently used — and, the post-mortem admits, not well trained on. And once in, they would discover that some team members had lost access entirely, because a security feature disables credentials that have not been used on the internal control panel recently. At 14:07, a team member executes the global WAF termination.

14:09 — Traffic levels and CPU are back to expected levels worldwide. The rest of Cloudflare's protection mechanisms continued to operate — the sites behind Cloudflare were reachable again, just without that one layer of inspection.

14:52 — Restoration is deliberately slow. Because of the sensitivity of the situation, the team performs both negative tests (asking "was it really that particular change?") and positive tests (verifying the rollback works) in a single city, on a subset of traffic, after removing paying customers' traffic from that location. Only when they are 100% satisfied that they understand the cause and have a fix in place is the WAF re-enabled globally.

The damage

The customer-facing outage lasted 27 minutes, per the full post-mortem; Cloudflare's first same-day note, written while the incident was still unfolding, said "about 30 minutes." For those minutes, the failure was total in kind: customers — and their customers — saw a 502 error page when visiting any Cloudflare domain. The 502s were generated by front-line servers that still had CPU available but could not reach the processes that serve web traffic, because those processes were drowning in pattern matching. At its worst, traffic dropped by 82% (the same-day post's written figure; a leader's spoken estimate during the incident, quoted in the post-mortem, was 80%).

The blast radius included Cloudflare itself. Its own operations were hurt while dealing with the incident — internal tools unreachable, a status page that was not updated quickly enough (Cloudflare lists that as one of its own failures), and, for customers, a quieter trap: customers could not reach the Cloudflare Dashboard or API, because both pass through Cloudflare's own edge. The people trying to check on the outage, or work around it, were locked out by the outage.

Two honesty notes belong here. This was not an attack — Cloudflare said so on the day, ruled it out during the incident, and the post-mortem confirms a bad deploy as the cause; nothing in either document suggests data was taken or harmed. And Cloudflare published no dollar figure for the loss, so this page invents none. The documented damage is breadth and duration: the front door of a large share of the web, closed for 27 minutes by a single pattern in a single rule — the company's first global outage in six years.

How they recovered

Step one: prove what it is, fast. With an unknown global CPU event, the first fear is an attack. Within about eighteen minutes of the first page, live CPU profiling, strace, and error logs had converged on the WAF — and just as importantly, had ruled the attack theory out, which changed the response from defense to rollback.

Step two: use the big switch, not the slow rollback. The written rollback plan for a rule change required running the complete WAF build twice — one of the eleven causes Cloudflare lists, because it took too long. The global terminate was the practical path: disable the whole component worldwide, immediately, and trade a layer of protection for a working service. It worked in two minutes: executed at 14:07, normal by 14:09.

Step three: get in through the side door. The termination nearly did not happen on time for an organisational reason, not a technical one: the normal way into the control panel depended on a service the outage had taken down. Recovery rode on a bypass that was rarely used and under-drilled, and on team members whose credentials had not timed out. Both facts are in Cloudflare's own list of what went wrong — which is another way of saying the company knows how close that margin was.

Step four: bring the protection back like it matters. With traffic flowing, the team resisted flipping the WAF straight back on. They tested in one city, on a subset of traffic, with paying customers removed from that location, checking both that the bad rule really was the cause and that the rolled-back rules really worked — then re-enabled globally at 14:52, forty-three minutes after the service had recovered. The outage ended at 14:09; the incident was not declared fixed until the protection was proven and restored.

The fix: technology and controls that came out of it

Cloudflare's post-mortem refuses a single root cause. It lists eleven converging causes — the backtracking regex; the CPU protection removed by mistake; an engine with no complexity guarantees; a test suite blind to CPU; a procedure allowing a non-emergency rule to go global unstaged; a rollback that needed two full builds; a traffic-drop alert that fired too slowly; a status page updated too slowly; internal systems hard to reach, with an untrained bypass; SRE credentials timed out; and customers locked out of the Dashboard and API. The fixes map onto that list, and Cloudflare reported their status as of the post-mortem:

Restore the guard, then audit everything it guarded. Release work on the WAF was frozen completely. The excessive-CPU protection that had been removed was re-introduced (done), and engineers manually inspected all 3,868 rules in the WAF Managed Rules for other possible backtracking — inspection complete at the time of writing.

Teach the tests to measure cost. Performance profiling for all rules was being added to the test suite (estimated July 19, 2019) — so a rule would be judged not only on whether it matched correctly, but on what it cost to match.

Change the engine, not just the rules. Cloudflare committed to switching to either the re2 or the Rust regex engine (estimated July 31, 2019), both of which have run-time guarantees — the match finishes in time proportional to the input instead of exploding combinatorially. Its appendix explains why this is the real fix: the mathematics of guaranteed-linear matching has been known since Ken Thompson's 1968 paper, and no amount of careful pattern-writing gives a backtracking engine that guarantee. Longer term, the Lua WAF itself was being ported to Cloudflare's new firewall engine, for speed and for another layer of protection.

Stage the fast road. The standard procedure was changed to do staged rollouts of rules in the same manner used for other software — while explicitly retaining the ability to deploy globally in an emergency, for active attacks. The fast road was not closed; it was given gates.

Fix the incident machinery itself. An emergency ability to take the Dashboard and API off Cloudflare's own edge, so customers are never again locked out by the failure they are trying to manage; and automated updates of the status page, so the public word keeps pace with the internal one. Read against the eleven causes, the remaining lessons — drill the bypass, keep responder credentials alive, alert faster on global traffic drops — are stated as failures even where the post-mortem does not attach a dated project to them.

What this means for you

Hyperscaler engineers

This is your failure shape at its purest: a configuration system so fast that "deployed" and "global" are the same moment. Quicksilver's p99 of 2.29 seconds is a wonderful property for cache purges and a terrifying one for untested rules — the propagation time of a change is also the propagation time of its blast radius. Three checks follow. First, what in your fleet still bypasses staged rollout, and is the bypass justified by a live emergency or by habit? Cloudflare kept its emergency path and gated the routine one; that distinction is the whole fix. Second, do your tests measure resource cost, or only correctness? A rule, query, or pattern that is right and ruinously expensive passes every functional test you own. Third, does your control plane depend on your data plane? If the panel that would save you authenticates through the service that is down, your recovery time is measured in bypasses and luck.

Enterprise network engineers

Your version ships on change night: a firewall or WAF rule pushed to every site at once, a regular expression in a log parser or email filter, an antivirus signature, a QoS policy — small text changes with fleet-wide reach and no canary. Cloudflare's eleven causes translate directly: stage rule changes to one site first even when the tool makes global push one click; ask of every pattern and policy not just "is it correct?" but "what is its worst-case cost on the longest input it will ever see?"; and know, before you need it, what your global terminate is — the one action that disables the suspect component everywhere — and whether you can still authenticate to run it on the day your identity service is the thing that is down. Also take the credential lesson personally: responders locked out by inactivity timeouts is a failure you can schedule a drill for this quarter.

HFT / low-latency engineers

Your matching engines parse market data and messages under microsecond budgets, which makes worst-case behaviour your whole world: an algorithm that is fast on average and unbounded on a crafted or merely unusual input is a latency incident waiting for its input. Cloudflare's engine lesson is yours verbatim — prefer engines and parsers with run-time guarantees, bound the work any single message can cause, and treat a protection removed during a performance refactor as a change that needs the same review as the feature it shipped with. And the deployment lesson scales down in time, not in importance: a config or rule push that reaches every matching engine at once has no canary by default. At your speeds, the global terminate has to work in seconds and be reachable when the primary path is the casualty — exactly the property Cloudflare discovered it had under-drilled.

Try it yourself

Cloudflare's failure began as a rule nobody had inspected for cost, shipped by a process nobody had staged. You can practise the inspection half of that lesson with Packet Path's free Firewall / ACL Analyzer: paste a rule set and it flags shadowed, redundant, and overly broad rules — the habit of reading rules as a system before they ship, not after they page you. Pair it with the free Network Config Verifier, which compares device configurations side by side and flags mismatches — because the other half of this story is change discipline: knowing exactly what changed, where it landed, and whether every node got the same thing.

Sources

Primary source — Cloudflare, John Graham-Cumming, "Details of the Cloudflare outage on July 2, 2019" (full post-mortem): the timeline (13:31 merge, 13:37 build and tests, 13:42 global deploy in simulate mode, first page about three minutes later, WAF identified at 14:00, global terminate proposed at 14:02 and executed at 14:07, traffic and CPU normal at 14:09, WAF re-enabled at 14:52), the 27-minute customer-facing outage, the regular expression and the appendix step counts (23 steps for x=x; 555 steps for x= plus 20 x characters; 4,067 steps for 20 x characters with no equals sign; 5,353 steps for the semicolon variant), the DOG–PIG–Canary staged process, Quicksilver (more than 180 cities, p99 of 2.29 seconds, about 350 changes per second), the 476 WAF change requests in 60 days, the eleven converging causes, and the fix list with its statuses (CPU protection re-introduced, 3,868 rules inspected, performance profiling estimated July 19, re2/Rust engine estimated July 31, staged rollouts for rules, Dashboard/API emergency removal, automated status updates, port to the new firewall engine). https://blog.cloudflare.com/details-of-the-cloudflare-outage-on-july-2-2019/

Primary source — Cloudflare, Matthew Prince, "Cloudflare outage caused by bad software deploy" (July 2, 2019; updated 20:09 UTC the same day): the 502 errors from 13:42 UTC, CPU spiking to 100% on machines worldwide, traffic dropping 82% at its worst, the decision at 14:02 UTC to issue a global termination on the WAF Managed Rulesets and service restoration at 1409 UTC as recorded that day, the rollback and testing of the specific rules, re-enabling at 1452 UTC, and the statement that the outage was not an attack. Where the two posts differ in framing — "about 30 minutes" in the first note, 27 minutes in the post-mortem; termination "at 1409" in the first note, executed at 14:07 with traffic normal at 14:09 in the post-mortem — this page uses the post-mortem's figures and says so. https://blog.cloudflare.com/cloudflare-outage/

Key takeaways

Keep reading