A regex took down Cloudflare. 27 minutes.
July 2, 2019, 13:42 UTC: one new rule for Cloudflare's Web Application Firewall goes live in 180+ cities in about two seconds.
July 2, 2019, 13:42 UTC: one new rule for Cloudflare's Web Application Firewall goes live in 180+ cities in about two seconds. It is an XSS rule in simulate mode, so it blocks nothing, but it still runs on every request, and it ends in .*(?:.*=.*). PCRE backtracks, every CPU core serving HTTP hits 100 %, and every Cloudflare-proxied site returns 502 for 27 minutes; traffic drops 82 %. The kill switch sits behind Cloudflare Access, which sits behind Cloudflare. Postmortem, from Cloudflare's own write-up: the CPU guard removed weeks earlier in a refactor meant to save CPU, the procedure that let any rule skip staging, the engine with no complexity guarantee, the 11 causes and the 7 fixes. Verdict on the fix: SHIP IT.
Read the written edition (English) ↗
What this video covers
- 13:31–13:42 UTC: PR merged, CI green (no CPU test), Quicksilver pushes the rule to 180+ cities; WAF rules skip the DOG → PIG → Canary stages
- 13:45–14:07: first page; CPU 100 % worldwide, traffic −82 %, 502s everywhere; the internal control panel is behind Cloudflare Access, which is down; some credentials have expired; a rarely drilled bypass
- 14:07–14:09: global WAF terminate; traffic and CPU normal after 27 minutes; 14:52 WAF back minus the rule
- Jul 2 (15:50 UTC) and Jul 12: Cloudflare publishes the same-day note and the full postmortem; CPU guard re-added, 3,868 rules re-read, staged rollouts, move to a linear-time regex engine
Questions and answers
How can a single regular expression take down a global network?
A regex with unbounded backtracking can pin every CPU core that evaluates it. According to Cloudflare's postmortem, a new firewall rule ended in .*(?:.*=.*), PCRE backtracked through every other split, and CPU hit 100 % on every core serving HTTP. Sites behind Cloudflare returned 502 errors for 27 minutes and traffic fell 82 %.
Why did a firewall rule in simulate mode still cause an outage?
Simulate mode stops a rule from blocking traffic, but the rule is still evaluated on every request. The rule merged at 13:31 UTC on July 2, 2019 was a cross-site scripting rule in simulate mode, so it blocked nothing, yet it ran on every request. Cloudflare's postmortem says the cost of evaluating it exhausted the CPU.
How did one rule reach every Cloudflare data center in two seconds?
Firewall rules skipped the staged rollout that other releases went through. According to the postmortem, the procedure let any rule go global at once because rules exist to stop live attacks. This rule was not an emergency, and Quicksilver still pushed it to 180+ cities in about two seconds, with no CPU test in CI to catch it.
Why did it take Cloudflare 27 minutes to switch off the faulty firewall component?
The internal control panel sat behind Cloudflare Access, which sat behind Cloudflare, so the outage blocked the way to the kill switch. Some credentials had expired from disuse, and a bypass that was rarely drilled took about five minutes. The global WAF terminate was applied at 14:07 UTC, and traffic was normal again at 14:09.
Should teams keep regex engines with backtracking in request-path code after the Cloudflare outage?
The Daily Diff's verdict was SHIP IT, for Cloudflare's fix. The CPU guard was restored, 3,868 rules were re-read, rules go through staged rollouts, and the engine moves to one with linear-time guarantees. The episode's practical advice is to avoid patterns like dot-star dot-star in anything that runs on every request and to keep the kill switch off the system it controls.
Transcript
0:00 One regular expression goes live on every Cloudflare server at once, and for the next twenty-seven minutes the sites behind it are a 502 page, which, for a firewall, is the strictest possible setting. July 2nd, 2019, 13:42 UTC. Cloudflare posts within two hours: not an attack, a bad deploy, traffic down 82 percent. Ten days later CTO John Graham-Cumming publishes the full postmortem, regex included, and Hacker News gives it 698 points,
0:26 which for an outage is a standing ovation. How it happens, why it is possible, and who actually gets the blame. This is The Daily Diff, postmortem. 13:31. A pull request merges: one new firewall rule against cross-site scripting, in simulate mode, so it blocks nothing. 13:37, tests pass; none measures CPU. 13:42, the rule ships to 180 cities in two seconds, because WAF rules skip the dog, pig and canary stages other releases get.
0:54 13:45, the first page. 13:49, Hacker News has a thread on the status page, which still says all systems operational. The rule ends in dot-star, dot-star, equals, dot-star: anything, then anything, then an equals sign. PCRE guesses greedily, fails, and backtracks through every other split. x equals x takes 23 steps. Twenty x's after the equals: 555.
1:15 Twenty x's, no equals sign: 4,067 steps to find nothing. Run that on every request and every core is at one hundred percent, doing nothing, thoroughly. Two guards should have caught it. The CPU limit on rules was removed by mistake weeks earlier, in a refactor meant to make the WAF use less CPU. And the procedure lets any rule skip staging, because rules exist to stop live attacks; this one was not an emergency, and it went global anyway.
1:37 14:00, the WAF is identified; no attack. 14:02, someone proposes the global terminate: one component, off, worldwide. The switch is behind Cloudflare Access. Cloudflare Access is behind Cloudflare. Some credentials have expired from disuse, so the fastest network on the internet spends five minutes on a bypass nobody drilled. 14:07, kill. 14:09, traffic normal. git blame: a rollout with one speed, global; a CPU guard refactored away by
2:04 accident; a regex engine with no upper bound. Not the engineer who wrote the rule: the postmortem lists eleven causes and names nobody. Blast radius: 27 minutes, 82 percent of traffic, 100 percent CPU on every core, in every city. The dashboard and API sit behind the same edge, so customers cannot even switch it off. Hacker News, under the postmortem: they had one problem, used a regular expression, now they have two. Old joke. Still compiles.
2:28 Verdict, postmortem: ship it. The CPU guard is back, all 3,868 rules get read by hand, rules go through staging, and the engine moves to one with linear-time guarantees, published by Ken Thompson in 1968. Monday: no dot-star dot-star in anything that runs per request, and keep the kill switch off the thing it kills. Send me the incident you are still not allowed to talk about, in the comments, or at the daily diff dot dev.
2:53 And that's the diff for today. I'm Niko from Axrisi. Merge responsibly.
Sources
- John Graham-Cumming, "Details of the Cloudflare outage on July 2, 2019" (Jul 12, 2019)blog.cloudflare.com
- Matthew Prince, "Cloudflare outage caused by bad software deploy (updated)" (Jul 2, 2019)blog.cloudflare.com
- Matthew Prince on X, Jul 2, 2019, 14:22 UTCx.com
- Matthew Prince on X, Jul 2, 2019, 14:36 UTC ("No evidence yet attack related")x.com
- Hacker News, Jul 2, 2019, 13:49 UTC — "Cloudflare Network Performance Issues" (631 points)news.ycombinator.com
- Hacker News, Jul 2, 2019 — "Cloudflare outage caused by bad software deploy" (348 points)news.ycombinator.com
- Hacker News, Jul 12, 2019 — "Details of the Cloudflare outage on July 2, 2019" (698 points)news.ycombinator.com
- TechCrunch, Jul 2, 2019techcrunch.com



