Google Cloud crashed on a blank field. Three hours.
A policy row with a few blank fields hits a null pointer, and Google Cloud crashes in every region at once — then Cloudflare falls with it.
A policy row with a few blank fields hits a null pointer, and Google Cloud crashes in every region at once — then Cloudflare falls with it. June 12, 2025, 17:49 UTC: Service Control, the binary that approves every Google Cloud API request, reads a quota-policy change with "unintended blank fields", hits a code path shipped two weeks earlier with no null check and no feature flag, and goes into a crash loop in every region — the row had replicated globally within seconds. External APIs return 503 for three hours; us-central1 takes 2 h 40 min because the restarting tasks stampede the same Spanner table with no randomized backoff. Three minutes into Google's outage, Cloudflare's Workers KV — whose source of truth sits on "a third-party cloud provider" — loses 90 % of requests, and Access, WARP, Workers AI, Pages, Stream and Turnstile go down with it for 2 h 28 min. Google's status page posts its first update an hour late, because it runs on Google Cloud.
Read the written edition (English) ↗
What this video covers
- A blank field, a null pointer, every region
- Timeline: May 29 → 17:45 → crash loop → red button → 17:52 Cloudflare
- us-central1: the herd effect
- Mechanism: check in the request path, global replication, one vendor
- git blame — the split
Questions and answers
What caused the June 12, 2025 Google Cloud outage?
A quota-policy row with blank fields crashed Service Control, the binary that approves every Google Cloud API request. According to Google's incident report, the row hit a code path with no null check and no feature flag, and the binary went into a crash loop in every region. External APIs returned 503 errors for about three hours.
Why can restarting services make a cloud outage last longer in one region?
Many tasks restarting at once can overload the same shared database. In us-central1, Google's restarting tasks stampeded one Spanner table with no randomized backoff, so Google throttled them by hand. That region took two hours forty minutes to recover, while other regions recovered by 19:48.
Why did Cloudflare services fail when Google Cloud went down?
Cloudflare's Workers KV kept its source of truth on a third-party cloud provider. According to Cloudflare, Workers KV lost 90% of requests, and Access, WARP, Workers AI, Pages, Stream and Turnstile went down with it for 2 h 28 min. The episode notes that Workers KV was mid-migration to Cloudflare's own R2 storage.
How can teams stop a bad configuration row from crashing a service in every region?
Three practices follow from this incident. Keep new code behind a feature flag that ships off, check for null where data enters, and stage quota data instead of replicating it globally at once. Google's report says a flag would have caught the bug in staging, and its fixes include failing open and exponential backoff.
Do the Google Cloud and Cloudflare postmortems from June 12 set a good example for incident reports?
The Daily Diff's verdict was SHIP IT. Both postmortems landed within thirty hours and both blamed their own companies. Google's fixes were concrete, namely fail open, flags off by default and exponential backoff. The takeaway for Monday is to ship new code behind a flag that starts off.
Transcript
A blank field, a null pointer, every region
0:00 A policy row with blank fields hits a null pointer, and Google Cloud crashes in every region at once — and Cloudflare falls with it, then calls Google "a third-party provider". June 12, 2025, 17:49 UTC. Google's report: Service Control, the binary that approves every API request, in a crash loop for three hours. Cloudflare's, the same evening: Workers KV, ninety percent failing, two hours twenty-eight.
0:23 How it happened, why one blank field went global in seconds, and who gets the blame. This is The Daily Diff, postmortem. May 29. Service Control gets a new quota check, shipped region by region with a
Timeline: May 29 → 17:45 → crash loop → red button → 17:52 Cloudflare
0:35 red button — but the new code path never runs during rollout; nothing triggers it yet. No null check. No feature flag. 17:45. A policy change with blank fields lands in the Spanner table. Quota is global, so the row replicates everywhere within seconds. Every Service Control binary reads it, hits the null pointer, crashes, restarts, reads it again. Every API call: 503.
0:55 Google is fast: triage in two minutes, root cause in ten. The red button is out in forty, and small regions recover first. The status page posts its first update an hour in, because it runs on Google Cloud. 17:52. Cloudflare's WARP team sees new devices fail to register. Workers KV, the store half of Cloudflare uses for config and identity, lives on a third-party cloud. Access fails every login — by design, it fails closed.
1:20 Workers AI fails every inference. 19:11, Gergely Orosz: two independent clouds down at once, never seen before. 19:32, Google support tells a user there are no known disruptions — try clearing your cookies. Everywhere else recovers by 19:48.
us-central1: the herd effect
1:34 us-central1 doesn't: restarting tasks stampede the same Spanner table, no randomized backoff, so Google throttles them by hand. Two hours forty. One: the policy check sits inside the request path; when the check dies,
Mechanism: check in the request path, global replication, one vendor
1:46 the API dies. Two: quota data goes global with no staging; a bad row is a global row. Three: Cloudflare knew. Workers KV was mid-migration to its own R2, down to one provider — the postmortem calls it a gap in coverage.
git blame — the split
2:00 git blame. Google, fifty-five percent: no null check, no feature flag, no backoff — their report says a flag would have caught it in staging. Global replication, twenty: one row, every region, seconds. Cloudflare, twenty: half a product line on one vendor's store, and a postmortem that never says Google. The status page, five, for living on what it reports on. Blast radius: seventy Google Cloud products, ten Workspace apps,
Blast radius
2:23 every region. Three hours. At Cloudflare: Access, WARP, Workers AI, Pages, Stream — two and a half. Hacker News, fourteen hundred points; top comment: the status page is green.
Verdict + the Monday line
2:33 Verdict, postmortem: ship it. Both postmortems land within thirty hours, both blame themselves, and Google's fixes are concrete: fail open, flags off by default, exponential backoff. The Monday line: new code behind a flag that ships off, and a null check where the data comes in. Send me the incident you're still not allowed to talk about, in the comments, or at the daily diff dot dev. And that's the diff for today.
2:53 I'm Niko from Axrisi. Merge responsibly.
Sources
- Google Cloud Service Health, Incident Report (Jun 13, 2025) + Mini Incident Report (Jun 12)status.cloud.google.com
- Cloudflare, "Cloudflare service outage June 12, 2025" (Jun 12, 2025, 22:00 UTC)blog.cloudflare.com
- Cloudflare Status, "Broad Cloudflare service outages" (18:19–21:31 UTC)www.cloudflarestatus.com
- Gergely Orosz, 19:11 UTC, "Their infra is fully independent AFAIK"x.com
- @Google support, 19:32 UTC, "there aren't any known service disruptions"x.com
- Thomas Kurian (Google Cloud CEO), Jun 13 01:35 UTCx.com
- Hacker News, "GCP Outage" (1,468 points)news.ycombinator.com
- Hacker News, "Cloudflare was down" (341 points)news.ycombinator.com
- Hacker News, "Google Cloud Incident Report – 2025-06-13" (209 points)news.ycombinator.com



