# Google Cloud crashed on a blank field. Three hours.

Published: 2026-09-19

A policy row with a few blank fields hits a null pointer, and Google Cloud crashes in every region at once — then Cloudflare falls with it. June 12, 2025, 17:49 UTC: Service Control, the binary that approves every Google Cloud API request, reads a quota-policy change with "unintended blank fields", hits a code path shipped two weeks earlier with no null check and no feature flag, and goes into a crash loop in every region — the row had replicated globally within seconds. External APIs return 503 for three hours; us-central1 takes 2 h 40 min because the restarting tasks stampede the same Spanner table with no randomized backoff. Three minutes into Google's outage, Cloudflare's Workers KV — whose source of truth sits on "a third-party cloud provider" — loses 90 % of requests, and Access, WARP, Workers AI, Pages, Stream and Turnstile go down with it for 2 h 28 min. Google's status page posts its first update an hour late, because it runs on Google Cloud.

Canonical: https://thedailydiff.dev/video/2026-09-19-google-cloud-null-pointer/

## What this video covers

- A blank field, a null pointer, every region
- Timeline: May 29 → 17:45 → crash loop → red button → 17:52 Cloudflare
- us-central1: the herd effect
- Mechanism: check in the request path, global replication, one vendor
- git blame — the split

## Chapters

- 0:00 A blank field, a null pointer, every region
- 0:32 Timeline: May 29 → 17:45 → crash loop → red button → 17:52 Cloudflare
- 1:32 us-central1: the herd effect
- 1:43 Mechanism: check in the request path, global replication, one vendor
- 2:00 git blame — the split
- 2:20 Blast radius
- 2:33 Verdict + the Monday line

## Transcript

### A blank field, a null pointer, every region

0:00 A policy row with blank fields hits a null pointer, and Google Cloud crashes in every region at once — and Cloudflare falls with it, then calls Google "a third-party provider". June 12, 2025, 17:49 UTC. Google's report: Service Control, the binary that approves every API request, in a crash loop for three hours. Cloudflare's, the same evening: Workers KV, ninety percent failing, two hours twenty-eight.

0:23 How it happened, why one blank field went global in seconds, and who gets the blame. This is The Daily Diff, postmortem. May 29. Service Control gets a new quota check, shipped region by region with a

### Timeline: May 29 → 17:45 → crash loop → red button → 17:52 Cloudflare

0:35 red button — but the new code path never runs during rollout; nothing triggers it yet. No null check. No feature flag. 17:45. A policy change with blank fields lands in the Spanner table. Quota is global, so the row replicates everywhere within seconds. Every Service Control binary reads it, hits the null pointer, crashes, restarts, reads it again. Every API call: 503.

0:55 Google is fast: triage in two minutes, root cause in ten. The red button is out in forty, and small regions recover first. The status page posts its first update an hour in, because it runs on Google Cloud. 17:52. Cloudflare's WARP team sees new devices fail to register. Workers KV, the store half of Cloudflare uses for config and identity, lives on a third-party cloud. Access fails every login — by design, it fails closed.

1:20 Workers AI fails every inference. 19:11, Gergely Orosz: two independent clouds down at once, never seen before. 19:32, Google support tells a user there are no known disruptions — try clearing your cookies. Everywhere else recovers by 19:48.

### us-central1: the herd effect

1:34 us-central1 doesn't: restarting tasks stampede the same Spanner table, no randomized backoff, so Google throttles them by hand. Two hours forty. One: the policy check sits inside the request path; when the check dies,

### Mechanism: check in the request path, global replication, one vendor

1:46 the API dies. Two: quota data goes global with no staging; a bad row is a global row. Three: Cloudflare knew. Workers KV was mid-migration to its own R2, down to one provider — the postmortem calls it a gap in coverage.

### git blame — the split

2:00 git blame. Google, fifty-five percent: no null check, no feature flag, no backoff — their report says a flag would have caught it in staging. Global replication, twenty: one row, every region, seconds. Cloudflare, twenty: half a product line on one vendor's store, and a postmortem that never says Google. The status page, five, for living on what it reports on. Blast radius: seventy Google Cloud products, ten Workspace apps,

### Blast radius

2:23 every region. Three hours. At Cloudflare: Access, WARP, Workers AI, Pages, Stream — two and a half. Hacker News, fourteen hundred points; top comment: the status page is green.

### Verdict + the Monday line

2:33 Verdict, postmortem: ship it. Both postmortems land within thirty hours, both blame themselves, and Google's fixes are concrete: fail open, flags off by default, exponential backoff. The Monday line: new code behind a flag that ships off, and a null check where the data comes in. Send me the incident you're still not allowed to talk about, in the comments, or at the daily diff dot dev. And that's the diff for today.

2:53 I'm Niko from Axrisi. Merge responsibly.

## Sources

- [Google Cloud Service Health, Incident Report (Jun 13, 2025) + Mini Incident Report (Jun 12)](https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW) — status.cloud.google.com
- [Cloudflare, "Cloudflare service outage June 12, 2025" (Jun 12, 2025, 22:00 UTC)](https://blog.cloudflare.com/cloudflare-service-outage-june-12-2025/) — blog.cloudflare.com
- [Cloudflare Status, "Broad Cloudflare service outages" (18:19–21:31 UTC)](https://www.cloudflarestatus.com/incidents/25r9t0vz99rp) — www.cloudflarestatus.com
- [Gergely Orosz, 19:11 UTC, "Their infra is fully independent AFAIK"](https://x.com/GergelyOrosz/status/1933240698716729511) — x.com
- [@Google support, 19:32 UTC, "there aren't any known service disruptions"](https://x.com/Google/status/1933246051512644069) — x.com
- [Thomas Kurian (Google Cloud CEO), Jun 13 01:35 UTC](https://x.com/ThomasOrTK/status/1933337436970709493) — x.com
- [Hacker News, "GCP Outage" (1,468 points)](https://news.ycombinator.com/item?id=44260810) — news.ycombinator.com
- [Hacker News, "Cloudflare was down" (341 points)](https://news.ycombinator.com/item?id=44261064) — news.ycombinator.com
- [Hacker News, "Google Cloud Incident Report – 2025-06-13" (209 points)](https://news.ycombinator.com/item?id=44274563) — news.ycombinator.com