One extra field crashed 8.5 million PCs. 78 minutes.

What this video covers

• Feb 28 – Apr 24, 2024: sensor 7.11 ships the IPC Template Type (21 fields declared, 20 supplied); stress test passes Mar 5; four instances work because field 21 is a wildcard
• Jul 19, 04:09 UTC: two new Template Instances; the Content Validator's logic error lets the non-wildcard one through; out-of-bounds read in the kernel; 05:27 UTC reverted
• Jul 19–29: manual remediation (Safe Mode, delete the file, BitLocker key); ~97% of sensors back by Jul 24, ~99% by Jul 29
• Jul 25 – Aug 19: runtime bounds check, compile-time field-count validation, Validator checks, canary + deployment rings, customer control over Rapid Response Content, two independent reviews; full RCA with crash dump on Aug 6

Transcript

0:00 One security update carries twenty-one fields into a driver built for twenty, and eight and a half million Windows machines blue-screen before breakfast. July 19, 2024, 04:09 UTC. CrowdStrike pushes Channel File 291 to every Falcon sensor. Seventy-eight minutes later they pull it; by then airlines, hospitals and banks stare at the same blue rectangle. The CEO's first post says not a cyberattack, true, and not the point. How it happens, why it is possible, and who gets the blame.

0:27 This is The Daily Diff, postmortem. February 28th. Sensor 7.11 ships a new detector for named pipes. It declares twenty-one input fields; the code that feeds it builds an array of twenty. Nobody notices: for four months every rule leaves field twenty-one as a wildcard, and nobody reads a wildcard. A stress test passes, four rules ship, all fine. July 19th, 04:09. Two new rules; one puts a real pattern in field twenty-one.

0:51 The cloud validator counts twenty-one against the declaration and says fine. Every online Windows host downloads it at once, because there is no canary ring. The sensor reads slot twenty-one, which is not its memory, and the kernel does what kernels do. 05:27, reverted. Too late for anyone who rebooted: the driver loads early in boot, reads the same file, and crashes again. Boot loop. Why is this possible? The rules are configuration pushed from the cloud, so not code skips the code

1:16 pipeline. The interpreter has no bounds check; it trusts the validator, and the validator trusts the declaration. And it all runs in kernel mode, where a bad pointer is not an exception but a page fault Windows says cannot be protected by try-except. Patrick Wardle reads it off a crash dump that afternoon: index zero x fourteen, slot twenty-one. The fix is manual. Safe Mode, delete C-00000291, reboot. With BitLocker, first type a forty-eight-digit

1:43 recovery key that lives on a server which is also blue. git blame. CrowdStrike, sixty-five percent: the RCA lists six findings, and finding six is a full sentence: Template Instances should have staged deployment. The kernel-mode design, twenty-five: CrowdStrike says Windows cannot yet host security outside the kernel, Microsoft says not a Microsoft incident, and the page fault does not care.

2:05 The Friday, ten. The commit message says not code. Blast radius: eight and a half million devices, under one percent of Windows. Delta cancels seven thousand flights and sues for five hundred million, Microsoft included. Insurers put the Fortune 500 bill near five billion. Hacker News: forty-five hundred points. The irony: an update meant to detect novel attack techniques delivered one. Verdict, postmortem: ship it, narrowly.

2:27 Bounds check in six days, staged rings, customer control over content updates, two outside reviews. Narrowly, because all of that was standard practice already. Monday: whatever your agent pulls from the cloud gets a canary ring. Configuration is code the moment something parses it where it cannot throw. Send me the incident you are still not allowed to talk about, in the comments, or at the daily diff dot dev. And that's the diff for today.

2:48 I'm Niko from Axrisi. Merge responsibly.

Sources

1. CrowdStrike, Root Cause Analysis — Channel File 291 (Aug 6, 2024, 12 pp.)www.crowdstrike.com
2. CrowdStrike, Executive Summary of the RCA (Aug 6, 2024)www.crowdstrike.com
3. CrowdStrike, Preliminary Post Incident Review (Jul 24, 2024)www.crowdstrike.com
4. CrowdStrike, Remediation and Guidance Hub — Channel File 291 (the workaround steps)www.crowdstrike.com
5. CrowdStrike, tech alert of Jul 19, 2024 (archived: Safe Mode → delete C-00000291*.sys → BitLocker key)web.archive.org
6. Microsoft, David Weston, "Helping our customers through the CrowdStrike outage" (Jul 20, 2024; the 8.5 million figure)blogs.microsoft.com
7. Delta Air Lines, Form 8-K (Aug 8, 2024; ~7,000 cancellations, at least $500 million)www.sec.gov
8. Adam Meyers (CrowdStrike SVP), written testimony, House Homeland Security Subcommittee (Sep 24, 2024)homeland.house.gov
9. George Kurtz on X, Jul 19, 2024, 09:45 UTCx.com
10. Patrick Wardle on X, Jul 19, 2024 (the crash-dump index 0x14)x.com
11. Hacker News, Jul 19, 2024 — "CrowdStrike Update: Windows Bluescreen and Boot Loops" (4,489 points)news.ycombinator.com
12. The Guardian, Jul 24, 2024 — "CrowdStrike global outage to cost US Fortune 500 companies $5.4bn" (Parametrix estimate)www.theguardian.com
13. Hacker News, Jul 21, 2024 — "Initial details about why CrowdStrike's CSAgent.sys crashed" (519 points)news.ycombinator.com
14. Wikipedia — 2024 CrowdStrike-related IT outagesen.wikipedia.org