Eleven lines of JavaScript broke the internet. One unpublish.
March 22, 2016, 21:30 UTC: a developer unpublishes all 273 of his npm packages after npm hands the package name "kik" to Kik, the messaging app, whose patent agent had promised "lawyers banging on your door".
March 22, 2016, 21:30 UTC: a developer unpublishes all 273 of his npm packages after npm hands the package name "kik" to Kik, the messaging app, whose patent agent had promised "lawyers banging on your door". One of the 273 is left-pad: eleven lines that put spaces in front of a string, downloaded 2.5 million times a month. Within minutes Babel, React and Node builds fail worldwide, hundreds per minute. A fork appears in ten minutes and changes nothing, because Babel pulls left-pad through line-numbers, which pins 0.0.3 exactly. At 23:55 npm restores the deleted version from a backup — the first "un-un-publish" in the registry's history — and a week later ships the 24-hour unpublish rule. Postmortem, from npm's own account and the e-mail thread Kik published: the registry that let any author delete any version with no dependents check, the transitive tree resolved live on every install, no lockfile by default. Git blame: 60 npm / 25 Kik / 15 the dependency habit. Verdict on the fix: SHIP IT.
Read the written edition (English) ↗
What this video covers
- Mar 11: Kik's patent agent asks Azer Koçulu to rename `kik`; an hour later: lawyers "banging on your door and taking down your accounts"; Azer: $30,000; Kik e-mails npm support
- Mar 18–20: npm's CEO transfers the name to Kik ("would reasonably expect it to be related to kik.com"); Azer asks for all his modules deleted
- Mar 22, 21:30–23:55 UTC: 273 packages unpublished; [email protected] 404s in Babel, Atom, Node projects; fork 1.0.0 at 21:42; npm's CTO tweets "un-un-publishing" at 23:03; 0.0.3 back from backup at 23:55 — 2.5 hours
- Mar 23 and Mar 29: npm's postmortem ("we dropped the ball") and the new policy: unpublish only within 24 hours, security placeholders for removed names (72 h since 2020)
Transcript
0:00 One developer deletes eleven lines of JavaScript, and within minutes Babel, React and Node builds are failing worldwide, hundreds a minute. The function is left-pad. It pads a string with spaces, and by the end of the evening npm does the one thing it says it cannot do: republish a deleted version. How it happens, why it is possible, and who gets the blame. This is The Daily Diff, postmortem. March 11th. A patent agent at Kik, the messaging app,
0:26 emails Azer Koçulu: rename your npm package, kik is our brand. Azer says no. An hour later: our trademark lawyers are going to be banging on your door and taking down your accounts. Azer names a price, thirty thousand dollars, and Kik forwards the thread to npm support. March 18th. npm's CEO rules that anyone typing npm install kik expects the messaging
0:48 app, and hands the name over. Azer asks for everything of his deleted. Two days later he does it himself: two hundred and seventy-three packages. March 22nd, 21:30 UTC. Every build that reaches for left-pad gets a 404. Ten minutes later a stranger republishes it as 1.0.0, and nothing changes, because Babel and Atom load it through a package called line-numbers, which wants 0.0.3 exactly.
1:13 At 23:03 npm's CTO tweets: un-un-publishing. At 23:55, 0.0.3 is back from a backup. Two and a half hours. Why can one person do this? Three reasons. Any version on npm is deletable by its author, instantly, with no check of who depends on it. Second, dependencies are transitive: Babel does not know left-pad exists, it knows line-numbers, and the whole tree is resolved live on every install.
1:36 Third, in 2016 nothing pins that tree by default; lockfiles are opt-in, so every CI run re-asks the internet what eleven lines of code look like. One registry, run by nobody who depends on it. git blame. npm, sixty percent: its own postmortem says unrestricted unpublishing caused the pain, and, quote, we dropped the ball. Kik, twenty-five: a patent agent, not a lawyer, promising lawyers at the door over a name Kik had already decided not to use. The dependency habit, fifteen: everyone who npm-installed eleven lines instead
2:06 of typing them. Including Kik, whose builds break too. Blast radius: two hundred and seventy-three packages pulled, one of them downloaded two and a half million times a month. The fix takes two and a half hours; the function takes eleven lines. On Hacker News, a post asking whether we have forgotten how to program hits seventeen hundred points, mostly from people whose builds are still red. Verdict, postmortem: ship it.
2:26 npm restores from backup, posts the postmortem next day, and within a week ships the rule that still stands: you can unpublish for twenty-four hours, after that you talk to a human. Monday: commit your lockfile, and if a function is eleven lines, it is a paragraph, not a dependency. Send me the incident you are still not allowed to talk about, in the comments, or at the daily diff dot dev. And that's the diff for today.
2:48 I'm Niko from Axrisi. Merge responsibly.
Sources
- npm, Inc., "kik, left-pad, and npm" (Mar 23, 2016)blog.npmjs.org
- npm, Inc., "changes to npm's unpublish policy" (Mar 29, 2016)blog.npmjs.org
- Azer Koçulu, "I've Just Liberated My Modules" (Mar 22, 2016; archive copy)web.archive.org
- Mike Roberts (Kik), "A discussion about the breaking of the Internet" — the full e-mail thread (Mar 23, 2016; archive copy)web.archive.org
- Laurie Voss (npm CTO) on X, Mar 22, 2016, 23:03 UTCx.com
- left-pad 0.0.3 on the npm registry (the eleven lines)registry.npmjs.org
- Hacker News, Mar 22, 2016 — "I've Just Liberated My Modules" (1,573 points)news.ycombinator.com
- Hacker News, Mar 23, 2016 — "NPM and Left-Pad: Have We Forgotten How to Program?" (1,725 points)news.ycombinator.com
- Hacker News, Mar 29, 2016 — "Changes to Npm's unpublish policy" (320 points)news.ycombinator.com
- The Register, Mar 23, 2016 — "How one developer just broke Node, Babel and thousands of projects in 11 lines of JavaScript"www.theregister.com
