A deploy to 7 of 8 servers lost $460 million. 45 minutes.
August 1, 2012, 9:30 a.m.
August 1, 2012, 9:30 a.m. ET: Knight Capital, the market maker behind roughly a tenth of all U.S. stock trading, has just deployed new order-routing code to seven of its eight SMARS servers. The eighth still runs "Power Peg", code retired in 2003, behind the very flag the new feature reuses. For 45 minutes it sends child orders that never stop: 4 million executions, 397 million shares, about $7 billion of positions, a $460 million loss. 97 warning e-mails had arrived before the open. Postmortem, from the SEC's own order: the manual deploy nobody reviewed, the flag with two meanings, the stop-counter moved in 2005 and never retested, the kill switch that did not exist, and the "fix" in a live market that made it worse. Verdict on the fix: REVERT.
Read the written edition (English) ↗
What this video covers
- Jul 27–31, 2012: RLP code copied by hand to 7 of 8 SMARS servers; no second reviewer, no written procedure
- Aug 1, 8:01 a.m.: 97 "Power Peg disabled" e-mails before the open; not designed as alerts, not read
- Aug 1, 9:30–10:15: 212 orders become 4M executions in 154 stocks; $3.5B long, $3.15B short; $460M lost; uninstalling the new code makes it worse
- Aug 2 – Dec 19: shares −63% to $2.58, $400M rescue at $1.50/share, merger with GETCO; Oct 2013: $12M SEC penalty, first Market Access Rule case
Transcript
0:00 A trading firm deploys new code to seven of eight servers, and the eighth buys and sells nearly seven billion dollars of stock in forty-five minutes, which is a lot, even for a Wednesday. August 1st, 2012. Knight Capital, a tenth of all U.S. stock trading, reports a 440 million dollar loss. The SEC later counts more than 460, fines Knight twelve million, and writes the postmortem itself: ten pages, zero adjectives.
0:25 How it happens, why it is possible, and who actually gets the blame. This is The Daily Diff, postmortem. July 27th. The New York Stock Exchange launches a retail liquidity program on August 1st, so Knight copies new code onto SMARS, its order router, a few servers a day, by hand. One technician, eight servers, seven copies. Nobody checks; no rule says anyone has to. 8:01 a.m. An internal system starts emailing staff an error that reads: Power
0:51 Peg disabled. Ninety-seven emails before the open. They are not designed as alerts, so nobody reads them, which is a design decision with a price tag. 9:30, the market opens. Seven servers run the new code. The eighth sees the same flag and runs Power Peg instead: code Knight retired in 2003 and never deleted. It sends child orders for 212 customer orders, and never stops.
1:13 Four million executions. 397 million shares. A holding account swells past its two-million-dollar limit, which is wired to nothing. The engineers, in a live market, uninstall the new code from the seven good servers, which switches Power Peg on everywhere. Around 10:15 it stops: three and a half billion long, three point one five billion short.
1:31 One: Power Peg's stop condition, the counter that says the parent order is filled, is moved in 2005 and never retested, so the retired code loops forever. Two: the new feature reuses Power Peg's old flag, so the same yes means two things on two servers. Three: no kill switch. Nothing compares orders out against orders in; nothing halts SMARS. The risk monitor is a screen watched by humans; it does not show the limits. git blame: a flag reused instead of a new one, dead code callable for nine
2:00 years, and a manual deploy with no second pair of eyes. Not the technician. The SEC order never names him, and neither will I. Blast radius: 460 million dollars, roughly ten million a minute. Shares down 63 percent to two dollars fifty-eight, a 400 million dollar rescue at a dollar fifty a share, a merger with Getco by Christmas. Hacker News, Thursday: how did it run 45 minutes without a human stepping in? It didn't. The humans made it worse.
2:25 Verdict, postmortem: revert. Knight never writes its own postmortem; the regulator does, fourteen months later, and the fix is a fine and a merger. Monday: new feature, new flag; dead code gets deleted, not disabled; anything that sends orders gets an off switch. Send me the incident you are still not allowed to talk about, in the comments, or at the daily diff dot dev. And that's the diff for today.
2:48 I'm Niko from Axrisi. Merge responsibly.
Sources
- SEC Administrative Order, Release No. 34-70694 (Oct 16, 2013), In the Matter of Knight Capital Americas LLCwww.sec.gov
- SEC press release 2013-222, "SEC Charges Knight Capital With Violations of Market Access Rule"www.sec.gov
- Knight Capital Group 8-K, Aug 2, 2012 (the $440M statement)www.sec.gov
- Knight Capital Group 8-K, Aug 6, 2012 (the $400M convertible preferred)www.sec.gov
- Knight / GETCO merger announcement, Dec 19, 2012www.sec.gov
- NYT DealBook, "Knight Capital Says Trading Glitch Cost It $440 Million" (Aug 2, 2012)archive.nytimes.com
- Hacker News, Aug 2, 2012 (73 points)news.ycombinator.com
- Hacker News, "The $440M software error at Knight Capital" (294 points)news.ycombinator.com
