A deploy to 7 of 8 servers lost $460 million. 45 minutes.
August 1, 2012, 9:30 a.m.
August 1, 2012, 9:30 a.m. ET: Knight Capital, the market maker behind roughly a tenth of all U.S. stock trading, has just deployed new order-routing code to seven of its eight SMARS servers. The eighth still runs "Power Peg", code retired in 2003, behind the very flag the new feature reuses. For 45 minutes it sends child orders that never stop: 4 million executions, 397 million shares, about $7 billion of positions, a $460 million loss. 97 warning e-mails had arrived before the open. Postmortem, from the SEC's own order: the manual deploy nobody reviewed, the flag with two meanings, the stop-counter moved in 2005 and never retested, the kill switch that did not exist, and the "fix" in a live market that made it worse. Verdict on the fix: REVERT.
Read the written edition (English) ↗
What this video covers
- Jul 27–31, 2012: RLP code copied by hand to 7 of 8 SMARS servers; no second reviewer, no written procedure
- Aug 1, 8:01 a.m.: 97 "Power Peg disabled" e-mails before the open; not designed as alerts, not read
- Aug 1, 9:30–10:15: 212 orders become 4M executions in 154 stocks; $3.5B long, $3.15B short; $460M lost; uninstalling the new code makes it worse
- Aug 2 – Dec 19: shares −63% to $2.58, $400M rescue at $1.50/share, merger with GETCO; Oct 2013: $12M SEC penalty, first Market Access Rule case
Questions and answers
What caused the Knight Capital trading loss in August 2012?
Knight Capital deployed new order-routing code by hand to seven of its eight SMARS servers. The eighth still ran Power Peg, code retired in 2003, behind the same flag the new feature reused. According to the SEC order, it sent child orders for 212 customer orders and never stopped, producing 4 million executions and 397 million shares in about 45 minutes, and a $460 million loss.
Why is reusing an old feature flag for new code dangerous?
The same flag value then means two different things on different servers. In Knight's case the new feature reused Power Peg's old flag, so the same setting enabled new behavior on seven servers and the retired Power Peg code on the eighth. Power Peg's stop counter had also been moved in 2005 and never retested, so the retired code looped forever.
Why didn't warnings or controls stop Knight Capital's runaway orders sooner?
Ninety-seven emails reading Power Peg disabled arrived before the open, but they were not designed as alerts and nobody read them. Nothing compared orders out against orders in, and there was no kill switch. The risk monitor was a screen watched by humans that did not show the limits. Engineers uninstalled the new code from the seven good servers, which switched Power Peg on everywhere.
What happened to Knight Capital after the 2012 trading glitch?
Shares fell 63% to $2.58, and Knight took a $400 million rescue at $1.50 a share. It agreed to a merger with GETCO by Christmas 2012. In 2013 the SEC fined Knight $12 million in its first Market Access Rule case and wrote the postmortem itself, about fourteen months after the incident.
What should engineering teams change after the Knight Capital incident?
The Daily Diff's verdict was REVERT. The episode's changes are to give each new feature a new flag, delete dead code rather than disable it, and give anything that sends orders an off switch. It also noted that Knight never wrote its own postmortem. The SEC did, and the manual deploy had no second reviewer and no written procedure.
Transcript
0:00 A trading firm deploys new code to seven of eight servers, and the eighth buys and sells nearly seven billion dollars of stock in forty-five minutes, which is a lot, even for a Wednesday. August 1st, 2012. Knight Capital, a tenth of all U.S. stock trading, reports a 440 million dollar loss. The SEC later counts more than 460, fines Knight twelve million, and writes the postmortem itself: ten pages, zero adjectives.
0:25 How it happens, why it is possible, and who actually gets the blame. This is The Daily Diff, postmortem. July 27th. The New York Stock Exchange launches a retail liquidity program on August 1st, so Knight copies new code onto SMARS, its order router, a few servers a day, by hand. One technician, eight servers, seven copies. Nobody checks; no rule says anyone has to. 8:01 a.m. An internal system starts emailing staff an error that reads: Power
0:51 Peg disabled. Ninety-seven emails before the open. They are not designed as alerts, so nobody reads them, which is a design decision with a price tag. 9:30, the market opens. Seven servers run the new code. The eighth sees the same flag and runs Power Peg instead: code Knight retired in 2003 and never deleted. It sends child orders for 212 customer orders, and never stops.
1:13 Four million executions. 397 million shares. A holding account swells past its two-million-dollar limit, which is wired to nothing. The engineers, in a live market, uninstall the new code from the seven good servers, which switches Power Peg on everywhere. Around 10:15 it stops: three and a half billion long, three point one five billion short.
1:31 One: Power Peg's stop condition, the counter that says the parent order is filled, is moved in 2005 and never retested, so the retired code loops forever. Two: the new feature reuses Power Peg's old flag, so the same yes means two things on two servers. Three: no kill switch. Nothing compares orders out against orders in; nothing halts SMARS. The risk monitor is a screen watched by humans; it does not show the limits. git blame: a flag reused instead of a new one, dead code callable for nine
2:00 years, and a manual deploy with no second pair of eyes. Not the technician. The SEC order never names him, and neither will I. Blast radius: 460 million dollars, roughly ten million a minute. Shares down 63 percent to two dollars fifty-eight, a 400 million dollar rescue at a dollar fifty a share, a merger with Getco by Christmas. Hacker News, Thursday: how did it run 45 minutes without a human stepping in? It didn't. The humans made it worse.
2:25 Verdict, postmortem: revert. Knight never writes its own postmortem; the regulator does, fourteen months later, and the fix is a fine and a merger. Monday: new feature, new flag; dead code gets deleted, not disabled; anything that sends orders gets an off switch. Send me the incident you are still not allowed to talk about, in the comments, or at the daily diff dot dev. And that's the diff for today.
2:48 I'm Niko from Axrisi. Merge responsibly.
Sources
- SEC Administrative Order, Release No. 34-70694 (Oct 16, 2013), In the Matter of Knight Capital Americas LLCwww.sec.gov
- SEC press release 2013-222, "SEC Charges Knight Capital With Violations of Market Access Rule"www.sec.gov
- Knight Capital Group 8-K, Aug 2, 2012 (the $440M statement)www.sec.gov
- Knight Capital Group 8-K, Aug 6, 2012 (the $400M convertible preferred)www.sec.gov
- Knight / GETCO merger announcement, Dec 19, 2012www.sec.gov
- NYT DealBook, "Knight Capital Says Trading Glitch Cost It $440 Million" (Aug 2, 2012)archive.nytimes.com
- Hacker News, Aug 2, 2012 (73 points)news.ycombinator.com
- Hacker News, "The $440M software error at Knight Capital" (294 points)news.ycombinator.com



