A reboot sent Telstra to 2006. Nine million phones.
An engineer in Melbourne powers a timing chassis back on at 2:50 a.m., and by breakfast Australia's biggest mobile network agrees that it is November 2006.
An engineer in Melbourne powers a timing chassis back on at 2:50 a.m., and by breakfast Australia's biggest mobile network agrees that it is November 2006. July 8, 2026: a GPS card in Telstra's NTP chassis reboots during a power-supply repair, its firmware has never had the GPS week-rollover update, so it picks the old epoch and lands 1,024 weeks in the past. Because an October 2025 workaround had made that card the network's only stratum-1 source — and a 2020 switch to NTP peering had left every other source downstream of it — the wrong date wins the vote. Handover nodes reset from 4 a.m., 45 % of all calls and data sessions are affected, nearly 9 million customers lose calls, texts or data, 604 Triple Zero calls error, Victoria's regional trains stop, and most services come back at 12:12. Telstra's own alarms sat on a platform checked in business hours; the company noticed because customers were googling "Telstra outage".
Read the written edition (English) ↗
What this video covers
- A GPS card wakes up in November 2006
- Timeline: 2010 design → 2020 chassis → Oct 2025 GPS card → 02:50
- The morning: 04:00 MME resets → 04:20 alarm storm → 12:12 restored
- Mechanism: NTP votes on agreement, not plausibility
- git blame — Telstra 55 · peering 25 · firmware 15 · the power supply 5
Transcript
A GPS card wakes up in November 2006
0:00 An engineer in Melbourne powers a timing chassis back on at ten to three in the morning, and by breakfast Australia's biggest mobile network agrees it is November two thousand six. Telstra's outside report says the date left one GPS card and the whole network voted for it. Nearly nine million customers lose calls, texts or data, and Victoria's regional trains stop for the day. How it happened, why one card outranked the national clock,
0:23 and who gets the blame. This is The Daily Diff, postmortem.
Timeline: 2010 design → 2020 chassis → Oct 2025 GPS card → 02:50
0:28 Twenty ten. The design is textbook. Three national time sources feed two servers, those feed three below, and every node takes time only from above. The auditors call it fit for purpose. Twenty twenty. New chassis. It can't feed its own lower server, so Sydney and Melbourne get cross-wired and each site drops to one reference. Then everything is switched to peering, where any server may take time from any
0:50 server it finds. The words timing loop appear in no document. October twenty twenty-five. Melbourne keeps losing Sydney, and its alarms say stratum five, meaning it takes time from its own customers. Nobody files a ticket. Instead the engineers switch on a GPS card that has sat idle for five years. The alarms stop.
1:07 Melbourne is now stratum one, the rank of the national institute, and nobody reviews it. Eighth of July, two fifty in the morning. A power supply needs replacing, so the chassis reboots, and the GPS card with it. Its firmware was never updated, so it picks the old epoch and lands one thousand twenty-four weeks in the past. Melbourne is the top of the tree, and downstream starts agreeing.
The morning: 04:00 MME resets → 04:20 alarm storm → 12:12 restored
1:26 Four a.m., the handover nodes start resetting. Four twenty, an alarm storm, on a platform whose alarms are read in business hours. Four twenty-nine, Telstra notices, because customers are googling Telstra outage. The engineers who did the change are on mandatory rest, and finding the date takes hours. By twelve twelve most customers are back.
Mechanism: NTP votes on agreement, not plausibility
1:47 Why was it possible? NTP has two defences. Lower stratum wins, and outliers get voted out. The GPS card made the broken source the highest rank, and every source that could disagree sat downstream, repeating it. NTP never asks whether a date is plausible, only whether the others agree. The protocol worked.
2:06 The architecture did not.
git blame — Telstra 55 · peering 25 · firmware 15 · the power supply 5
2:07 git blame. Telstra, fifty-five percent. Nobody owned the clock, so nobody reviewed the change, and two vendor bulletins about this exact bug went unread. Peering mode, twenty-five, for letting servers vote with their own echo. The firmware, fifteen, published and never installed. Five percent to the power supply, for picking ten to three on a Wednesday. Blast radius. Forty-five percent of all calls and data sessions that day,
Blast radius: 45 % of sessions, a $30M fine for a $30K box
2:30 the CEO tells the Senate. The fine on the table is thirty million dollars, for a free update missing from a thirty thousand dollar box. The card had survived the real rollover in twenty nineteen because nobody turned it off. The repair killed it. Verdict, postmortem: SHIP IT, on the response.
Verdict + the Monday line: one clock wearing three hats
2:46 Telstra hires outside auditors, publishes the report unedited, with the line that it never treated time as critical, and moves all three sites off the old servers before it lands. Monday line: run an NTP trace and count how many of your time sources are one clock wearing three hats. Send me the incident you're still not allowed to talk about, in the comments, or at thedailydiff.dev. And that's the diff for today.
3:10 I'm Niko from Axrisi. Merge responsibly.
Sources
- Technology Audit Partners, "Findings Report on the July 8th Telstra Outage" (published by Telstra, Sep 2, 2026)www.telstra.com.au
- Vicki Brady (Telstra CEO), "What we've learned from the external investigation of our July outage" (Sep 2, 2026)www.telstra.com.au
- Netnod, Sven-Christian Ebenhag, "Telstra outage: The night a network decided the year was 2006" (Sep 17, 2026)www.netnod.se
- Hacker News on the Netnod post (93 points)news.ycombinator.com
- The Guardian, Senate inquiry: 45 % of sessions, ~9M customers, 604 Triple Zero errors (Jul 17, 2026)www.theguardian.com
- ABC News, "Telstra recently warned about timing issue linked to national outage" (Jul 10, 2026)www.abc.net.au
- The Register, the day itself (Jul 8, 2026)www.theregister.com
- ACS Information Age, the $30,000 box and the $30M fine (Jul 14, 2026)ia.acs.org.au
