# Unjiniyela usule i-database yokukhiqiza ye-GitLab. 300 gigabytes.

Published: 2026-09-10

NgoJanuwari 31, 2017, 23:27 UTC: unjiniyela we-GitLab, elwa nekhophi eliphukile ekupheleni kobusuku obude, ususa uhla lwemibhalo yedatha ye-PostgreSQL ku-db1 esikhundleni se-db2. I-db1 ingeyokuqala. Cishe ama-300 GB we-database ye-GitLab.com ashabalale ngesekhondi noma amabili, kanti kunezindlela eziyisihlanu zokwenza ikhophi yasenqolobaneni nezokuphendula, ayikho esebenzayo. Ukuhlolwa kokwenzekile: umugqa wesikhathi kusukela ekukhuphukeni kwe-spam kuya egameni elingelona, ​​kungani i-pg\_basebackup yabukeka ibambekile, kungani i-pg\_dump ibihluleka buthule (9.2 binaries ku-9.6 database, ama-imeyili okuhluleka abuyiselwe yi-DMARC), ukubuyisela kwamahora ayi-18 kusuka kusifinyezo se-staging eseneminyaka engu-6 esakazwa bukhoma ku-YouTube, nokuthi ubani ngempela okumele asolwe. Isinqumo ngempendulo: SHIP IT.

Canonical: https://thedailydiff.dev/zu/video/2026-09-10-gitlab-rm-rf/

## Lokho okuhlanganiswa yile vidiyo

- Jan 31, 2017: rm -Rvf kuhla lwemibhalo yedatha yokuqala; ~300 GB kususiwe, 4.5 GB kusale
- 5 kwabangu-5 abenzi bekhophi yasenqolobaneni bayahluleka: ibhakede le-S3 elingenalutho (ukungafani kwenguqulo ye-pg\_dump), akukho zifinyezo ze-Azure ku-DB, ikhophi esusiwe, ikhophi ye-LVM yansuku zonke ngaphandle kwama-webhook
- Feb 1, 18:00 UTC: I-GitLab.com ibuyile kusuka kusifinyezo esenziwe ngesandla esineminyaka engu-6; idokhumende elibukhoma, ukusakazwa bukhoma, ukuhlolwa kokwenzekile okungenasici ngohlu lokulungisa

## Umbhalo obhaliwe ohunyushwe

Kuhunyushwe kusuka ekulandiseni kokuqala kwesiNgisi. Umsindo namazwibela atholakalayo alawulwa yi-YouTube.

0:00 Unjiniyela kwa-GitLab usebenzisa i-rm -rf kuseva yedatha engafanele, futhi amakhulu amathathu ama-gigabytes we-GitLab dot com ayanyamalala ngesekhondi noma amabili, cishe isikhathi esithatha ukufunda igama leseva. Januwari 31st, 2017, 11:27 p.m. UTC. I-GitLab ihlenga ukuthi isule ngephutha idatha yokukhiqiza, ivula amanothi ayo esigameko ku-inthanethi, futhi isakaza ukubuyisela ku-YouTube, ukusakazwa okubili okubukhoma endaweni. Ngosuku olulandelayo, ngokubhala: kuzindlela eziyisihlanu zokwenza ikhophi yasenqolobaneni,

0:25 ayikho esebenza ngokuthembeka. Kwenzeka kanjani, kungani kungenzeka, nokuthi ubani ngempela okumele asolwe. Lena i-The Daily Diff, ukuhlolwa kokwenzekile. 5:20 p.m.: unjiniyela wenza isifinyezo sokukhiqiza ukuhlola i-load balancer ku-staging. 7 p.m.: i-spam ihlasela idatha, kanye nomsebenzi osusa umsebenzi we-GitLab owayebikiwe ngomonakalo. 11 p.m.: ikhophi isale ngemuva kakhulu kangangokuthi eyokuqala isivele isusile

0:48 ilog elidingayo; ukulungisa okuwukuphela kwakho ukuvala ikhophi bese ukopisha eyokuqala futhi. I-pg\_basebackup iyahluleka ngaphandle kwemiphumela. Empeleni ilindile, buthule, eyokuqala; akekho owaziyo lokho, futhi incwadi yemiyalelo ayisho. Unjiniyela, owayehlose ukuphuma ngo-11, unquma ukuthi uhla lwemibhalo yedatha engenalutho luyinkinga futhi ulususa. Ku-db1. Eyokuqala. Uyaqaphela ngemuva kwesekhondi noma amabili; cishe amakhulu amathathu ama-gigabytes,

1:11 4.5 asale. Amakhophi asenqolobaneni. Okokuqala: pg\_dump ku-S3, nsuku zonke. Ibhakede lingenalutho. Umsebenzi we-cron usebenza kuseva yohlelo lokusebenza ngaphandle kwedatha, ngakho iphakethe likhetha I-PostgreSQL 9.2 binaries ye-9.6 database, iyahluleka, futhi ithumela ama-imeyili ukuhluleka, okubuyiselwa ngenxa yokushoda kwe-DMARC. Okwesibili: I-Azure disk snapshots, enikwe amandla kumaseva amafayela, hhayi ama-database.

1:32 Okwesithathu: ikhophi, esuswe ngamabomu ngehora eledlule. Okwesine: isifinyezo sansuku zonke, esineminyaka engu-24, wonke ama-webhook asuswe yi- ukuvumelanisa kwe-staging. Okwesihlanu: isifinyezo esenziwe ngesandla kusukela ngo-5:20, sokuhlolwa okungahlobene. Leso siwinile. Ukubuyisela kusho ukukopisha idiski ye-staging emuva ekukhiqizeni nge-Azure eshibhile isitoreji ngamashumi ayisithupha megabits ngesekhondi: amahora ayishumi nesishiyagalombili. I-GitLab dot com ibuyile ngoFebhuwari 1st ngo-6 p.m. UTC, idatha endala ngamahora ayisithupha.

1:58 git blame: amagama amabili eseva ahlukene ngohlamvu olulodwa, nezinhlelo eziyisihlanu zokwenza ikhophi yasenqolobaneni akekho owake ake abuyisela kusuka kuzo. Hhayi unjiniyela. Ukuhlolwa kokwenzekile, okusayinwe yi-CEO, kumenza angaziwa, kufaka umbala obomvu kumkhombandlela wokukhiqiza, futhi kunikeza ukuqina kwedatha umnikazi, ngoba kuze kube manje bekungekho noyedwa. Ubukhulu bokulimala: amahora ayishumi nesishiyagalombili phansi, amahora ayisithupha edatha isuswe, cishe amaphrojekthi ayizinkulungwane ezinhlanu, amazwana ayizinkulungwane ezinhlanu,

2:18 abasebenzisi abasha abangamakhulu ayisikhombisa, nabantu abayizinkulungwane ezinhlanu ababuka ibha yokuqhubeka. I-Hacker News inika idokhumenti elibukhoma amaphuzu angu-1,162 futhi icaphuna olulodwa umugqa kubo: kwamakhophi asenqolobaneni ayisihlanu, akekho. Isinqumo, ukuhlolwa kokwenzekile: ship it, ngempendulo. Bayenza isigameko esidlangalaleni, basola inqubo, futhi bashicilela uhlu lokulungisa ngezinombolo zezinkinga. Isenzo sangoMsombuluko: buyisela ikhophi yasenqolobaneni. Uma ungakaze uyibuyisele, awunayo.

2:42 Ngithumele isigameko ongakavunyelwa ukukhuluma ngaso, kumazwana, noma ku-the daily diff dot dev. Futhi yilokho umehluko wanamuhla. NginguNiko wase-Axrisi. Hlanganisa ngokuzibophezela.

## Imithombo

- [GitLab, "Postmortem of database outage of January 31" (Feb 10, 2017)](https://about.gitlab.com/blog/postmortem-of-database-outage-of-january-31/) — about.gitlab.com
- [GitLab, "GitLab.com database incident" (Feb 1, 2017)](https://about.gitlab.com/blog/gitlab-dot-com-database-incident/) — about.gitlab.com
- [@gitlabstatus, "We accidentally deleted production data…"](https://twitter.com/gitlabstatus/status/826591961444384768) — twitter.com
- [@gitlabstatus, emergency maintenance notice](https://twitter.com/gitlabstatus/status/826572933304827904) — twitter.com
- [Hacker News, "GitLab Database Incident – Live Report" (1,162 points, 598 comments)](https://news.ycombinator.com/item?id=13537052) — news.ycombinator.com
- [Hacker News, the postmortem thread (377 points)](https://news.ycombinator.com/item?id=13619714) — news.ycombinator.com