Unjiniyela usule i-database yokukhiqiza ye-GitLab. 300 gigabytes.
NgoJanuwari 31, 2017, 23:27 UTC: unjiniyela we-GitLab, elwa nekhophi eliphukile ekupheleni kobusuku obude, ususa uhla lwemibhalo yedatha ye-PostgreSQL ku-db1 esikhundleni se-db2.
NgoJanuwari 31, 2017, 23:27 UTC: unjiniyela we-GitLab, elwa nekhophi eliphukile ekupheleni kobusuku obude, ususa uhla lwemibhalo yedatha ye-PostgreSQL ku-db1 esikhundleni se-db2. I-db1 ingeyokuqala. Cishe ama-300 GB we-database ye-GitLab.com ashabalale ngesekhondi noma amabili, kanti kunezindlela eziyisihlanu zokwenza ikhophi yasenqolobaneni nezokuphendula, ayikho esebenzayo. Ukuhlolwa kokwenzekile: umugqa wesikhathi kusukela ekukhuphukeni kwe-spam kuya egameni elingelona, kungani i-pg_basebackup yabukeka ibambekile, kungani i-pg_dump ibihluleka buthule (9.2 binaries ku-9.6 database, ama-imeyili okuhluleka abuyiselwe yi-DMARC), ukubuyisela kwamahora ayi-18 kusuka kusifinyezo se-staging eseneminyaka engu-6 esakazwa bukhoma ku-YouTube, nokuthi ubani ngempela okumele asolwe. Isinqumo ngempendulo: SHIP IT.
Funda uhlelo olubhaliwe (isiNgisi) ↗
Lokho okuhlanganiswa yile vidiyo
- Jan 31, 2017: rm -Rvf kuhla lwemibhalo yedatha yokuqala; ~300 GB kususiwe, 4.5 GB kusale
- 5 kwabangu-5 abenzi bekhophi yasenqolobaneni bayahluleka: ibhakede le-S3 elingenalutho (ukungafani kwenguqulo ye-pg_dump), akukho zifinyezo ze-Azure ku-DB, ikhophi esusiwe, ikhophi ye-LVM yansuku zonke ngaphandle kwama-webhook
- Feb 1, 18:00 UTC: I-GitLab.com ibuyile kusuka kusifinyezo esenziwe ngesandla esineminyaka engu-6; idokhumende elibukhoma, ukusakazwa bukhoma, ukuhlolwa kokwenzekile okungenasici ngohlu lokulungisa
Imibuzo nezimpendulo
Kwenzekeni ekususweni kwedathabheyisi yokukhiqiza ye-GitLab.com ka-2017?
NgoJanuwari 31, 2017, kusihlwa, injini ye-GitLab eyayizama ukulungisa ikhophi eyayiphukile, yakhipha uhla lwemibhalo ye-PostgreSQL ku-db1 esikhundleni se-db2. I-db1 kwakuyiyona eyinhloko. Ngaphandle kwe-300 GB, kwasala i-4.5 GB. I-GitLab yabeka ku-Twitter ukuthi isuse idatha yokukhiqiza ngephutha, yavula amanothi ayo esigameko esidlangalaleni futhi yasakaza ukululama ku-YouTube.
Kungani kungekho nanye yezindlela zokusekela nokukopisha ezinhlanu ze-GitLab ezisebenzayo lapho zizidinga?
Ngokwe-postmortem ye-GitLab, i-pg_dump yansuku zonke ku-S3 ibihluleka ngoba umsebenzi we-cron ukhethe amabhayinawe we-PostgreSQL 9.2 wedathabheyisi 9.6, futhi ama-imeyili ayo okuhluleka ayebuyiselwa ngenxa ye-DMARC elahlekile. Ama-snapshots e-Azure ayefaka amaseva wamafayela kodwa hhayi amadathabheyisi. Ikhophi yayisuswe ngenhloso, futhi i-snapshot yansuku zonke yayisuswe ama-webhook ayo. I-snapshot eyenziwe ngesandla kuphela, eyenziwe ngesandla, eyenziwe ngenxa yokuhlolwa okungenasihlobo, yasinda.
Kungani ukubuyisela i-GitLab.com kuthathe cishe amahora ayishumi nesishiyagalombili?
Ukubuyisela kwakusho ukukopisha idiski yesiteji emuva ekukhiqizeni phezu kwesitoreji esishibhile se-Azure ngemegabhithi angamashumi ayisithupha ngesekhondi, okwathatha amahora ayishumi nesishiyagalombili. I-GitLab.com yabuyela ngoFebhuwari 1 ngo-18:00 UTC, isebenzisa i-snapshot eyenziwe ngesandla eneminyaka eyisithupha. Ukuphuma kwamandla kubiza amahora ayisithupha edatha, cishe izinkulungwane ezinhlanu zamaphrojekthi, izinkulungwane ezinhlanu zokuphawula kanye nabasebenzisi abasha abangamakhulu ayisikhombisa, ngokusho kwesigameko.
Kungani ama-postmortem angenakho ukusola egxila kunqubo esikhundleni sonjiniyela owenze umyalo?
I-postmortem ye-GitLab, esayinwe yi-CEO, yagcina unjiniyela engaziwa futhi yabonisa ukuhluleka kumagama amabili e-hostname aqhelelana ngendlela eyodwa kanye nezinhlelo zokusekela ezinhlanu okungakaze kubuyiswe kuzo. Yenza umbala obomvu kunkombandlela wokukhiqiza futhi yanikeza ukuqina kwedatha umnikazi, ngoba kwakungenawo ngaphambili. Uhlu lokulungisa lwakhishwa ngezinombolo zezinkinga.
Yini okufanele amaqembu ayithathe esigamekweni sedathabheyisi ye-GitLab, futhi ingabe impendulo ye-GitLab yayilungile?
Isinqumo se-The Daily Diff kwakuyi-SHIP IT, ngempendulo. I-GitLab yenza isigameko esidlangalaleni, yasola inqubo futhi yashicilela uhlu lokulungisa ngezinombolo zezinkinga. Isenzo sangoLwesihlanu salesi sigameko ukubuyisela ikhophi eyisipele, ngoba ikhophi eyisipele engakaze ibuyiselwe ayikwazi ukubalwa njengekhophi eyisipele. I-Hacker News yanikeza idokhumende lesigameko esibukhoma amaphuzu angu-1,162.
Umbhalo obhaliwe ohunyushwe
Kuhunyushwe kusuka ekulandiseni kokuqala kwesiNgisi. Umsindo namazwibela atholakalayo alawulwa yi-YouTube.
0:00 Unjiniyela kwa-GitLab usebenzisa i-rm -rf kuseva yedatha engafanele, futhi amakhulu amathathu ama-gigabytes we-GitLab dot com ayanyamalala ngesekhondi noma amabili, cishe isikhathi esithatha ukufunda igama leseva. Januwari 31st, 2017, 11:27 p.m. UTC. I-GitLab ihlenga ukuthi isule ngephutha idatha yokukhiqiza, ivula amanothi ayo esigameko ku-inthanethi, futhi isakaza ukubuyisela ku-YouTube, ukusakazwa okubili okubukhoma endaweni. Ngosuku olulandelayo, ngokubhala: kuzindlela eziyisihlanu zokwenza ikhophi yasenqolobaneni,
0:25 ayikho esebenza ngokuthembeka. Kwenzeka kanjani, kungani kungenzeka, nokuthi ubani ngempela okumele asolwe. Lena i-The Daily Diff, ukuhlolwa kokwenzekile. 5:20 p.m.: unjiniyela wenza isifinyezo sokukhiqiza ukuhlola i-load balancer ku-staging. 7 p.m.: i-spam ihlasela idatha, kanye nomsebenzi osusa umsebenzi we-GitLab owayebikiwe ngomonakalo. 11 p.m.: ikhophi isale ngemuva kakhulu kangangokuthi eyokuqala isivele isusile
0:48 ilog elidingayo; ukulungisa okuwukuphela kwakho ukuvala ikhophi bese ukopisha eyokuqala futhi. I-pg_basebackup iyahluleka ngaphandle kwemiphumela. Empeleni ilindile, buthule, eyokuqala; akekho owaziyo lokho, futhi incwadi yemiyalelo ayisho. Unjiniyela, owayehlose ukuphuma ngo-11, unquma ukuthi uhla lwemibhalo yedatha engenalutho luyinkinga futhi ulususa. Ku-db1. Eyokuqala. Uyaqaphela ngemuva kwesekhondi noma amabili; cishe amakhulu amathathu ama-gigabytes,
1:11 4.5 asale. Amakhophi asenqolobaneni. Okokuqala: pg_dump ku-S3, nsuku zonke. Ibhakede lingenalutho. Umsebenzi we-cron usebenza kuseva yohlelo lokusebenza ngaphandle kwedatha, ngakho iphakethe likhetha I-PostgreSQL 9.2 binaries ye-9.6 database, iyahluleka, futhi ithumela ama-imeyili ukuhluleka, okubuyiselwa ngenxa yokushoda kwe-DMARC. Okwesibili: I-Azure disk snapshots, enikwe amandla kumaseva amafayela, hhayi ama-database.
1:32 Okwesithathu: ikhophi, esuswe ngamabomu ngehora eledlule. Okwesine: isifinyezo sansuku zonke, esineminyaka engu-24, wonke ama-webhook asuswe yi- ukuvumelanisa kwe-staging. Okwesihlanu: isifinyezo esenziwe ngesandla kusukela ngo-5:20, sokuhlolwa okungahlobene. Leso siwinile. Ukubuyisela kusho ukukopisha idiski ye-staging emuva ekukhiqizeni nge-Azure eshibhile isitoreji ngamashumi ayisithupha megabits ngesekhondi: amahora ayishumi nesishiyagalombili. I-GitLab dot com ibuyile ngoFebhuwari 1st ngo-6 p.m. UTC, idatha endala ngamahora ayisithupha.
1:58 git blame: amagama amabili eseva ahlukene ngohlamvu olulodwa, nezinhlelo eziyisihlanu zokwenza ikhophi yasenqolobaneni akekho owake ake abuyisela kusuka kuzo. Hhayi unjiniyela. Ukuhlolwa kokwenzekile, okusayinwe yi-CEO, kumenza angaziwa, kufaka umbala obomvu kumkhombandlela wokukhiqiza, futhi kunikeza ukuqina kwedatha umnikazi, ngoba kuze kube manje bekungekho noyedwa. Ubukhulu bokulimala: amahora ayishumi nesishiyagalombili phansi, amahora ayisithupha edatha isuswe, cishe amaphrojekthi ayizinkulungwane ezinhlanu, amazwana ayizinkulungwane ezinhlanu,
2:18 abasebenzisi abasha abangamakhulu ayisikhombisa, nabantu abayizinkulungwane ezinhlanu ababuka ibha yokuqhubeka. I-Hacker News inika idokhumenti elibukhoma amaphuzu angu-1,162 futhi icaphuna olulodwa umugqa kubo: kwamakhophi asenqolobaneni ayisihlanu, akekho. Isinqumo, ukuhlolwa kokwenzekile: ship it, ngempendulo. Bayenza isigameko esidlangalaleni, basola inqubo, futhi bashicilela uhlu lokulungisa ngezinombolo zezinkinga. Isenzo sangoMsombuluko: buyisela ikhophi yasenqolobaneni. Uma ungakaze uyibuyisele, awunayo.
2:42 Ngithumele isigameko ongakavunyelwa ukukhuluma ngaso, kumazwana, noma ku-the daily diff dot dev. Futhi yilokho umehluko wanamuhla. NginguNiko wase-Axrisi. Hlanganisa ngokuzibophezela.
Imithombo
- GitLab, "Postmortem of database outage of January 31" (Feb 10, 2017)about.gitlab.com
- GitLab, "GitLab.com database incident" (Feb 1, 2017)about.gitlab.com
- @gitlabstatus, "We accidentally deleted production data…"twitter.com
- @gitlabstatus, emergency maintenance noticetwitter.com
- Hacker News, "GitLab Database Incident – Live Report" (1,162 points, 598 comments)news.ycombinator.com
- Hacker News, the postmortem thread (377 points)news.ycombinator.com



