I-Polars 2.0: kungani ama-join engasawugcini umugqa wawo

Lokho okuhlanganiswa yile vidiyo

• Ukubiza i-collect() ku-lazy query manje isebenzisa injini yokusakaza ngokuzenzakalelayo, engagcili ukuhleleka komugqa we-join, group_by kanye ne-unpivot.
• Umhlahlandlela wokufuduka uthi ukuhleleka akuseyona indlela yokuhleleka yangaphambi kuka-2.0 futhi akugciniwe; kuncoma ukuhlela okucacile uma uncike kukho.
• Lapho sisebenzisa, i-inner join ezenzakalelayo ibuyise ukuhleleka kokufaka okuyi-False; i-maintain_order="left_right" ku-join ibuyise i-True. Umshini owodwa, ukusebenza okukodwa, isimo se-join esisodwa.
• Ukuchitheka kwangaphandle-kwangaphakathi kuvuliwe ngokuzenzakalelayo: kuqala cishe ku-80% ye-RAM, nesabelomali sediski esingu-64 GB esizenzakalelayo. I-out-of-core join kanye ne-group_by kukhona kumgwaqo.
• I-collect_schema() ibamba ikholomu elahlekile ngaphambi kokuba kufundwe noma iyiphi idatha, kodwa i-cast ephazamiseka kunani elilodwa ihluleka kuphela ku-collect().
• Ukuqhathaniswa kwe-TPC-H ne-TPC-DS kungukusebenza komthengisi ngokwakhe kudatha etholakele, futhi inothi layo lithi imiphumela ayiqhathaniseki nokuqhathaniswa okusemthethweni.

Umbhalo obhaliwe ohunyushwe

Kuhunyushwe kusuka ekulandiseni kokuqala kwesiNgisi. Umsindo namazwibela atholakalayo alawulwa yi-YouTube.

Kungani ukuhleleka okuzenzakalelayo kushintshile?

0:00 Ucabanga ukuthi i-join ibuyisela imigqa yakho ngokulandelana kwayo. I-Polars 2.0 yayeka ukuthembisa lokho, ngokuzenzakalelayo, ngenhloso. Kule vidiyo: kungani ukushintsha okuzenzakalelayo? Kwatholani? Futhi yimuphi wemibhalo yakho ophulwa kuqala? Lena yi-The Daily Diff, ngaphansi kwe-hood. I-Polars iyi-library ye-DataFrame yomthombo ovulekile: injini yethebula oyibiza ku-Python,

0:20 ngengqikithi yayo ebhalwe ku-Rust. Kumahhala ngaphansi kwelayisense le-MIT, futhi uyifaka nge-pip install polars. Imininingwane eyodwa kuqala. I-switch ebuyisa indlela endala iyimpikiswano ye-join uqobo. Ngizobuyela kuyo ekugcineni. Kusukela ku-2.0, i-collect isebenzisa injini yokusakaza ngokuzenzakalelayo. Ukusakaza kuhlanganisa i-query ibe izingxenye ezisebenza ngokufana, futhi izingxenye ziqeda lapho ziqeda khona.

0:42 Okokuhlangana, i-group by kanye ne-unpivot, ukukhishwa akusagcili ukuhleleka. Ubani ohlanganiswa kuqala? Noma ngubani oqhathanisa okukhiphayo nefayela eligciniwe, umugqa ngomugqa. Lokho kuhlolwa kungaphumelela ku-laptop yakho futhi kuhluleke kumshini ohlukile. Okulandelayo, umhlahlandlela wokufuduka.

Yini eshiwo umhlahlandlela wokufuduka?

0:57 Ithi ukuhleleka komugqa okukhonjiswe ngenhla akugciniwe. Ngakho-ke uma uncike ekuhlelekeni, hlela ngokusobala. I-Out-of-core nayo ivuliwe ngokuzenzakalelayo. Iqala ukuchitheka kudiski cishe kuma-80% e-RAM, ngesabelomali esingu-64 gigabyte. Hlela, imisebenzi yewindi kanye nezinkulumo eziningi zingachitheka manje. Ama-joins nama-group-by ayeza.

Ingabe ukuqhathanisa kungukulwa okulungile?

1:17 I-Polars ithi ishaya i-DataFusion ne-DuckDB kuma-TPC-H ne-TPC-DS ukuqhathanisa. Izinombolo zivela ekusebenzeni kwazo, kudatha etholakele, futhi inothi labo lithi azihlangene nemiphumela esemthethweni. Izinombolo zabo zithi i-Polars ishesha cishe ngokuphindwe ka-3.8 kusuka kuma-cores ayishumi nesithupha kuya kwamakhulu ayikhulu namashumi ayisishiyagalolunye nambili, ku-TPC-H. Baphinde bathi izintambo eziningi zomshini omkhulu zinciphisa ukusebenza kwama-query amancane.

Yini esiyibona lapho siyisebenzisa?

1:41 Sisebenzise i-join efanayo kabili ku-Polars 2.0.0. Ukusebenza okuzenzakalelayo kuhlanganisa imigqa. Ngefulegi, ukuhleleka kokufaka kuyasinda. Thola ama-join afaka ukuhlolwa. Noma iyiphi i-join engenaso isistimu emva kwayo isinqumo sokuhleleka komugqa ongakaze usenze. I-Polars 2.0 iqine kakhulu mayelana nezinhlobo.

Ingabe ukuqina kusho ngaphambili?

1:58 Ukuhlolwa kwe-schema kubamba ikholomu elahlekile ngaphambi kokuba kufundwe noma iyiphi idatha. Ayikwazi ukubamba i-cast ehluleka kumugqa owodwa, ngoba lelo phutha lilindele i idatha. Manje i-loop engiyivulile.

Ngithola kanjani ukuhleleka kwakudala?

2:08 I-switch ihlala ku-join. Setha i-maintain order ku-join, noma engeza i-sort emva kwayo. Umhlahlandlela usikisela ukuhlela. Ifulegi le-join ligcine ukuhleleka ekusebenzeni kwethu, ngokuhleleka kohlangothi lwesokunxele kugciniwe njenge kwesokunxele, kwesokudla. Isinqumo, ngaphansi kwe-hood: sidinga ukubuyekezwa.

Siyini isinqumo?

2:22 Ngingabuyekeza yonke i-join ngaphambi kokuthuthukisa. Unombuzo mayelana nalokhu? Wubeke kumazwana. Futhi lokho kungukuhlukana kwanamuhla. Ngingu-Niko wase-Axrisi. Hlanganisa ngokuzibophezela.

Imithombo

1. Release of Polars 2.0Polars (pola.rs)
2. Polars 2.0 upgrade guidePolars documentation
3. Polars homepagePolars (pola.rs)
4. polars-2.0-benchmark repositoryPolars on GitHub
5. Release of Polars 2.0 (Hacker News discussion)Hacker News