Comments on: Python GIL vs. nogil: Boost I/O Performance 10x With One Line Change https://www.backblaze.com/blog/python-gil-vs-nogil-boost-i-o-performance-10x-with-one-line-change/ Cloud Storage & Cloud Backup Tue, 11 Jul 2023 17:06:57 +0000 hourly 1 https://wordpress.org/?v=6.4.3 By: Kevin J. Rice https://www.backblaze.com/blog/python-gil-vs-nogil-boost-i-o-performance-10x-with-one-line-change/#comment-330000 Fri, 26 May 2023 11:38:56 +0000 https://www.backblaze.com/blog/?p=108695#comment-330000 In reply to Paweł Polewicz.

This was inside a datacenter, internal apps only, so no hosting cost issues.
I would be interested to test this idea, IF I WERE STILL THERE.
This was something I did about 5 years ago. So, not useful now.
Still, had to use multi-P because multi-T with GIL caused one URL to lock up preventing another URL from downloading so timing was inaccurate.
The main gain then would be memory requirements – I wouldn’t need 192 GB for 3000 processes, I might have 100 processes each multi-T handling 30 URLs each., etc.
Thanks for the insight.

]]> By: Paweł Polewicz https://www.backblaze.com/blog/python-gil-vs-nogil-boost-i-o-performance-10x-with-one-line-change/#comment-329997 Thu, 25 May 2023 13:34:49 +0000 https://www.backblaze.com/blog/?p=108695#comment-329997 In reply to Kevin J. Rice.

It seems like nogil could save thousands of dollars of monthly hosting costs on that url checking app. Swapping multiprocessing for threads should be pretty easy and switching from CPython to nogil was, in my case, a single line.

]]> By: Michał Nowacki https://www.backblaze.com/blog/python-gil-vs-nogil-boost-i-o-performance-10x-with-one-line-change/#comment-329996 Thu, 25 May 2023 03:05:49 +0000 https://www.backblaze.com/blog/?p=108695#comment-329996 I once had to write a collector of a large amount of json (rather big ones) files, grouping them. Threading was no improvement, as the bottleneck was on parsing jsons. Multiprocessing didn’t help, because of serialization/deserialization between processes (the end goal was to have them all in a single, large, object). This would have fixed it!

]]> By: Kevin J. Rice https://www.backblaze.com/blog/python-gil-vs-nogil-boost-i-o-performance-10x-with-one-line-change/#comment-329994 Wed, 24 May 2023 13:43:07 +0000 https://www.backblaze.com/blog/?p=108695#comment-329994 I had a URL checking app to write, to generate metrics. I had to check about 3000 URLs, once per minute. Some of them were lightweight. Rest calls, others had some heft.
If all were using curllib, response speed of one would be influenced by downloading another.
Best answer was multi-processing.
I only needed four cores but 192 gig of memory because each import chewed space and 3000 parallel processes was comedically heavy.
Still, perfect Metrics – interactive check generated the same answer as my automated solution.
Sometimes for timing reasons you can’t use threads.

]]>