Python: Multithreading code

   2084   2   1
User Avatar
Member
196 posts
Joined: Aug. 2011
Offline
Hi,

For processing meshes (altering topology), I have a python script that is relatively expensive and can easily run 10 mins (iterating over all points and computing values based on closest neighbors).

I'm wondering if I can split this up into threads and run the script in parallel threads, then process the results once all threads are done. (E.g. processing 20% of points in 5 threads.)

Does somebody have an example (of similar nature) I could look into?


Thanks in advance,
m.
User Avatar
Member
806 posts
Joined: Oct. 2016
Offline
Moin,

there's a bunch of Python modules that would allow you to do parallel processing on data - it depends on your personal taste and the data wrangling at hand which one suits your needs best.
I would, instead, look at VEX or looped code with parallel processing in Houdini. It might be the “easiest” approach to get a faster result.
In theory, you could also think about writing a cache out and “trivially” use your OS' multi-threading to run Python code on parts of the cache in parallel. With 10 minutes current running time, this might be sufficient.

Since this kind of performance-seeking is part of my job, I'll just throw in: You could look at converting your code to CUDA or openCL. Obviously, parallel processing generally expects your data to be independent, if your calculations of one point depend on the outcome of another point's math, you need to look at alternative options for speed-up (probably code optimization or “dirty short cut hacks”).

As for examples, I suggest looking at GitHub and search for python parallel processing or something along those terms.

Marc
---
Out of here. Being called a dick after having supported Houdini users for years is over my paygrade.
I will work for money, but NOT for "you have to provide people with free products" Indie-artists.
Good bye.
https://www.marc-albrecht.de [www.marc-albrecht.de]
User Avatar
Member
196 posts
Joined: Aug. 2011
Offline
Haha, I like the (German) ‘moin’!


Sadly I don't have the time to learn VEX right now (I wish!). Sadly also, I don't have a strong enough programming base as scripting is only a small part of my tasks. Doesn't sound like a low-hanging fruit.

Well, for now running the script a few times takes much less me-time than the time to speed the script up.. I guess I'll leave this for now until I must have this thing run faster.


Thanks for the inputs!

Matt
  • Quick Links