Hi,

For processing meshes (altering topology), I have a python script that is relatively expensive and can easily run 10 mins (iterating over all points and computing values based on closest neighbors).

I'm wondering if I can split this up into threads and run the script in parallel threads, then process the results once all threads are done. (E.g. processing 20% of points in 5 threads.)

Does somebody have an example (of similar nature) I could look into?


Thanks in advance,
m.

Moin,

there's a bunch of Python modules that would allow you to do parallel processing on data - it depends on your personal taste and the data wrangling at hand which one suits your needs best.
I would, instead, look at VEX or looped code with parallel processing in Houdini. It might be the “easiest” approach to get a faster result.
In theory, you could also think about writing a cache out and “trivially” use your OS' multi-threading to run Python code on parts of the cache in parallel. With 10 minutes current running time, this might be sufficient.

Since this kind of performance-seeking is part of my job, I'll just throw in: You could look at converting your code to CUDA or openCL. Obviously, parallel processing generally expects your data to be independent, if your calculations of one point depend on the outcome of another point's math, you need to look at alternative options for speed-up (probably code optimization or “dirty short cut hacks”).

As for examples, I suggest looking at GitHub and search for python parallel processing or something along those terms.

Marc

Haha, I like the (German) ‘moin’!


Sadly I don't have the time to learn VEX right now (I wish!). Sadly also, I don't have a strong enough programming base as scripting is only a small part of my tasks. Doesn't sound like a low-hanging fruit.

Well, for now running the script a few times takes much less me-time than the time to speed the script up.. I guess I'll leave this for now until I must have this thing run faster.


Thanks for the inputs!

Matt