Search - User list
Full Version: Many Work Items overloading PDG
Root » PDG/TOPs » Many Work Items overloading PDG
martin-egger
Hey,

I've developed a wedging setup for our RoyalRender farm that runs primarily on ROPs, but for the contact sheet creation I do rely on PDG. My current issue is that I'm now overloading my system with a large number of work items (24k image files for 24 wedges with 1000 frames each). Simple operations like string splits and so on just lead to Houdini freezing indefinitely, before I even get to the imagemagick part. RAM or CPU don't show a spike.




I was wondering if that is just in and out of itself a ridiculous number to execute, or I'm missing some settings. I thought PDG was supposed to be scalable like this. The setup does work on smaller batches with like 300 frames*16 wedges on other shots perfectly, but even there it does take quite quite some time to compute which I would hope it somehow wouldn't.

I thought about concatenating work items into batches where one work item stores each image path as an array, but the system is quite fragile and I'd probably need to rebuild it from scratch and lose a lot of the simplicity of tops.

Thanks for your help,
Martin

chrisgreb
PDG itself can scale to orders of magnitude larger graphs, however there could be something in Houdini's UI or in the jobs themselves that are causing such lag.

Is this only happening during the cook? If so maybe Local Scheduler is running too many concurrent processes for your system to handle?

Is it possible to attach the hip file here or on a support ticket?
jparker
FWIW, I found limiting the number of the work item grid items visible on nodes can help UI responsiveness.
Ostap
Thanks for bringing this question.
I have tried to discuss something similar here [www.sidefx.com].
AhmedHindy
jparker
FWIW, I found limiting the number of the work item grid items visible on nodes can help UI responsiveness.

so how did you do that?
jparker
In the network editor, go to the "Tasks" menu item, then limit the maximum numbers of rows and tasks-per-row.

-Jon
This is a "lo-fi" version of our main content. To view the full version with more information, formatting and images, please click here.
Powered by DjangoBB