Many Work Items overloading PDG

   1737   5   3
User Avatar
Member
30 posts
Joined: May 2017
Offline
Hey,

I've developed a wedging setup for our RoyalRender farm that runs primarily on ROPs, but for the contact sheet creation I do rely on PDG. My current issue is that I'm now overloading my system with a large number of work items (24k image files for 24 wedges with 1000 frames each). Simple operations like string splits and so on just lead to Houdini freezing indefinitely, before I even get to the imagemagick part. RAM or CPU don't show a spike.




I was wondering if that is just in and out of itself a ridiculous number to execute, or I'm missing some settings. I thought PDG was supposed to be scalable like this. The setup does work on smaller batches with like 300 frames*16 wedges on other shots perfectly, but even there it does take quite quite some time to compute which I would hope it somehow wouldn't.

I thought about concatenating work items into batches where one work item stores each image path as an array, but the system is quite fragile and I'd probably need to rebuild it from scratch and lose a lot of the simplicity of tops.

Thanks for your help,
Martin

Edited by martin-egger - Dec. 15, 2020 07:57:34

Attachments:
Screenshot_185.png (155.1 KB)

- Martin Egger
Student at Filmakademie BW
& Freelance FX TD
User Avatar
Member
603 posts
Joined: Sept. 2016
Offline
PDG itself can scale to orders of magnitude larger graphs, however there could be something in Houdini's UI or in the jobs themselves that are causing such lag.

Is this only happening during the cook? If so maybe Local Scheduler is running too many concurrent processes for your system to handle?

Is it possible to attach the hip file here or on a support ticket?
User Avatar
Member
149 posts
Joined:
Offline
FWIW, I found limiting the number of the work item grid items visible on nodes can help UI responsiveness.
User Avatar
Member
209 posts
Joined: Nov. 2010
Offline
Thanks for bringing this question.
I have tried to discuss something similar here [www.sidefx.com].
User Avatar
Member
41 posts
Joined: Oct. 2020
Offline
jparker
FWIW, I found limiting the number of the work item grid items visible on nodes can help UI responsiveness.

so how did you do that?
https://www.youtube.com/channel/UC4NQi8wpYUbR9wLolfHrZVA [www.youtube.com]
User Avatar
Member
149 posts
Joined:
Offline
In the network editor, go to the "Tasks" menu item, then limit the maximum numbers of rows and tasks-per-row.

-Jon
  • Quick Links