Hey Guys.

is it possible to see how many threads are running on a task? I can see how many tasks are running, but not necessarily how many threads that equates to. it would be great to get a breakdown of the thread utilisation and what nodes using in terms of threads and ram. could be nice see total threads and ram usage per tops node in the network!

The other question I have is, lets say I have a precache, then sim. If the sim will not take all threads so the precache can do its thing thats fine, but once those threads from the precache free up, are they able to dynamically jump on and help the sim? or is the sim only going to use whats available from the minimum required to get started and be locked from there?

With limited testing so far, it's simply up to the ‘Maximum CPUs to Use’ on the local scheduler, which is then allocated to a work item.

The local scheduler is not currently thread aware. It just knows how many processes are running on the local machine, but has no insights into the inner workings of each process.

To add to that, most of the nodes do their work out of process not in the current Houdini session. For example, the ROP Fetch spawns a Hython process that evaluates the target ROP node. Those external processes have their thread count set when they're spawned by the scheduler, and it will not change dynamically as the process runs.

The CPUs per Job and Maximum Processes settings on the Local Scheduler are used to determine how many jobs to run at the same time. A lengthier explanation was written here: https://www.sidefx.com/forum/topic/61771/ [www.sidefx.com]

ok so i guess then it is important to specify the number of threads we want on a sim, otherwise, it might only get a low number if other tasks are running, and be stuck with it to the end.

Any guidance you can give in multi-GPU systems that are targeting GPU render engines?

So, you have the default local scheduler handling the spawning of CPU jobs, based on its setting, but when it comes to GPU jobs under ROP Fetch, you cant spawn simultaneous jobs on a single GPU, you'd need to spread the jobs out across the number of available GPU's.

Would we need to create a custom gpu scheduler where when it spawns the hython process it can set the GPU affinity for the job? So, if I tell the gpu scheduler that there are 4 GPU's available, it could automatically spawn 4 GPU jobs, setting each job to a GPU (-1, -2, -3, -4, etc.)?

…I wanna figure this out before I move things into the Cloud, under GridMarkets for example.

I`m also interested into controling GPUs, or openCL devices assignent per task in my case. For example i have 2 GPU and i`m wedging some openCL sims(pyro,flip etc) and want each task to run on different openCL device…

I think there are a few possibilities around GPU. What you mentioned there is one, which we will certainly discuss more and follow up with you. The other possibility is to make PDG GPU aware, being able to send a kernel to the GPU, and create workitems at the level of workgroups in OpenCL - so effectively making the GPU a local “farm”. Would be good to get your feedback around that possibility as well.

That would be very helpful. When you think about instances like the P3 types, you can get a lot done with a single engine licence, but it definitely needs to be managed well.

https://aws.amazon.com/ec2/instance-types/p3/ [aws.amazon.com]

psanitra
I`m also interested into controling GPUs, or openCL devices assignent per task in my case. For example i have 2 GPU and i`m wedging some openCL sims(pyro,flip etc) and want each task to run on different openCL device…

I haven't tested this, but there are env vars to control openCL GPU assignment for Houdini, for example:
HOUDINI_OCL_DEVICENUMBER

Which you can set per-task.

Or this might be supported by your farm scheduler for example:
https://deadline.thinkboxsoftware.com/feature-blog/2017/6/23/to-affinity-and-beyond [deadline.thinkboxsoftware.com]

Otherwise, maybe you could accomplish this using virtual machines with different physical GPU assignments?

Any progress or ETA on making PDG GPU aware, from the standpoint of managing jobs that target GPU's?

It's on the roadmap but not soon I'm afraid. Still lots of higher priority things to address. If however folks feel this should be really up there more, please up vote.

I havent tried, but I suspect that, for PDG and ROPs, that Deadline can handle the scheduling of frames/number of GPUs for a render job?

Daryl Dunlap
I havent tried, but I suspect that, for PDG and ROPs, that Deadline can handle the scheduling of frames/number of GPUs for a render job?

Could you clarify further on the GPU setting? Depending on what you want, we might already have support implicitly.

So currently, the TOP Deadline scheduler does not do anything specific for GPUs, as its just a generic scheduler. If you want to target slaves with GPUs, then you can utilize the Job Pool and Job Group parms under TOP Deadline node's Job Parms section to decide which slaves to use. In Deadline, you can set GPU affinity for slave machines via Deadline's repository configuration (https://deadline.thinkboxsoftware.com/feature-blog/2017/6/23/to-affinity-and-beyond).

With PDG, if you use a TOP ROP Geometry, you can specify number of frames to render per job.

If on the other hand, you want to specify GPU affinity in the job, that would require the job itself to set it up. This is what the Deadline's Houdini ROP node. Their parm interface allows to set GPU affinity that eventually calls the relevant hscript APIs for the render job (e.g. Redshift_setGPU -s). This part is not supported by PDG as its too specific to the job. But you can use Deadline's Houdini ROP node with a TOP ROPFetch, if you wish.

#1 - its scary that I understood all of that.
#2 - I'll try it locally and see how it performs on GridMarkets.

Thanks.