Hi,
I'm creating a setup where the end user can choose to use LocalScheduler, or DeadlineScheduler via toggle option. Obviously I could approach this choice a number of ways but I just wanted to ask if there is a "best" way.
Complicating things slightly is that a few of the TOPs have per-node Scheduler overrides to allow me to use Deadline's grouping (see separate thread) options.
One approach is to use expressions with some if: statements that look at the toggle, and set the Default scheduler appropriately, but also "clear" the per-TOP Scheduler overrides if "LocalScheduler" is selected.
Alternatively, I can put in Switch TOPs and duplicate the nodes that require the overrides of the Deadline Scheduler and switch them out as needed.
In both cases, I don't see an alternative to an expression in the Default Scheduler of the TOP network itself.
Have I missed anything? I can get my work done with any of these, and I lean towards the Switch TOP simply because the network is clearer to parse (i.e. no expressions tucked away unexpectedly) however thought I'd ask.
18.5.462 Win 10 Py 3
Cheers,
Peter B
Found 1327 posts.
Search results Show results as topic list.
PDG/TOPs » Minor Q: Choosing default scheduler options
- pbowmar
- 7024 posts
- Offline
PDG/TOPs » Q: More granular control of work item assignment in Deadline
- pbowmar
- 7024 posts
- Offline
This seems to work well, thanks Chris!
For anyone reading this, you may need to look into modifying the Generate When parm to "When All Inputs Cooked" if you want to use concurrent tasks.
Multiple workers seems to work fine regardless.
Cheers,
Peter B
For anyone reading this, you may need to look into modifying the Generate When parm to "When All Inputs Cooked" if you want to use concurrent tasks.
Multiple workers seems to work fine regardless.
Cheers,
Peter B
PDG/TOPs » Deadline concurrent tasks not working if single batch cooks
- pbowmar
- 7024 posts
- Offline
I'm experiencing something similar, I think, though I could be wrong!
Further complicating it, I have two Deadline schedulers so that I can use different Groups.
I definitely get concurrent jobs running on a single worker for most of the TOP nodes, but some steadfastly refuse to run concurrent jobs even though there are many frames that could run at the same time.
I have a Wait for All before the "problem" TOP node, hoping that would make it work, but it doesn't seem to help...
SOLVED (maybe): Looks like you need to set Generate When to "When All Upstream Items Cooked". Not sure if that reduces efficiency though. In my case I'm using Generation IFD TOP followed by Render IFD. I have to set Render IFD to Generate When to "When All Upstream Items Cooked" and then I get concurrent tasks working well.
I also have the Generate IFD set to "All Frames in 1 batch" because IFDs should always gen very fast Given that, efficiency should be OK-ish but for a longer IFD gen, not sure where the sweet spot lies.
Cheers,
Peter B
Further complicating it, I have two Deadline schedulers so that I can use different Groups.
I definitely get concurrent jobs running on a single worker for most of the TOP nodes, but some steadfastly refuse to run concurrent jobs even though there are many frames that could run at the same time.
I have a Wait for All before the "problem" TOP node, hoping that would make it work, but it doesn't seem to help...
SOLVED (maybe): Looks like you need to set Generate When to "When All Upstream Items Cooked". Not sure if that reduces efficiency though. In my case I'm using Generation IFD TOP followed by Render IFD. I have to set Render IFD to Generate When to "When All Upstream Items Cooked" and then I get concurrent tasks working well.
I also have the Generate IFD set to "All Frames in 1 batch" because IFDs should always gen very fast Given that, efficiency should be OK-ish but for a longer IFD gen, not sure where the sweet spot lies.
Cheers,
Peter B
PDG/TOPs » PDG node/workitem more detailed run times information?
- pbowmar
- 7024 posts
- Offline
Imre, I assume you mean using LocalScheduler? Since I'm using Deadline, all that info is there in the Deadline Monitor, so this never occurred to me! However, glad this is in there now!
PDG/TOPs » RFE: Additional info on Deadline tasks
- pbowmar
- 7024 posts
- Offline
Hi,
I'm using the Deadline scheduler successfully, so this is more of an enhancement request.
While setting the Job name on Deadline is easy, the granularity isn't ideal, it would be helpful to see which TOP node/work item was associated with each task. Especially since I need to use multiple Deadline scheduler TOP nodes to be able to use "groups" in the Deadline sense.
Is it possible to add the TOP node to Extra Info 1 and the Work Item to Extra Info 2 for the Deadline Tasks? I haven't attempted to dig into the PDGDeadline plugin at all, but my guess (wild guess) is that's where I'd look to add that?
Cheers,
Peter B
I'm using the Deadline scheduler successfully, so this is more of an enhancement request.
While setting the Job name on Deadline is easy, the granularity isn't ideal, it would be helpful to see which TOP node/work item was associated with each task. Especially since I need to use multiple Deadline scheduler TOP nodes to be able to use "groups" in the Deadline sense.
Is it possible to add the TOP node to Extra Info 1 and the Work Item to Extra Info 2 for the Deadline Tasks? I haven't attempted to dig into the PDGDeadline plugin at all, but my guess (wild guess) is that's where I'd look to add that?
Cheers,
Peter B
PDG/TOPs » Q/Confusing default - Render IFD
- pbowmar
- 7024 posts
- Offline
PDG/TOPs » Q/Confusing default - Render IFD
- pbowmar
- 7024 posts
- Offline
Ah got it, thanks for clarifying! I may do a channel reference from the Gen IFD path to the Render IFD path then just to avoid any miscommunication inside our tooling.
Cheers,
Peter B
Cheers,
Peter B
PDG/TOPs » Q/Confusing default - Render IFD
- pbowmar
- 7024 posts
- Offline
Thanks Chris, though I'm a little confused by your comment that "you will not get an expected output" since I expect the output to be whatever I set on the Generate IFD TOP. Or am I misunderstanding?
If you mean the default value of $HIPFILE/etc/etc that's the same on Generate IFD TOP and the Render IFD TOp anyway, right?
Cheers,
Peter B
If you mean the default value of $HIPFILE/etc/etc that's the same on Generate IFD TOP and the Render IFD TOp anyway, right?
Cheers,
Peter B
PDG/TOPs » Q/Confusing default - Render IFD
- pbowmar
- 7024 posts
- Offline
Hi,
Using the generateIFD/Render IFD combo, I notice that the default Output Path on the Render IFD TOP has a default value.
Which, if I understand correctly, means that will _always_ override the output path specified in the Generate IFD TOP.
If I'm correct, I would strongly recommend removing the default value in the Render IFD TOP's Output Path field, currently it's very confusing. IMO of course.
Cheers,
peter B
Using the generateIFD/Render IFD combo, I notice that the default Output Path on the Render IFD TOP has a default value.
Which, if I understand correctly, means that will _always_ override the output path specified in the Generate IFD TOP.
If I'm correct, I would strongly recommend removing the default value in the Render IFD TOP's Output Path field, currently it's very confusing. IMO of course.
Cheers,
peter B
PDG/TOPs » Q: More granular control of work item assignment in Deadline
- pbowmar
- 7024 posts
- Offline
PDG/TOPs » Q: More granular control of work item assignment in Deadline
- pbowmar
- 7024 posts
- Offline
Hi,
Am I missing more granular control of task/work item assignment in Deadline?
Simple example, I want to assign certain work items (tasks in DL) to a specific Deadline group, let's say because it has a lot of RAM, and other work items to a group that is high-CPU.
I see Job File key-values and Plugin File Key-Values, though I don't yet 100% understand how those are consumed. Are there attributes I can add to work items that will get ultimately interpreted by Deadline?
I've read through the PDGDeadline.py plugin with maybe 60% understanding (at the moment) fwiw
Still learning Deadline too so happy to be nudged at looking at something in Deadline docs.
18.5.418 Win 10 Py 3
Cheers,
Peter B
Am I missing more granular control of task/work item assignment in Deadline?
Simple example, I want to assign certain work items (tasks in DL) to a specific Deadline group, let's say because it has a lot of RAM, and other work items to a group that is high-CPU.
I see Job File key-values and Plugin File Key-Values, though I don't yet 100% understand how those are consumed. Are there attributes I can add to work items that will get ultimately interpreted by Deadline?
I've read through the PDGDeadline.py plugin with maybe 60% understanding (at the moment) fwiw
Still learning Deadline too so happy to be nudged at looking at something in Deadline docs.
18.5.418 Win 10 Py 3
Cheers,
Peter B
PDG/TOPs » Q/RFE - BakeTexture TOP node
- pbowmar
- 7024 posts
- Offline
Cool thanks Chris, I'm not fussed about the caching, this is all a pretty linear render-dependency setup. Though would be good to have working of course for other projects!
PDG/TOPs » Q/RFE - BakeTexture TOP node
- pbowmar
- 7024 posts
- Offline
Hi,
I notice that the BakeTexture ROP doesn't have a PDG equivalent. I'm currently using a ROP Fetch TOP to work around this.
Q: Are there any known issues or potential snags using ROP Fetch TOP with BakeTexture ROP?
RFE: BakeTexture ROP TOP please
18.5.418 Python 3 Win 10
Cheers,
Peter B
I notice that the BakeTexture ROP doesn't have a PDG equivalent. I'm currently using a ROP Fetch TOP to work around this.
Q: Are there any known issues or potential snags using ROP Fetch TOP with BakeTexture ROP?
RFE: BakeTexture ROP TOP please
18.5.418 Python 3 Win 10
Cheers,
Peter B
PDG/TOPs » Scheduler "Total Slots" for Single CPU - Does it Matter?
- pbowmar
- 7024 posts
- Offline
I can't speak for SideFX but my experience on a 32 core machine (64 virtual) is to use a custom value of 60. However, be warned about RAM since each job will use at minimum 1gb or so, and likely higher depending on what you are processing.
I have 128gb so I have a fair bit of RAM room, and also my jobs tend not to be heavy disk IO. All these things factor in to the “max efficiency” balance.
So, the 32 cores will be “full” all the time, and the virtual cores are adequate for disk IO, loading in the HIP file etc per frame (or job).
I recommend the new PDG Services as well, which makes jobs execute _much_ faster as well. If you're using Services, then the number of the “Pool size” is essentially replacing the Slot Count. So I set my Pool Size to 60, unless RAM becomes a concern.
hth
Cheers,
Peter B
I have 128gb so I have a fair bit of RAM room, and also my jobs tend not to be heavy disk IO. All these things factor in to the “max efficiency” balance.
So, the 32 cores will be “full” all the time, and the virtual cores are adequate for disk IO, loading in the HIP file etc per frame (or job).
I recommend the new PDG Services as well, which makes jobs execute _much_ faster as well. If you're using Services, then the number of the “Pool size” is essentially replacing the Slot Count. So I set my Pool Size to 60, unless RAM becomes a concern.
hth
Cheers,
Peter B
Houdini Lounge » Python 3 Builds
- pbowmar
- 7024 posts
- Offline
Tractor is thankfully long behind me We're using Deadline, and specifically the SESI provided PDG Deadline TOP, which of course SESI support quickly if there is a problem.
Technical Discussion » Presets in OTLs
- pbowmar
- 7024 posts
- Offline
btw I find it amusing that this 16 year old thread was useful to me today, since I couldn't remember how to embed presets in HDAs. Shame that doesn't have a UI yet as it continues to be very needed.
Technical Discussion » Presets in OTLs
- pbowmar
- 7024 posts
- Offline
You can use the “hidx” command-line tool to expand them, which makes them a little more editable. However, if they have binary in them because you edited the UI or something, then you're still SOL unfortunately.
Houdini Lounge » Python 3 Builds
- pbowmar
- 7024 posts
- Offline
I would recommend you give feedback to the 3rd party vendors that have not yet provided Py3 versions of their software.
I do know Redshift v 3.0 has a Py3 version so they're on the ball at least We have switched to Py3 internally, it took a couple of weeks plus some testing but seems to be generally working and we're about to deploy it to the artists so that's the real test
Cheers,
Peter B
I do know Redshift v 3.0 has a Py3 version so they're on the ball at least We have switched to Py3 internally, it took a couple of weeks plus some testing but seems to be generally working and we're about to deploy it to the artists so that's the real test
Cheers,
Peter B
Houdini Engine for Unreal » Engine V2 plugin for UE4.25.4 - nothing in Actors toolbar
- pbowmar
- 7024 posts
- Offline
PDG/TOPs » Q: HOM: Equivalent of RMB "Cook selected Node"?
- pbowmar
- 7024 posts
- Offline
Hi,
I'm attempting to use HOM to replicate the behaviour of the RMB Cook Selected Node menu item on a TOP.
How does node.executeGraph() know to cook only the current node? The help says:
“Generates and then cooks the PDG graph owned by the parent network of this node”
however it does not actually cook the entire graph of the parent network! It only cooks the current node.
Which is cool, and does what I want.
So just so I understand, executeGraph() actually does not execute the whole graph, just the node specified (and obviously its inputs).
If I want the equivalent of “Cook Output Node” I would find the node using topnode.outputNode() then use executeGraph() on that?
If so that is fine, and I was confused by the docs
18.5.351 Py 3 Win 10
Cheers,
Peter B
I'm attempting to use HOM to replicate the behaviour of the RMB Cook Selected Node menu item on a TOP.
How does node.executeGraph() know to cook only the current node? The help says:
“Generates and then cooks the PDG graph owned by the parent network of this node”
however it does not actually cook the entire graph of the parent network! It only cooks the current node.
Which is cool, and does what I want.
So just so I understand, executeGraph() actually does not execute the whole graph, just the node specified (and obviously its inputs).
If I want the equivalent of “Cook Output Node” I would find the node using topnode.outputNode() then use executeGraph() on that?
If so that is fine, and I was confused by the docs
18.5.351 Py 3 Win 10
Cheers,
Peter B
-
- Quick Links