mqserver is used by PDG jobs when they are running with deadline and hqueue schedulers to RPC back to PDG, it's also used to coordinate 'services', used in local graph cooks. The python module pdgutils can be used to communicate with mqserver at a low level, but jobs use the pdgcmd/pdgjson APIs as documented here:
https://www.sidefx.com/docs/houdini/tops/jobapi.html [www.sidefx.com]
There aren't any online docs for pdgutils yet, but there is the usual pydoc on the module.
> what sort of things you can communicate
The communication is primarily job -> PDG, through pdgcmd. However mqserver is a general purpose pub-sub server that could be used for custom messaging as well.
If you want to use this for a custom scheduler, I would recommend looking at our implementation for HQueue Scheduler in $HH/pdg/types/houdini/hqueue.py. The simplest approach is probably to only support MQUsage.CONNECT, which uses a known running mqserver so there's no need to spin anything up per submission.
Found 424 posts.
Search results Show results as topic list.
PDG/TOPs » MQserver - more information on capabilities & communication
-
- chrisgreb
- 603 posts
- Offline
PDG/TOPs » Deadline scheduler - Redshift Archive and Standalone jobs?
-
- chrisgreb
- 603 posts
- Offline
What is the Deadline ROP doing? Just guessing, since I don't have Redshift installed: I think you would have to set up a ROP Fetch TOP that cooks the Redshift ROP first in 'archive' mode followed by a ROP Fetch TOP that cooks in 'render' mode. These would both be using houdini engine licenses. Possibly the render step could be done with a pure python script or generic generator so it doesn't use an engine license. This is what the Render IFD and USD Render TOP nodes do.
PDG/TOPs » RFE: Additional info on Deadline tasks
-
- chrisgreb
- 603 posts
- Offline
Ya, we could do that. We could probably expose them all as extra job parms.
Are you skipping Extra Info 0 intentionally ?
Are you skipping Extra Info 0 intentionally ?
PDG/TOPs » Minor Q: Choosing default scheduler options
-
- chrisgreb
- 603 posts
- Offline
PDG/TOPs » Minor Q: Choosing default scheduler options
-
- chrisgreb
- 603 posts
- Offline
> Complicating things slightly is that a few of the TOPs have per-node Scheduler overrides
This shouldn't complicate things actually. The overrides for one scheduler should not interfere with overrides for another.
This shouldn't complicate things actually. The overrides for one scheduler should not interfere with overrides for another.
PDG/TOPs » Q/Confusing default - Render IFD
-
- chrisgreb
- 603 posts
- Offline
This will be fixed in the next build of 18.5 - default changed to not override, and the image path will be passed from generate IFD to render IFD via a work item attrib so that caching will work as expected.
PDG/TOPs » Q/Confusing default - Render IFD
-
- chrisgreb
- 603 posts
- Offline
As it stands, when the renderifd work items are generated they won't know the path of the raster file that will be produced by rendering the ifd (because there is no override). This means it can't cook from cache and you can't make use of expected output downstream (statically).
PDG/TOPs » Q/Confusing default - Render IFD
-
- chrisgreb
- 603 posts
- Offline
Yes, it will override the output by default. If you clear the parm it will not override it, but then you will not get an expected output (which may not matter for you workflow anyway).
Your criticism makes sense, I'll add an RFE to clean up the UX there.
Your criticism makes sense, I'll add an RFE to clean up the UX there.
PDG/TOPs » Tractor farm error with 18.5
-
- chrisgreb
- 603 posts
- Offline
18.5 uses $HHP to find scripts it needs to copy, you should verify that $HHP is in your Houdini environment. It's added by the houdini_setup scripts.
PDG/TOPs » Single slot only for a specific ROP fetch ? (GPU problems)
-
- chrisgreb
- 603 posts
- Offline
You can limit that one node to only execute one work item at a time by adding the Local Scheduler 'single' Job Parm to that node and toggling it on.
https://www.sidefx.com/docs/houdini/tops/schedulers.html#jobparms [www.sidefx.com]
However, since this is a Rop Fetch, you can avoid the overhead of restarting houdini for those work items by creating a Rop Fetch Service and assigning it to that node. If the service has a pool size of 1, this is also going to have the happy side effect of ensuring that only one runs at once.
https://www.sidefx.com/docs/houdini/tops/services.html [www.sidefx.com]
https://www.sidefx.com/docs/houdini/tops/schedulers.html#jobparms [www.sidefx.com]
However, since this is a Rop Fetch, you can avoid the overhead of restarting houdini for those work items by creating a Rop Fetch Service and assigning it to that node. If the service has a pool size of 1, this is also going to have the happy side effect of ensuring that only one runs at once.
https://www.sidefx.com/docs/houdini/tops/services.html [www.sidefx.com]
PDG/TOPs » Q: More granular control of work item assignment in Deadline
-
- chrisgreb
- 603 posts
- Offline
We've found Deadline is a bit less flexible than other farm schedulers for things like this, so the UX is a bit convoluted. The pool and other such attributes live only at the Job level, so we need to create separate jobs for each pool that is used in the cook.
The way to accomplish this is to create a Deadline Scheduler TOP for each pool that you need. For example a 'deadlinescheduler_highmem' in addition to 'deadlinescheduler1'. Then you can use the TOP Scheduler override on the TOP node to do the assignment.
The way to accomplish this is to create a Deadline Scheduler TOP for each pool that you need. For example a 'deadlinescheduler_highmem' in addition to 'deadlinescheduler1'. Then you can use the TOP Scheduler override on the TOP node to do the assignment.
Edited by chrisgreb - Jan. 21, 2021 09:18:44
PDG/TOPs » Q/RFE - BakeTexture TOP node
-
- chrisgreb
- 603 posts
- Offline
BakeTexture will work and report results, however the PDG caching won't work because ROP Fetch TOP can not predict what files will be generated. This is a shortcoming we would like to address.
PDG/TOPs » PDG deadline never ending looping over its own renders
-
- chrisgreb
- 603 posts
- Offline
Could you try setting PDG_USE_PDGNET=0 in your Houdini environment? This will cause the deadline scheduler to use a different method for communicating back to Houdini.
I think this may be a bug that was fixed in subsequent builds of 18.0.
I think this may be a bug that was fixed in subsequent builds of 18.0.
PDG/TOPs » PDG deadline never ending looping over its own renders
-
- chrisgreb
- 603 posts
- Offline
> Just to make sure I am correct. Is this right for education?
Actually, I probably saw this when I opened your file because I have apprentice license available instead of educational. So in your case you might not have a problem (you should generate .usd instead of .usdnc) so you don't need to modify the Output File path on the ROP USD.
Actually, I probably saw this when I opened your file because I have apprentice license available instead of educational. So in your case you might not have a problem (you should generate .usd instead of .usdnc) so you don't need to modify the Output File path on the ROP USD.
PDG/TOPs » Generate Work item by new file uploaded
-
- chrisgreb
- 603 posts
- Offline
You could run a monitoring script that will kick off a TOP network cook when the file is updated. There is a similar example in the docs:
https://www.sidefx.com/docs/houdini/tops/cooking.html#cookhom [www.sidefx.com]
https://www.sidefx.com/docs/houdini/tops/cooking.html#cookhom [www.sidefx.com]
PDG/TOPs » PDG deadline never ending looping over its own renders
-
- chrisgreb
- 603 posts
- Offline
If you are using non-commercial license you should set the Output File parm on ropusd1 to `$HIP/USD/$HIPNAME.$OS.usdnc` because that's what will end up being written.
Is the endless rendering occurring with Local Scheduler as well? Do you have the deadline log files? Maybe the jobs are being restarted for some reason.
Is the endless rendering occurring with Local Scheduler as well? Do you have the deadline log files? Maybe the jobs are being restarted for some reason.
Edited by chrisgreb - Jan. 8, 2021 08:39:19
PDG/TOPs » Houdini Top node: Imagemagick error
-
- chrisgreb
- 603 posts
- Offline
Houdini doesn't come with imagemagick - the TOP node needs to find your installed copy of it.
PDG/TOPs » Running PDG Job nodes in a sequence.
-
- chrisgreb
- 603 posts
- Offline
The usual way to do this is to insert a "Wait For All" in between each stage. There isn't much cost to doing a separate File Pattern for each step, so it may be simpler to do it that way:
File Pattern -> Export 1 -> Wait For All -> File Pattern -> Export 2 -> ..
File Pattern -> Export 1 -> Wait For All -> File Pattern -> Export 2 -> ..
PDG/TOPs » PDG to Deadline Connection
-
- chrisgreb
- 603 posts
- Offline
You need at least 2 workers, one to run the houdini PDG cook and the other to execute the work item jobs which are created. They can be on the same machine, because the Houdini cook is generally not very expensive cpu-wise.
PDG/TOPs » PDG to Deadline Connection
-
- chrisgreb
- 603 posts
- Offline
Mark WallmanWhat does it look like in the Deadline monitor? Does the first task have any output? It could be that there aren't enough matching workers to take additional tasks?
but just sits there not rendering anything. Is there something else I need to do to get this to work? Best Mark
-
- Quick Links