Please update to 17.5.362 or newer.
The TOP Deadline scheduler has gone through a significant update both in terms of scheduling behaviour and UX.
The most significant change is that a Deadline scheduler node will now schedules each work item in a PDG graph as a Deadline Task under a single Deadline Job. This reduces the “job noisy-ness” and improves performance when dealing with a large number of work items.
Another change, and perhaps not directly a user concern, is the use of a new Deadline plugin called PDGDeadline. Previously the default Deadline CommandLine plugin was used for PDG cook, but it had limited use. This new plugin allows for greater control of setting up and managing each task. The plugin is shipped with Houdini, and will be copied over to the job directory to be used for each cook.
The UI has gone through a significant change as well in order to reduce initial setup and have things just work. Though customizing the scheduler is still available but hidden behind override toggles in the new Advanced section.
For the Scheduler UI section:
The Working Directory is simplified to take an absolute path, with variables if needed. If you have a common farm setup, where all slaves have same shared path as your local machine, then you can supply the path directly in Local Shared Path. Otherwise, you can toggle on the Remote Shared Path and specify a variable-based path that can be mapped in Deadline's Mapped Paths which gets resolved on the slave. This is necessary in a mixed farm setup (e.g. windows, mac, linux slaves).
Added Machine Limit, Machine List, and Blacklist job parameters.
A new Deadline section has been added, which for the most part you don't need to worry about. But it is useful in several ways:
- Verbose Logging - Turn this on to get a log print out of all that is happening when cooking. Very useful for debugging problems. If you have trouble and to submit a bug report, please also copy this verbose log out to a file and attach it. I am thinking of automatically doing this if the toggle is on.
- PDGMQ Server As Task - Slightly advanced feature, but if you get broken socket errors, turning this on might help. It runs the new task tracker (PDGMQ server) as its own task, instead of as a background process.
- Force Reload Plugin - This Deadline job parameter is disabled if PDGMQ Server As Task is also disabled. It is meant to clean the slate between tasks which we can't do if running the PDGMQ server as a background process that we have to keep alive.
Advanced Section: Don't change anything here unless you know what you are doing.
- Task Submit Batch Max - Number of items to submit at a time (on each tick update). You can play with this if you have large number of work items and want faster scheduling vs. less responsive UI.
- Task Check Batch max - Number of tasks in flight to check status of at a time. Again useful when dealing with large number of work items.
- Repository - If you have multiple Deadline repositories, and want to specify the non-system default one, then set it here.
- PDG Deadline Plugin - The scheduler uses a custom new Deadline plugin which is written for the PDG cook process. Only change this if you have written your own plugin that also conforms to the expected behaviour. Please contact me via support if you want to do this as there is no documentation to help with this at the moment.
For the new task tracker, you can specify the ports here. Useful if you have network issues or limitations such as firewalls when communicating with the farm.
For the Job Parms section:
HFS will need to point to the Houdini installation that all slaves will be using. Again, for a straightforward setup where your local machine is same as your slaves, then you can simply leave it as $HFS. But if you have slaves using a different Houdini installation path, then you'll need to supply a variable here that Deadline can map via its Mapped Paths setting. For example, you can set HFS=\$HFS (note the \), and then apply a $HFS= mapping in the Mapped Paths. The \ is used to escape the Houdini evaluation of $HFS.
Similarly, for Python, the default uses the system python. The \$PDG_EXE will be automatically evaluated by the PDGDeadline plugin to replace it with .exe on Windows, and strip it out on other platforms. Deadline requires .exe extension for executable on Windows. If you specify another Python path, you'll need to keep \$PDG_EXE if you are using Windows-based slaves.
Note that Hython field has been removed. The PDGDeadline plugin evaluates it on the slave by formulating it from $HFS (e.g. $HFS/bin/hython on linux).
Added new Pre Task Script and Post Task Script which run before and after each task.
Allows work items to inherit the local environment, set HOUDINI_MAXTHREADS, and add work item specific environment values.
Other new features and changes:
- Submit Graph As Job now supported which allows to schedule the hip file to be cooked on the farm and quit Houdini.
- Added Job Name parm to allow specifying the Deadline Job name.
- Improved performance and error handling.
- Fixed displaying the correct task report file via Task Info > Show Log.
- Fixed shell command execution.
- Removed deadline_jobpreload.py script as its not longer required to be run.
The previous version ran into issues with large number of work items, but this updates handles hundreds of thousands of work items.
The help page has been updated with these changes: https://www.sidefx.com/docs/houdini/nodes/top/deadlinescheduler.html [www.sidefx.com]
There is a current limitation when using more than 1 TOP Deadline scheduler in a graph. I'll be looking into improving this next.
When using a mixed OS farm (or if using variables for HFS path), the slaves might error out with not finding the PDGDeadline plugin. This is a Deadline bug which they've said they will fix (the bug is that the custom plugin directory path is not evaluated via Mapped Paths before looking for the plugin). For now, the workaround is to copy the entire PDGDeadline folder ($HFS/houdini/pdg/plugins/PDGDeadline) into the Deadline repository's custom plugin folder ($repo/custom/plugin/PDGDeadline). If you do this, then make sure that you also update this when installing a new version of Houdini in case the plugin has changed. It is annoying, but if it becomes a real issue, I can look into improving this.
As always, feedback is always welcome Or submit bug tickets if you run into issues.