Houdini 20.5 Nodes TOP nodes

Python Scheduler TOP node

A Python-based programmable Scheduler for PDG.

Since 17.5

This node provides an interface to write Python code for a custom PDG Scheduler and use it to schedule work items. The base Scheduler’s callbacks are exposed as Python text fields which should be filled in according to their function definition. Each callback field is then evaluated and executed during the general PDG work item schedule workflow. The Scheduler node API reference has a detailed explanation for each of the callback functions that you can implement.

The Python Scheduler node can be used as is in a TOP network, or it can be saved out to a Python file using the Save To Python button. The Save To Python functionality will write out a complete implementation of a custom Scheduler node with the filled in callback entries. The Python file should be placed in the PDG Python path (e.g. /home/username/houdini17.5/pdg/types/) so that it will be automatically picked up and registered with the Scheduler Factory. This happens on the launch of Houdini, which means changes to the custom Scheduler will require a relaunch of Houdini.

Once registered, the custom Scheduler node can be created by selecting it from the TAB menu under Schedulers. If it does not show up there, either the Python file was not found by Houdini at launch, or the Python file has errors.

Parameters

Save To Python

Save this node out to a Python file, along with its custom callback entries.

Working Directory

Specifies the relative directory where the work generates intermediate files and output. The intermediate files are placed in a subdirectory. For the Local Scheduler or HQueue, typically $HIP is used. For other schedulers, this should be a relative directory to Local Shared Root Path and Remote Shared Root Path; this path is then appended to these root paths.

Load Item Data From

Determines how jobs processed by this scheduler should load work item attributes and data.

Temporary JSON File

The scheduler writes out a .json file for each work item to the PDG temporary file directory. This option is selected by default.

RPC Message

The scheduler’s running work items request attributes and data over RPC. If the scheduler is a farm scheduler, then the job scripts running on the farm will also request item data from the submitter when creating their out-of-process work item objects.

This parameter option removes the need to write data files to disk and is useful when your local and remote machines do not share a file system.

Delete Temp Dir

Determines when PDG should automatically delete the temporary file directory associated with the scheduler.

Never

PDG never automatically deletes the temp file directory.

When Scheduler is Deleted

PDG automatically deletes the temp file directory when the scheduler is deleted or when Houdini is closed.

When Cook Completes

PDG automatically deletes the temp file directory each time a cook completes.

Compress Work Item Data

When on, PDG compresses the work item .json files when writing them to disk.

This parameter is only available when Load Item Data From is set to Temporary JSON File.

Validate Outputs When Recooking

When on, PDG validates the output files of the scheduler’s cooked work items when the graph is recooked to see if the files still exist on disk. Work items that are missing output files are then automatically dirtied and cooked again. If any work items are dirtied by parameter changes, then their cache files are also automatically invalidated. Validate Outputs When Recooking is on by default.

Check Expected Outputs on Disk

When on, PDG looks for any unexpected outputs (for example, like outputs that can result from custom output handling internal logic) that were not explicitly reported when the scheduler’s work items finished cooking. This check occurs immediately after the scheduler marks work items as cooked, and expected outputs that were reported normally are not checked. If PDG finds any files that are different from the expected outputs, then they are automatically added as real output files.

Limit Jobs

When enabled, sets the maximum number of jobs that can be submitted by the scheduler at the same time.

For farm schedulers like Tractor or HQueue, this parameter can be used to limit the total number of jobs submitted to the render farm itself. Setting this parameter can help limit the load on the render farm, especially when the PDG graph has a large number of small tasks.

Block on Failed Work Items

When on, if there are any failed work items on the scheduler, then the cook is blocked from completing and the PDG graph cook is prevented from ending. This allows you to manually retry your failed work items. You can cancel the scheduler’s cook when it is blocked by failed work items by pressing the ESC key, clicking the Cancels the current cook button in the TOP tasks bar, or by using the cancel API method.

Scheduling

Schedule

onSchedule Callback

Called when a work item needs to scheduled. Returns True if the work item has been successfully scheduled.

Schedule Static

onScheduleStatic Callback

Called when the scheduler should process a static dependency graph.

Submit As Job

submitAsJob Callback

Called when the scheduler should cook the entire TOP Network as a standalone job. Returns the status URI for the submitted job.

Initialization and Cleanup

Start

onStart Callback

Called when the scheduler is registered with PDG. Should initialize or setup scheduler state.

Stop

onStop Callback

Called when the scheduler is unregistered with PDG. Should clean up scheduler state.

Start Cook

onStartCook Callback

Called when a cook starts, before any work items are scheduled. Returns False to cancel the cook, True otherwise.

Stop Cook

onStopCook Callback

Called when cook completes or is canceled. Return value is ignored.

Shared Server

End Shared Server

endSharedServer Callback

Closes an existing shared server instance. Returns True on success.

Transfer File

transferFile Callback

Called when a file should be copied from the local machine to a remote location (e.g. a shared network drive). These would be files that are necessary to do the wok, so they are copied to the working directory. Returns True if copied the file.

Logging

Log URI

getLogURI Callback

Returns the URI that points to the log of the given work item (e.g. file:///myfarm/tasklogs/jobid20.log).

Status URI

getStatusURI Callback

Returns the URI that points to the status of the given work item (e.g. http://myfarm/status/jobid20)

See also

TOP nodes