Houdini 18.0 Executing Tasks

Processor Node Callbacks

Processor nodes generate work items that can be executed by a scheduler

On this page

Overview

Processors are one of the main types of node in a PDG graph. Each processor node is capable of producing new work items using upstream input work items, parameters on the node itself and/or external resources. The node is able to set attribute values on the work items, configure the command line for the work item and define if the work item should be evaluated in a child process or in the current Houdini session. The Merge and Attribute Create are simple processors that create work items with attribute values, while the ROP Fetch is a more complicated processor that creates work items that need to be scheduled for execution.

All processor nodes can be either Static or Dynamic, which determines when the processor creates work items. A static processor generates its work items during the static pre-pass that happens before the PDG graph cook. A dynamic processor creates one or more work items each time a work item in an input node cooks. This behavior is configured using the Work Item Generation parameter that appears on all processor nodes. An instance of a processor node will only generate one type of work item, based on that parameter. It isn’t possible for a processor to have a mix of static and dynamic work items at the same time.

Node Callbacks

Each processor node has several callbacks that can be implemented to control how it operates. When writing a processor node the only callback you're required to implement is onGenerate, since that hook is responsible for actually producing work items. If the work items are marked as in process, the onCookTask callback must be implemented in order to provide the logic used to cook the work items. Work items are marked as in process if the inProcess keyword argument is set to True when contstructing the item. All other node callbacks are optional and are only used to further customize the behavior of the node.

Callbacks can return a pdg.result to indicate whether they succeeded or failed. The other special result types, such as pdg.result.All, are not used for processor nodes. If one or more processor nodes return a failure during the static generation pass, the cook is stopped. For dynamic processors a failure will not stop the cook, and work items produced by other onGenerate calls from different upstream items will continue to execute.

onGenerate(item_holder, upstream_items, generation_type)pdg.result

This callback is evaluated any time the processor node should generate new work items. The pdg.WorkItemHolder argument is a factory object used to create new work items in the node. Work items are not added to the processor node or PDG graph until the callback successfully returns. If the callback fails, anything created during the portion of the callback that did evaluate is deleted and discarded.

Work items are added to the holder using the addWorkItem method, which can be passed either a list of keyword arguments or a pdg.WorkItemOptions helper object. The arguments are used to configure the name, index, and type of the new work item. Each of the entries listed in the pdg.WorkItemOptions API doc can also be passed as a keyword argument, using the same name:

# Using options:
options = pdg.WorkItemOptions()
options.name = "example"
options.inProcess = True
item_holder.addWorkItem(options)

# Or using kwargs:
item_holder.addWorkItem(name="example", inProcess=True)

The addWorkItem method returns the new work item, so that the callback code can set attribute values or the work item’s command line:

new_item = item_holder.addWorkItem()
new_item.setStringAttrib('example_attrib', 'example value')
new_item.setCommand('echo "hello"')

The upstream items list passed to the callback contains no work items if the node has no inputs. If the node has input nodes, the list contains one work item if the node is dynamic or the full list of upstream work items if the node is static. Finally, the generation_type argument determines whether the callback is being called on a static node, dynamic node or for work item regeneration. pdg.generationType lists the details of the enum passed to the callback function.

onRegenerateStatic(item_holder, existing_items, upstream_items)pdg.result

This callback is called during the static cook process when the node has existing work items, but a parameter has changed since the last cook. It gives the node the opportunity to dirty, delete, modify or add new work items. By default, all nodes have a standard built-in implementation that will recreate and merge the work items by calling the onGenerate hook. You generally only need to implement this callback if your node is using external resources to create work items. For example, the File Pattern node creates a work item for each file matching a particular pattern, and therefore defines its own custom regeneration logic that checks timestamps and existence of files on disk.

The arguments to this callback function are similar to onGenerate, however the extra existing_items list contains the list of static work items already in the node.

Note

It is not currently possible to regeneration work items in a processor node that is set to be dynamic.

onAddInternalDependencies(dependency_holder, work_items, is_static)pdg.result

This method is called to add sibling/internal dependencies between work items created during an onGenerate callback. These are dependencies that exist between work items in the same node. The input to this callback is the list of work items produced in the last onGenerate callback. Note that internal dependencies can only be added between work items that were created during the same onGenerate call. The is_static boolean indicates whether the incoming work items were created statically or dynamically.

PDG only cooks work items once all of their dependencies are cooked, so adding internal dependencies is useful if your node generates work items that need to be completed in a specific order. For example, the begin node in a feedback loop block adds internal dependencies between the iterations in the loop.

onPreCook()pdg.result

This callback is called once on all nodes that will be evaluated during a cook, at the beginning of the cook. It is called before any onGenerate callbacks and gives the node the opportunity to load shared resources or initialize variables.

onPostCook()pdg.result

This callback is called once on all nodes that were active during the cook, after the last work item has completed cooking. This gives nodes an opportunity to unload shared resources used during the cook.

onPrepareTask(work_item)pdg.result

This callback is called once for each work item in the node immediately before it is scheduled for cooking. This method is called before PDG checks for cached files on disk, so it can be used as a way to add expected output files for caching purposes. It can also be used to access output files or attributes of the work item’s dependencies, modify the command line of the work item, or do other last-minute work needed before cooking the item. If this method returns pdg.result.Failure the work item is marked as failed.

The ROP Fetch node uses this callback when running a distributed simulation to ensure that the sim tracker is running and pass tracker information to sim work items before they cook. The work items in the ROP Fetch can be generated statically and the tracker port/IP aren’t known until after the cook begins, so the tracker details are added onto simulation items using this callback.

onCookTask(work_item)pdg.result

This callback is called when an in-process work item needs to cook. It is only run for work items that are explicitly created as in-process items, by passing the inProcess=True argument when adding them to the work item holder during the onGenerate callback. This callback is able to write attributes to the work item and attach output results, as well as running whatever in-process logic is necessary to cook the work item. The callback runs inside the main Houdini process on a background thread.

When set to evaluate in process, the Python Script node uses this callback to execute its script.

onSelectTask(work_item)pdg.result

This is a UI callback specific for use with TOPs. When a work item is selected in the TOPs UI, this hook is executed after the item has been selected. For example, is is used by the Wedge node to push work item parameter values to the scene, but could also be used to run custom visualization logic. Note there is also an event hook for work item selection as well. See the pdg.EventType event type list and the event handling overview for details.

onDeselectTask(work_item)pdg.result

This callback is the opposite of the onSelect callback. It runs whenever a work item is deselected in the TOPs user interface. For example, the Wedge node uses this to restore parameter values back to their original value on deselection.

Executing Tasks

Basics

Next steps

  • Running external programs

    How to wrap external functionality in a TOP node.

  • File tags

    Work items track the "results" created by their work. Each result is tagged with a type.

  • Feedback loops

    You can use for-each blocks to process looping, sequential chains of operations on work items.

  • Command servers

    Command blocks let you start up remote processes (such as Houdini or Maya instances), send the server commands, and shut down the server.

  • TOP Service Manager

    The TOP Service Manager manages pools of persistent Houdini sessions that can be used to reduce work item cooking time

  • Integrating PDG and render farm software

    How to use different schedulers to schedule and execute work.

  • Visualizing work item performance

    How to visualize the relative cook times (or file output sizes) of work items in the network.

  • Event Handling

    You can register a Python function to handle events from a PDG node or graph

  • Tips and tricks

    Useful general information and best practices for working with TOPs.

Reference

  • All TOPs nodes

    TOP nodes define a workflow where data is fed into the network, turned into "work items" and manipulated by different nodes. Many nodes represent external processes that can be run on the local machine or a server farm.

  • Processor Node Callbacks

    Processor nodes generate work items that can be executed by a scheduler

  • Partitioner Node Callbacks

    Partitioner nodes group multiple upstream work items into single partition

  • Scheduler Node Callbacks

    Scheduler nodes execute work items

  • Python API

    The classes and functions in the Python pdg package for working with dependency graphs.