Houdini 20.5 Executing tasks with PDG/TOPs

PDG node network interface

How to use the unique network editor features of TOP networks.

On this page

See also how to cook the network.

TOP nodes

Work item dots

The distinguishing feature of TOPs nodes is the grid of dots representing the work items in each node. If a node hasn’t generated work items yet, it won’t show any dots. You can pre-generate the “static” work items (those items known ahead of time) using Tasks ▸ Generate Static Work Items. Dynamic work items appears as the network runs (see cooking the network).

To...Do this

Show information about an individual work item

  • Press on a work item dot to show a pop-up info window containing information about the work item, including its attribute values.

  • You can also ⌃ Ctrl + click a dot to open a persistent work item info window, or right-click the dot and choose Task info.

Hide the dots on one node

  • Click the up-pointing triangle in the bottom-left corner of the node to “collapse” the node, hiding the dots.

  • Click the down-pointing triangle on a collapsed node to show the dots again.

Hide the dots on all nodes

In the menus at the top of the Network Editor, turn off Tasks ▸ Show Task Graph.

This may be necessary if networks have huge numbers of visible work items. See also the controls in the display options for limiting the number of dots shown on each node.

Limit the number of dots shown on nodes

See the display options.

Show a table of all work item data

Click the table icon in the bottom-right corner of a TOP node to open the task graph table.

The color of the dots normally indicates the individual work item’s status (you can switch to show relative speed as color instead, see visualizing performance).

Dot

Color

Status

Description

Bright green

In progress (cooking)

The work item is currently running.

Medium green

Done (cooked)

The work item finished without any warning or error.

Dark blue-grey

Waiting

The work item has started but is waiting (to be scheduled or for upstream work to finish).

Grey

Uncooked/Not cooking

The work item has been generated but hasn’t run yet.

Orange

Warning

The work item generated a warning.

Red

Failed

The work item encountered an error (or the executable returned a non-zero status code).

Brown

Canceled

The user canceled the cook while the task item was running.

Work item pages and batches

When a TOP node’s work item count exceeds 200, work item pages will appear on the node. Each work item page contains 200 or less work items and each node can contain a maximum of 50 work item pages.

Work item pages provide a quick and easy way to access your numerous work items directly on their nodes. However, when a node has work items numbering in the thousands, it may be easier for you to access and manage your large number of work items through the Task Graph Table.

Work item pages

Tip

When navigating work items on a node with the [ and ] keys, you will automatically switch between work item pages.

When batch processing is enabled for a TOP node, you can display its work item dots as batch rings by setting Task Collapse Mode to By Batch in the Task Graph Display Options.

Work item batch rings

Status icon and counts

To the left of each TOP node, Houdini shows an icon indicating the overall status of the work items in the node, and a table of the number of work items with each status. When the node is cooking, a ring around the icon shows the progress of the work items. (If you hide work items with certain statuses using the display options, those items are not included in the progress.)

Icon

Status

Description

Not started

This node is not running.

Waiting

This node has started, but is waiting for its items to be scheduled or for upstream dynamic items to finish.

Cooking

This node is running, the work items inside are being scheduled on the machine/render farm.

Done

All work items in this node have finished running with no warnings/errors.

Warning

At least one work item in this node raised a warning as it ran.

Error

At least one work item in this node raised an error as it ran.

Blocked

The cook for at least one work item in this scheduler node is blocked. This node is waiting for its failed work items to be resolved.

Work item info window

The work item info window contains information about individual jobs including their attribute values.

To open the work item info window for a single job, do the following:

  • -click + hold on a work item dot to open a temporary work item info pop-up window.

  • ⌃ Ctrl + -click a work item dot or -click a work item dot and choose Work Item Info from the context-sensitive menu that appears to open a persistent work item info window.

Work item info window

Button

Name

Description

Keep Open

When on, the current work item info window remains open even when you select different work items, TOP nodes, or switch contexts.

Show delocalized path

When on, all directory and network paths are displayed as non-local or delocalized paths. For example, paths would begin with $HIP instead of C:/. This button is toggled on by default.

Show No Copy attributes

When on, displays all the hidden or no copy attributes for the current work item.

Show attribute filter

When on, displays the attribute filter. You can use the attribute filter to specify which attributes appear in the work item info window. Only the attributes whose names match the text in the Filter field will appear in the window. You can also access the attribute filter with the ⌃ Ctrl + f hotkey. Pattern

Reload attributes

When on, reloads the current work item’s attributes. For example, this is useful when something has changed upstream and you want to see the latest information for the current work item.

Tasks bar

The tasks bar is the PDG toolbar that appears at the top of the Network Editor when viewing a TOP network.

Tasks bar

The tasks bar lets you quickly access task-related menu items and managers, and it also displays important work item cook information like how many work items are scheduled, what their states are, how many nodes have cooked, and the cook progress bar.

UI Element

Name

Description

Tasks menu button

Expands the Tasks menu.

Cancel all cooks button

Cancels all work item cooks in the current TOP network.

Cook output button

Cooks the work items for the node flagged as the Output node in the network.

Cook selected button

Cooks the work items for the node that is currently selected.

Error count

Displays the number of work items whose cooks failed and issued errors.

Warning count

Displays the number of work items that successfully cooked but issued warnings.

Cooked count

Displays the number of work items that successfully cooked with no errors or warnings.

Cooking count

Displays the number of work items that are currently cooking.

Waiting count

Displays the number of scheduled work items that are waiting to cook.

Node cook total

Displays the total number of nodes that have cooked so far.

Task Graph Table button

Opens the Task Graph Table window for the currently selected TOP node.

Status bar

When you select a work item and then navigate away from the /topnet context, the status bar replaces the tasks bar. The status bar allows you to view work item information from outside of TOPs.

UI Element

Name

Description

Active Work Item toggle

When on, you can access the work item drop-down list. When off, you cannot change your current selection in the work item drop-down list. The list is disabled.

Work item drop-down list

  • Lists all the work items that are on the currently selected work item dot’s TOP node.

  • Allows you switch between the work items on the current TOP node.

  • Task statuses automatically update in the work item drop-down list as their work items cook.

Information

Displays the work item info window for the task selected in the work item drop-down list. -click-hold the ? icon to temporarily view a work item’s information, or -click the ? icon to view a work item’s information in a persistent window.

Task Graph Table button

Opens the Task Graph Table window for the currently selected work item dot’s TOP node.

Tasks menu

This menu appears on the TOP tasks bar.

Generate Static Work Items

Makes every node in the network generate any static work items it knows will be needed ahead of time, based on parameter values and/or incoming static items.

Generate Selected Nodes

Generates the work items for the selected TOP nodes.

Cook Output Node

Starts cooking the network to produce the final output.

Cook Selected Nodes

Cooks all the selected TOP nodes.

Dirty All

Marks all nodes in the network as needing to recook. This means the next time the network cooks, all work items will run.

Delete All Results From Disk

Deletes all result files tracked by work items in the graph.

Dirty Selected Nodes

Marks the selected nodes as needing to recook. The next time the network cooks, all work items in the node will run.

Dirty and Cook Selected Nodes

Forces all work items in the network to recook down to the selected node. This can be useful to test different parts of the network.

Dirty and Cook Output Node

Forces all work items in the network to recook down to the final output.

Delete Selected Node Results From Disk

Deletes all result files tracked by work items in the selected node.

Cancel Current Cook

Stops cooking the network and may cancel in-flight processes.

Pause Current Cook

Pauses cooking the network. Any in-flight processes will continue to run.

Terminate Remote Session

Terminate the connected remote session. This is used when viewing a Remote Graph.

Show Task Graph

Whether to show dots representing work items inside the nodes.

Task Graph Table

Opens the Task Graph Table window. See the Task Graph Table section below.

Task Graph Display Options

Opens a window with TOP-specific display options

Show Task Graph Performance

When on, the Network Editor colors dots according to relative speed or output size, rather than by status. See visualizing performance for more information.

Performance Stat

When Show task graph performance is on, this controls whether to color work items by relative cook time or output size.

PDG Services

Opens the PDG Services window.

PDG Path Map

Opens the PDG Path Map.

Task Graph Table

  • To show the Task Graph Table, click the table icon in the bottom-right corner of a TOP node or select Task Graph Table from the Tasks menu in the Network Editor.

    The Task Graph Table shows the metadata of all work items in the graph, or all work items in a particular node.

  • Use the Task Graph Table - Nodes options window and pin toggle to choose which work items appear in the Task Graph Table.

  • Use the Task Graph Table - Columns options window to choose which attribute or property columns appear in the Task Graph Table.

Task Graph Table - Nodes

Opens the Task Graph Table - Nodes options window.

Task Graph Table - Nodes options window

The Task Graph Table - Nodes options window lets you specify which nodes' work items currently appears in the Task Graph Table.

Node display mode

Show selected nodes

When on, the Task Graph Table is context-aware and only displays the work items of the nodes currently selected in the Network Editor.

Show custom list of nodes

When on, the Task Graph Table only displays the work items of the nodes currently selected in the Custom node list. The nodes that appears in the Custom node list change from topnet to topnet.

Custom node list

List nodes containing

Filters the list of TOP nodes that appears in the Task Graph Table - Nodes options window by path. This is useful when you only want to list nodes from a certain path, an individual subnet, or nodes with specific words in their names.

This filter field only accepts text string inputs. It does not accept operators or wildcards as inputs.

Filtering the node list

Select All

Selects all the nodes listed in the Custom node list.

Select None

Deselects all the nodes listed in the Custom node list.

Task Graph Table - Columns

Opens the Task Graph Table - Columns options window.

Task Graph Table - Columns options window

The Task Graph Table - Columns options window lets you specify what work item information currently appears as columns in the Task Graph Table.

Task Graph Table columns
Filter Attribute List

Show attributes containing

Filters the list of column options by attribute or property name. This is useful when you only want to list column options that contain specific text.

This filter field only accepts text string inputs. It does not accept operators or wildcards as inputs.

Filtering the list of column options

List all attributes

When on, all work item attributes and properties are listed as column options in the Filter Attribute List.

Show only work item properties

When on, only work item properties are listed as column options in the Filter Attribute List.

Show only work item attributes

When on, only work item attributes are listed as column options in the Filter Attribute List.

Select All

Selects all the attributes and/or properties listed in the Filter Attribute List.

Select None

Deselects all the attributes and/or properties listed in the Filter Attribute List.

Remote

Opens the Remote PDG Instance Options window.

The Remote PDG Instance Options window lets you connect the Task Graph Table to a remote instance of PDG.

Pin

You can pin and unpin the current contents of the Task Graph Table by clicking the pin icon at the top of the Task Graph Table window.

When the Task Graph Table pin is active, only the work items for the TOP nodes that were selected when you activated the pin are displayed in the Task Graph Table. If you navigate away from those nodes, like by selecting other nodes in the Network Editor or by switching contexts (for example, switching from task-level to object-level), the Task Graph Table will continue to display only the work items that belong to the pinned TOP nodes.

Also, if the pinned TOP nodes change in any way, then all the information for their work items is automatically updated in the Task Graph Table.

Task Graph Display Options

You can open the Task Graph Display Options window through the Tasks menu in the Network Editor or the Network View Display Options window (hotkey: D key).

Task Graph Display Options window

Show Tasks

Turns off the display of dynamic work items with certain states.

Note

These controls do not affect static work items.

Cooked

Show cooked work items in the nodes.

Cooking

Show work items that are cooking.

Dirty

Show work items that have changed since their last cook.

Canceled

Show work items that have been canceled.

Failed

Show work items that have failed to cook.

Sort By

Automatic

If the network is visualizing performance, sort by performance, otherwise sort by index. This is the default.

State

Sort by the work item state, for example “cooking”, “cooked”, “failed”, and so on.

ID

Sort by each work item’s unique ID.

Index

Sort by the index attribute. This tends to keep child items in line with their parents.

None

Work items appear in no particular order.

Reverse Sort Direction

Reverses the sort direction, when sorting by one of the sort criteria specified using the Sort By menu.

Task Collapse Mode

Off

Do not auto-collapse nodes. This is the default and the recommended setting.

By Batch

If batching is enabled on a node, then all the work items on that node appear as batch rings instead of work item dots.

By Attribute

Work items possessing the attribute specified by Work Item Attribute to Collapse By groups by the value of that attribute and the groups appear as batch rings.

Work Item Attribute to Collapse By

Specifies which attribute to collapse work items by when Task Collapse Mode is set to By Attribute.

Isolate Highlight Tasks

When you select a work item, this hides all work items that aren’t ancestors or descendants of the selected item.

Max Dependency Depth

The maximum dependency depth that should be displayed when selecting a work item. The default value of 0 indicates that dependencies should be drawn from the selected work item all the way to the top and bottom of the node graph. Positive values will limit the number of lines displayed. For example, a value of 1 will result in no dependencies appearing in the TOP network, aside from the active selection. A value of 2 will draw one level of dependency lines and work item highlights beyond the current selection in both directions.

Max Number of Rows

The maximum number of rows of dots to show inside nodes. When extra rows are hidden, the visible rows will end with a “fade out” effect to show that more work items exist but are not shown.

Tasks Per Row

How many dots to show on each row inside nodes. The default is 10. Decreasing this number makes the dots bigger, increasing it makes the dots smaller.

Use Work Item Color Attribute

When on, you can use an attribute as your work item dot color instead of the work items' current states. This allows you customize the color of the work item dots on your nodes.

For example, if you have custom states for your work items, then you could write some custom python code to map the color attribute you specify with those states. You could also use an Attribute Create TOP node to create a custom color attribute and then map the color attribute to certain work items by placing the node in a specific location in your TOP network.

Example: A custom color attributecreate node placed between a wedge node and a ropgeometry node will assign each wedge’s downstream work item dots a different color.

To specify which attribute to use, type the name of the attribute in the Work Item Color Attribute field.

Work Item Color Attribute

Specifies which attribute to use for your custom work item dot color.

This field only accepts a single attribute name as input.

Executing tasks with PDG/TOPs

Basics

Beginner Tutorials

Next steps

  • Running external programs

    How to wrap external functionality in a TOP node.

  • File tags

    Work items track the results created by their work. Each result is tagged with a type.

  • PDG Path Map

    The PDG Path Map manages the mapping of paths between file systems.

  • Feedback loops

    You can use for-each blocks to process looping, sequential chains of operations on work items.

  • Service Blocks

    Services blocks let you define a section of work items that should run using a shared Service process

  • PDG Services

    PDG services manages pools of persistent Houdini sessions that can be used to reduce work item cooking time.

  • Integrating PDG with render farm schedulers

    How to use different schedulers to schedule and execute work.

  • Visualizing work item performance

    How to visualize the relative cook times (or file output sizes) of work items in the network.

  • Event handling

    You can register a Python function to handle events from a PDG node or graph

  • Tips and tricks

    Useful general information and best practices for working with TOPs.

  • Troubleshooting PDG scheduler issues on the farm

    Useful information to help you troubleshoot scheduling PDG work items on the farm.

  • PilotPDG

    Standalone application or limited license for working with PDG-specific workflows.

Reference

  • All TOPs nodes

    TOP nodes define a workflow where data is fed into the network, turned into work items and manipulated by different nodes. Many nodes represent external processes that can be run on the local machine or a server farm.

  • Processor Node Callbacks

    Processor nodes generate work items that can be executed by a scheduler

  • Partitioner Node Callbacks

    Partitioner nodes group multiple upstream work items into single partitions.

  • Scheduler Node Callbacks

    Scheduler nodes execute work items

  • Custom File Tags and Handlers

    PDG uses file tags to determine the type of an output file.

  • Python API

    The classes and functions in the Python pdg package for working with dependency graphs.

  • Job API

    Python API used by job scripts.

  • Utility API

    The classes and functions in the Python pdgutils package are intended for use both in PDG nodes and scripts as well as out-of-process job scripts.