Houdini 20.5 Executing tasks with PDG/TOPs

Tips and tricks

Useful general information and best practices for working with TOPs.

On this page

Cooking

  • Test your network on small data sets, and make sure everything works, before running on big data sets.

  • Use Tasks ▸ Generate Static Work Items (See static vs. dynamic) to get a sense of the work to be done before cooking the network.

Work items

Load times

Work items on the HDA Processor TOP node and the ROP Fetch TOP node run out-of-process. This means that these Processor nodes have to start a new Houdini session for each of their work items. As a result, more time may be spent starting the process for a work item than is spent to do the actual work, and this can slow down the load times for your work items.

There are two ways to avoid this issue:

  • The ROP Fetch TOP and HDA Processor TOP support batch cooking. This is the process by which multiple frames of work are cooked as a single process. For example, you can turn on the All Frames in One Batch parameter on the ROP Fetch TOP node, the ROP Geometry TOP node, and other Processor nodes to cook an entire frame range with a single process instead of using one process per frame.

  • The ROP Fetch TOP and HDA Processor TOP support PDG Services. Services are long-running pools of processes that can cook multiple work items at the same time. Services also avoid the cost of starting up a Houdini session since the Service worker processes are pre-created and reused by your work items.

Attributes

  • Use “pull” attribute references. Use @attribute syntax in parameters to control what gets done (see pull and push references). Not only is it much easier than pushing attributes or writing custom code, PDG has an important feature that lets you resume a cook after being interrupted – but only if you used @attribute syntax so PDG can predict what output files would have been named.

  • In interactive Houdini, attribute references come from the currently selected item. When other networks (such as SOP networks for generating geometry and COP2 networks for compositing renders) in the HIP file have @attribute values, the “current” values of those references are driven by which work item is selected in the network interface.

    You can, for example, set a viewer pane to show the SOP network output, and then click around the work items in the TOP network, showing the different geometry output by the different items.

    You can also take advantage of this to quickly preview output geometry. The Show Output shelf tool creates a Geometry object, and inside create a File SOP, then sets the File node’s path to `@pdg_output` (the backticks are necessary to escape the expression since the path is a string parameter). When you view the network in a viewer, as you click work items around the TOP network, you can see their geometry output. You can then delete the object/network when you're done inspecting geometry output.

    See attribute references for more information.

Simulations

  • Do not generate simulation frames in parallel. When generating simulation output using the ROP Geometry Output node, go to the ROP Fetch tab and turn on All Frames in One Batch.

    If you leave this option off, the separate frames will be generated in parallel. However, since simulations must cook everything up to the frame to render, this leads to a situation where to generate frame 2, Houdini must cook frames 1 and 2, to generate frame 3 it must cook frames 1, 2, and 3, and so on. This is very inefficient.

Output files

See file paths for more information.

  • Try to separate temporary “work in progress” files from “final output” files in separate directories. For example, generate intermediate geometry files in __PDG_TEMP__/geo, and final geometry files in __PDG_DIR__/geo. This makes it easier to find the important output, and lets you clear out the temp files wholesale to force a rebuild.

  • You may want to avoid using destructive file operations (move/rename, remove). They can make files appear to be missing, triggering the dependency system to do extra work.

Executing tasks with PDG/TOPs

Basics

Beginner Tutorials

Next steps

  • Running external programs

    How to wrap external functionality in a TOP node.

  • File tags

    Work items track the results created by their work. Each result is tagged with a type.

  • PDG Path Map

    The PDG Path Map manages the mapping of paths between file systems.

  • Feedback loops

    You can use for-each blocks to process looping, sequential chains of operations on work items.

  • Service Blocks

    Services blocks let you define a section of work items that should run using a shared Service process

  • PDG Services

    PDG services manages pools of persistent Houdini sessions that can be used to reduce work item cooking time.

  • Integrating PDG with render farm schedulers

    How to use different schedulers to schedule and execute work.

  • Visualizing work item performance

    How to visualize the relative cook times (or file output sizes) of work items in the network.

  • Event handling

    You can register a Python function to handle events from a PDG node or graph

  • Tips and tricks

    Useful general information and best practices for working with TOPs.

  • Troubleshooting PDG scheduler issues on the farm

    Useful information to help you troubleshoot scheduling PDG work items on the farm.

  • PilotPDG

    Standalone application or limited license for working with PDG-specific workflows.

Reference

  • All TOPs nodes

    TOP nodes define a workflow where data is fed into the network, turned into work items and manipulated by different nodes. Many nodes represent external processes that can be run on the local machine or a server farm.

  • Processor Node Callbacks

    Processor nodes generate work items that can be executed by a scheduler

  • Partitioner Node Callbacks

    Partitioner nodes group multiple upstream work items into single partitions.

  • Scheduler Node Callbacks

    Scheduler nodes execute work items

  • Custom File Tags and Handlers

    PDG uses file tags to determine the type of an output file.

  • Python API

    The classes and functions in the Python pdg package for working with dependency graphs.

  • Job API

    Python API used by job scripts.

  • Utility API

    The classes and functions in the Python pdgutils package are intended for use both in PDG nodes and scripts as well as out-of-process job scripts.