Houdini 20.5 Executing tasks with PDG/TOPs

Job API

Python API used by job scripts.

On this page

Overview

Work items (jobs) which cook out-of-process will often want to send data or their statuses back to PDG. The PDG Scheduler can track when a job starts and if it succeeds or fails independent of the job script. However, if the job produces output files that contain information PDG should know about, then you can use this API to report that information back to PDG. In addition to outputting files, you can also use this API to write attributes into running work items.

The Job API has two main modules:

  • pdgjson.py provides a high-level API which matches pdg.WorkItem. You can use this module to read and write work item attributes and input/output files. This is the recommended API.

  • pdgcmd.py provides a lower-level API which is used by pdgjson.py. You can also use this module when necessary.

These python modules are automatically copied to $PDG_TEMP/scripts by the PDG Scheduler.

Basic Usage

Python Module

Behavior

pdgjson

Provides a high-level WorkItem API similar to what is available in PDG. Useful top-level imports are pdg.WorkItem, pdg.File, pdg.attribType, pdg.attribFlag, pdg.workItemType, and pdg.workItemState. You can obtain the WorkItem object by calling pdgjson.WorkItem.fromJobEnvironment().

from pdgjson import WorkItem

# create WorkItem object from the data serialized to `$PDG_TEMP/data/workitem.json`
work_item = WorkItem.fromJobEnvironment()

# read attrib values from the running work item
val = work_item.attribValue('myintattrib')

# Send data back to PDG work item via network calls
work_item.setStringAttrib('runtime_attrib', 'test value', 0)
work_item.addOutputFile('/tmp/myoutput.txt')

pdgcmd

Provides a low-level API to communicate with PDG via RPC function calls.

from pdgcmd import addOutputFile

# Send data back to PDG work item via network calls
pdgcmd.setStringAttrib('runtime_attrib', 'test value', 0)
pdgcmd.addOutputFile('/tmp/myoutput.txt')

pdgcmd Functions

Methods

Executing tasks with PDG/TOPs

Basics

Beginner Tutorials

Next steps

  • Running external programs

    How to wrap external functionality in a TOP node.

  • File tags

    Work items track the results created by their work. Each result is tagged with a type.

  • PDG Path Map

    The PDG Path Map manages the mapping of paths between file systems.

  • Feedback loops

    You can use for-each blocks to process looping, sequential chains of operations on work items.

  • Service Blocks

    Services blocks let you define a section of work items that should run using a shared Service process

  • PDG Services

    PDG services manages pools of persistent Houdini sessions that can be used to reduce work item cooking time.

  • Integrating PDG with render farm schedulers

    How to use different schedulers to schedule and execute work.

  • Visualizing work item performance

    How to visualize the relative cook times (or file output sizes) of work items in the network.

  • Event handling

    You can register a Python function to handle events from a PDG node or graph

  • Tips and tricks

    Useful general information and best practices for working with TOPs.

  • Troubleshooting PDG scheduler issues on the farm

    Useful information to help you troubleshoot scheduling PDG work items on the farm.

  • PilotPDG

    Standalone application or limited license for working with PDG-specific workflows.

Reference

  • All TOPs nodes

    TOP nodes define a workflow where data is fed into the network, turned into work items and manipulated by different nodes. Many nodes represent external processes that can be run on the local machine or a server farm.

  • Processor Node Callbacks

    Processor nodes generate work items that can be executed by a scheduler

  • Partitioner Node Callbacks

    Partitioner nodes group multiple upstream work items into single partitions.

  • Scheduler Node Callbacks

    Scheduler nodes execute work items

  • Custom File Tags and Handlers

    PDG uses file tags to determine the type of an output file.

  • Python API

    The classes and functions in the Python pdg package for working with dependency graphs.

  • Job API

    Python API used by job scripts.

  • Utility API

    The classes and functions in the Python pdgutils package are intended for use both in PDG nodes and scripts as well as out-of-process job scripts.