Houdini 20.0 Executing tasks with PDG/TOPs

Using attributes

TOP attributes are like point attributes, You can use them to set parameters in Houdini nodes called by the work items.

On this page

Overview

A work item contains attributes, named bits of information similar to attributes on a point in Houdini geometry. The script or executable associated with the work item can read the attributes to control the work they do. Attributes are passed down to work items created from “parent” work items, so you can use them to have the result of a work item affect the processing of its child items, group items together, and pass information down through the network.

Custom code can read attributes to control generation of work items, however the most common use of attributes is to reference them in parameters of TOP nodes and/or Houdini networks and nodes that are referenced by the TOP network.

  • For example, let’s say you want to wedge different render qualities. You can use the Wedge TOP to create work items that have a pixelsamples attribute set to different values. Then, in the ROP Mantra Render TOP, you can set Pixel samples using @pixelsamples to reference the attribute value.

  • You can also reference work item attributes in external assets/networks called by the TOP network. For example, the HDA Processor cooks a Houdini asset for each work item. You can use @attribute references in that asset’s parameters to pull values from the work item.

You can reference a component of a vector using @attribute.component, where component is a zero-based number, or x, y, or z (equivalent of 0, 1, 2). For example @pos.x or @pdg_output.0. You can reference all components of an attribute array as a space-separated string using pdgattribvals.

Attribute types

Work items support basic string, float and integer array attributes much like Houdini geometry. All primitive attribute types are arrays, and the array length can be arbitrarily long. Work items also have a File attribute type, which is similar to a string but stores additional metadata such as the file tag and the modification time of the file. File attributes are displayed as clickable links in the work item info panel, and can be used with the path handling mechanism in TOPs to make them usable on both a local machine and farm machine.

There are two additional types of attributes that can be created on work items for use cases that require complex structured data. The first of these is the Python Object attribute type, which can be assigned a single PyObject value. TOPs is only able to display basic information about the contents of the attribute using the Python Object’s string representation, and many nodes cannot operate directly on data stored in that type of attribute. Full access to the object is possible through the PDG Python API or in TOP nodes that explicitly make use of a Python Object attribute such as the ShotGrid nodes. If the Python Object consists of nested array and dictionary objects, values in the object can be accessed using the @ operator in HScript. For example @obj.key.subkey.2.value, which would be equivalent to the Python code work_item['obj'].object['key']['subkey][2]['value]. @obj will always evaluate to the standard string representation of the Python Object. This is the same as doing repr(work_item['obj'].object) in Python.

The second complex attribute type is the Geometry attribute, which can store native Houdini geometry data. The geometry attribute type primarily used by the Geometry Import and Invoke TOP nodes, which are able to operate on geometry directly in the current Houdini process. Like the Python Object type, most nodes aren’t able to directly access or make use of the attribute, however the attribute contents can be inherited between work items even if the node creating the work items cannot use geometry. A TOP Geometry node in a SOP network is able to pass geometry attributes on output work items back to SOPs.

Attribute flags

Work item attributes can have various flags set that determine the way they're accessed and presented to other parts of the TOP network. Common flags include No Copy which indicates that an attribute should not be copied onto downstream work items, and Env Export which causes the attribute value to be exported as an environment variable when cooking out of process. A full list of supported flags is listed in the pdg.attribFlag documentation.

Global vs. work item attributes

By default, nodes that operate on attributes create those attributes directly on their work items. These are called work item attributes. However, it is also possible to make attributes accessible to all nodes in a TOP graph (/topnet) by storing the attribute data globally on the graph itself. These are called global attributes.

You can create global attributes with the Attribute Create TOP node or with the PDG Python API using the Python Script TOP or Python Processor TOP node.

You can also access global attributes with the @attribute syntax as long as no work item attributes with the same name exist in their scene, or with the PDG Python API by using any of the methods for creating and querying attribute data. For example, like the pdg.Graph subclasses from pdg.AttributeOwner.

You can bind global attributes to a work item. This means that if you delete the work item, then all its bound global attributes will also be removed from the graph. This is useful in situations where the work being done by a particular work item creates a global attribute. For example, like when a work item loads a configuration file and sets global attributes based on the fields found in that file.

Notes

  • Global attributes work in-process and out-of-process.

  • When the Save Graph Attributes to .hip parameter is turned on for a TOP Network, all its unbound global attributes will be persistent to its .hip file.

Using attributes in expressions (pull)

When nodes are “called” as part of a TOP network cooking (for example, when TOPs cooks an HDA Processor, or renders frames using a ROP Mantra Render), it fills in any @attribute references using the cooking work item’s attributes.

Tip

When nodes cook for display in the viewer, they fill in @attribute references using the attributes of the currently selected work item. This means you can click around work items and see the visible output of a network change to reflect the output of the selected item.

These are called “pull” references.

If you don’t want to or can’t put TOPs-specific expressions in certain parameters, you can instead use an alternate “push” mechanism where you specify in the TOP network parameters to overwrite at run time.

String parameters

An @attribute reference by itself is an Hscript expression, just like ch('../geo1/tx') * 2 is an expression. You can use @attribute references as part of HScript expressions to compute values in node parameters.

Remember that if you want to use an expression in a string parameter, you must escape it by enclosing it in backticks (`). For example, to reference the work item’s built-in output attribute in a file path parameter, you must enclose the reference in backticks:

`@pdg_output`

You can incorporate an expression within “normal” text in the parameter. For example, to use the work item’s frame number in a file path:

$HIP/output/$OS_`@pdg_frame`.exr

Important Note

If it’s ambiguous what an @attribute expression refers to (for example, a geometry point attribute and a TOP work item attribute have the same name), SOP attributes take precedence over TOP work item attributes.

However, you can avoid that situation by using the prefixes for the @ operator in expressions. These prefixes force the operator to use data from a particular source. The valid prefixes are: G for geometry, C for context options, and P for PDG work item attributes. For example, P@size will always resolve to the size work item attribute even if there is an available context option or geometry attribute with the same name. This @ operator technique is very useful when you want to use a mix of context options and PDG attributes in the same TOP network.

Replacing parameter contents at run time (push)

When using the Wedge TOP to vary parameter values, if you don’t want to put TOP-specific expressions on other nodes (for example, if the ROP you want to render is shared with other pipelines or used manually), you can have the Wedge node replace parameter values with wedge attributes at run time.

See the help for the Wedge TOP for how to override parameters at render time.

Built-in attributes

Note

When referencing built-in attributes, use @pdg_name, for example, @pdg_index.

index

Used as a key for sorting work items within a node.

id

The unique ID of the work item.

name

Makes it easier to distinguish individual work items in the interface.

nodename

The name of the node that created the work item.

schedulername

The name of the scheduler assign to the work item’s node.

label

The label of the work item. Falls back to name if the work item does not have a custom label.

frame

The frame number this item is working on. Of course, not all work items are doing rendering, however needing to store a frame number is common enough that we included it in the built-in attributes.

input

A list of filenames used as the input for this bit of work. For example, if the work item represents “draw text over a rendered image”, this would be the path to the image to manipulate.

inputsize

The number of input files.

output

A list of filenames produced as the output of this bit of work. For example, if the work item represents “draw text over a rendered image”, this would be the path to the output image with the text drawn into it.

outputsize

The number of output files.

Tip

As processor nodes generate new work items from incoming work items, the output attribute of the “parent” work item is automatic assigned as the input attribute of the “child” work item.

Attribute Pattern Syntax

When using nodes like the Attribute Delete or Attribute Copy it is often useful to specify a list of attributes using a pattern. PDG has a special attribute pattern syntax that can you can use to describe which attributes should be matched in nodes that can operate on multiple attributes. The pattern is applied from left to right and consists of a space-separated sequence of match conditions using the symbols described below.

It is also possible to use the pattern matching functionality in a Python Script or in your own custom nodes. You can construct an pdg.AttributePattern object from a string and use it with the pdg.AttributeOwner.attribMatch function to apply the pattern to attributes on a work item. Alternatively, you can call the pdg.AttributePattern.contains function directly to match against a specific attribute name and type.

Pattern

Meaning

*

Match all attribute names, of any type

name

Match a specific attribute name

prefix*

Match any attribute name with the prefix. for example file*

*suffix

Match any attribute name with the suffix, for example *size

^pattern

Exclude attributes matching the pattern from the results of the preceding patterns. For example, the pattern * ^size* will match all attributes except for ones that begin with the prefix “size”.

:type

Suffix that indicates the pattern should only match attributes of the specified pdg.attribType. The type can be float, integer, string, file, geometry or pyobject. For example, size*:float *:integer ^sizesmall will match all float attributes with the prefix size, except for the one named sizesmall, and all integer attributes.

Value Pattern Syntax

You can use a value pattern to specify a sequence of values using a standard syntax. For example, you can create a string or numeric value array using a pattern with the Attribute Array node.

The values in the pattern are processed from the left to right. Multiple patterns or constant values can be specified by adding white space between them. It possible to include special characters in string values by putting the string in double quotes, or by using the \ escape character before the special character. For example, the pattern regular "quoted string" escaped\ string would produce the string array ['regular', 'quoted string', 'escaped string'].

By default ranges will include the first element, but exclude the last element in the range. For example, the pattern 1-5 refers to the sequence of values 1, 2, 3, 4. Value ranges can only be used if the start and end values are numeric, or if the values are strings with a numeric suffix. The string prefix must be the same for both values. For example node1-node10 is a valid pattern but node1-other10 is not valid. The step size for a range must always be a purely numeric value can cannot contain a string.

Ranges can also be specified using a prefix, followed by a set of brackets [..] that contain the numeric value range. This is useful with specifying a list of strings with a common string prefix, followed by a range of numeric suffixes. For example, the pattern prefix[1-3] generates the sequence prefix1, prefix2, prefix3.

Some nodes, such as the Range Generate, will choose to include the last element in the range instead of excluding it. This makes the value pattern behave more like other frame range parameters that accept a start, end, and step. The behavior of the last element in value ranges can also be configured when creating a custom pdg.ValuePattern instance in Python code.

You can exclude one or more values by putting the ^ exclusion character at the beginning of a pattern. Any values from previous pattern segments that match the exclusion pattern will be ignored, however the exclusion will not apply to subsequent patterns. Exclusions can also be defined as a range. For example 1-10 ^4-6 will produce the sequence 1, 2, 3, 6, 7, 8, 9. If another pattern is added to the end the exclusion will not apply to it – for example, 1-10 ^4-8 5 6 will produce the sequence 1, 2, 3, 5, 6, 7, 9.

It’s also possible to use the value pattern functionality in a Python Script or in your own custom nodes. You can construct an pdg.ValuePattern object from a string and use the pdg.ValuePattern.contains function to test if a value is in the pattern. The entire array of values that match the pattern is available through the pdg.ValuePattern.floatArray, pdg.ValuePattern.intArray, or pdg.ValuePattern.stringArray functions.

Pattern

Meaning

value

A constant value. Can be used with float, integer or string values.

start-end

A range of values between start and end. Can only be used with float or integer values sequences, or strings with a numeric suffix.

start-end:step

A range of values with a custom step size. Can only be used with float or integer values sequences, or string with a numeric suffix.

^pattern

Excludes any values from the pattern that are found in preceding pattern segments. pattern can be any valid Value Pattern. For example, 1-20 ^4-15 5 6 will produce the value sequence 1, 2 3, 5, 6, 15, 16, 17, 18, 19.

prefix[start-end:step]

Generates a range of values with a common string prefix. For example, prefix[2-10:2] will produce the value sequence 'prefix2', 'prefix4', 'prefix6', 'prefix8'.

Filter Pattern Syntax

The filter syntax is an extended form of the value pattern syntax that allows you to specify the name of an attribute and a pattern to match with the attribute, and apply that pattern to a work item using the pdg.FilterPattern class. A filter pattern consists of a space-separated set of rules. Each rule begins with the name of an attribute, a comparison operator like > or =, and a value pattern to compare against the attribute. The attribute name can also include an optional array index suffix, specified using either a letter or number. For example, @pdg_id>5 and @scale.x <= 2.0f @scale.z <= 3.0 @array[5]=10 are two examples of valid filter patterns.

The name can be any valid work item attribute name, or one of the following built-ins: @pdg_id, @pdg_index, @pdg_frame, @pdg_loopdepth, @pdg_name, @pdg_label, @pdg_output.

The = and != operator can be compared with arbitrary value patterns that include ranges of values, for example @name=task[2-10] compares the name attribute against a list of possible string values. The >, >=, <, and <= operators are only valid if the pattern on the right hand side produces a single scalar value – they cannot be used with ranges. For example, @scale.z>5 is valid, but @scale.z>1-5 is not.

Entries in the pattern can be turned into exclusions by prepending them with the ^ token. For example, the pattern @width>5 ^@height<4 will match all work items with a width attribute that’s larger than five, except for ones that also have a height less than 4.

Executing tasks with PDG/TOPs

Basics

Beginner Tutorials

Next steps

  • Running external programs

    How to wrap external functionality in a TOP node.

  • File tags

    Work items track the results created by their work. Each result is tagged with a type.

  • PDG Path Map

    The PDG Path Map manages the mapping of paths between file systems.

  • Feedback loops

    You can use for-each blocks to process looping, sequential chains of operations on work items.

  • Service Blocks

    Services blocks let you define a section of work items that should run using a shared Service process

  • PDG Services

    PDG services manages pools of persistent Houdini sessions that can be used to reduce work item cooking time.

  • Integrating PDG with render farm schedulers

    How to use different schedulers to schedule and execute work.

  • Visualizing work item performance

    How to visualize the relative cook times (or file output sizes) of work items in the network.

  • Event handling

    You can register a Python function to handle events from a PDG node or graph

  • Tips and tricks

    Useful general information and best practices for working with TOPs.

  • Troubleshooting PDG scheduler issues on the farm

    Useful information to help you troubleshoot scheduling PDG work items on the farm.

  • PilotPDG

    Standalone application or limited license for working with PDG-specific workflows.

Reference

  • All TOPs nodes

    TOP nodes define a workflow where data is fed into the network, turned into work items and manipulated by different nodes. Many nodes represent external processes that can be run on the local machine or a server farm.

  • Processor Node Callbacks

    Processor nodes generate work items that can be executed by a scheduler

  • Partitioner Node Callbacks

    Partitioner nodes group multiple upstream work items into single partitions.

  • Scheduler Node Callbacks

    Scheduler nodes execute work items

  • Custom File Tags and Handlers

    PDG uses file tags to determine the type of an output file.

  • Python API

    The classes and functions in the Python pdg package for working with dependency graphs.

  • Job API

    Python API used by job scripts.

  • Utility API

    The classes and functions in the Python pdgutils package are intended for use both in PDG nodes and scripts as well as out-of-process job scripts.