Houdini 18.0 Executing Tasks

External configuration and data

How to read external configuration and source data in TOPs and use it to drive work.

On this page


It is often useful to drive a TOP network using a configuration file, or input data, or both. For example, you might have a settings file that defines a bunch of variables to use when running the network, and you might run the network once for each part listed in a manifest, or for each asset listed in an asset management system.

TOPs lets you read external data and generate work items/attributes based on the data.

Loading data

  • SQL Database:

    SQL Input runs a query on a database server and generates a work item for each returned row, with attributes taken from the columns.

    SQL Output takes incoming work items and writes an INSERT query to insert a row for each work item, with the columns taken from attributes you specify.

  • CSV (comma separated values, typically exported by a spreadsheet):

    CSV Input reads a CSV file and generates a work item for each row, with attributes taken from the columns.

    CSV Output takes incoming work items and writes out a new CSV file with a row for each work item, with the columns taken from the attributes.

  • JSON files:

    JSON Input generates work items with attributes based on JSON files. Because JSON is so free-form, the node has parameters to try to extract items and attributes from various data "shapes".

    JSON Output takes incoming work items and writes out a JSON file with a list of objects representing work items, containing key/value pairs taken from the attributes.

  • The Environment Edit node lets you add extra environment variables to the environment in which work is run.

Manipulating data

  • Using Python:

    If you don’t mind programming, the simplest and most flexible way to pre-process data, or manipulate existing attributes, is with a Python snippet.

    If you want to edit one or more attributes on every incoming work item:

    1. Add a Python Script node. The Python Script node lets you edit incoming work items one at a time.

    2. In the parameters, turn on In process.

    3. Write a script to manipulate attributes. For example:

      # Assume we've ingested data that sets the "detail" attribute to a string
      # such as "low", "medium", or "high", and we want to translate that into
      # a numeric value, eg. -1, 0, or 1.
      # Define a dictionary mapping string values to numeric values
      lookup = {"low": -1, "medium": 0, "high": 1}
      # Get the value of the "detail" string attribute
      detail = work_item.intAttribValue("detail")
      # Translate the string into the numeric equivalent
      level = lookup.get(detail, 0)
      # Create a new "level" attribute with the numeric equivalent
      work_item.setIntAttrib("level", level)

    If you want to compute aggregate statistics (for example, the average of an attribute):

    1. Add a Wait for All. This will pause processing until all work items are available, so the average will include all values.

    2. After the Wait for All, add a Python Script node.

    3. In the parameters, set Work item generation to "Dynamic" and turn on In process.

    4. In the script, you can use parent_item to refer to the single item in the Wait for All nodes. That item has a partitionItems list attribute you can use to access the items in the partition:

      # Get the work items in the input Wait for All's partition
      items = part.partitionItems
      # Calculate the average of the "foo" attribute
      total = sum(it.floatAttribValue("foo") for it in items)
      average = total / float(len(items))
      # Set the average as an attribute on the outgoing item
      work_item.setFloatAttrib("average", average)
  • Alternatively, you can use TOP nodes that manipulate attributes:

    Attribute Create. You can add or redefine attributes using this node, and you can use expressions to compute the value of the new attribute based on existing attributes.

    Attribute Copy duplicates attributes from work items in one branch onto the work items in another branch, matching the work items up by index or by an attribute value.

    Attribute Delete removes attributes from work items. This can be useful, for example, before outputting to CSV to prevent "scratch" attributes from being written to disk.

    Attribute from String creates attributes by parsing components of an input string. This node is useful for extracting frame or shot information from a file path or parsing string data loaded by the CSV or JSON input nodes.

Executing Tasks


Beginner Tutorials

Next steps


  • All TOPs nodes

    TOP nodes define a workflow where data is fed into the network, turned into "work items" and manipulated by different nodes. Many nodes represent external processes that can be run on the local machine or a server farm.

  • Processor Node Callbacks

    Processor nodes generate work items that can be executed by a scheduler

  • Partitioner Node Callbacks

    Partitioner nodes group multiple upstream work items into single partitions.

  • Scheduler Node Callbacks

    Scheduler nodes execute work items

  • Custom File Tags and Cache Handlers

    PDG uses file tags to determine the type of an output file.

  • Python API

    The classes and functions in the Python pdg package for working with dependency graphs.