Houdini 20.5 Nodes TOP nodes

ROP Fetch TOP node

Creates work items that cook a ROP node or network.

On this page
Since 17.5

This node cooks a ROP node or network using TOPs in either the current .hip file or an external file. It creates one work item per frame, based on the frame generation parameters.

Optionally, you can batch the work items into a single job for all frames, or divide them into smaller groups of frames. This node cooks each batch as a single job on the farm or locally. Batching all frames into one job is typically used for simulations so that the simulation data can be kept in memory rather than be cached to disk. Smaller batch sizes, like a 4-frames-per job, can also help avoid overhead loading the .hip file when cooking a ROP that cooks quickly.

Tip

Oftentimes a custom driver node will write out several files which are not automatically picked up as output files by PDG.

However, if you give the driver node a toggle parameter named pdg_logoutput, and you toggle that parameter to true before it is cooked by the ROP Fetch, then its printed output will be parsed for lines that match the pattern OUTPUT_FILE:{file_path}; and each matched file_path will become output files for the ROP Fetch work items.

Tip

The $HH/help/files/pdg_examples/top_ocean example shows how you can use this node to cook ROPs in an existing scene file.

TOP Attributes

hip

string

The path to the .hip file that contains the ROP node.

outputparm

string

The name of the parameter on the target ROP node that contains the output file path. For example, when used with a ROP Geometry node, this attribute will be set to sopoutput.

executeparm

string

The name of the parameter on the target ROP node that is pressed in order to trigger a cook of the ROP node. If the node is an instance of hou.RopNode the hou.RopNode.render method will be used instead of pressing a button parm. By default, this attribute is set to execute unless a custom button name is specified using the Execute Parm Name parameter.

rop

string

The ROP node path within the .hip file.

top

string

If the ROP is contained within a TOP asset, such as the ROP Geometry node, then this attribute is set to the path to that TOP node.

sourceop

string

If the Source Node Path is set, then this attribute contains the path to the source data used by the ROP this work item is cooking.

control

string

When running a distributed sim, this attribute is set to the path to the Control DOP node.

sharedserver

string

When running a distributed sim, this attribute is set to the identifier for the shared sim tracker process.

trackerhost

string

When running a distributed sim, this attribute is set to the hostname of the machine that has the distributed sim tracker.

trackerport

integer

When running a distributed sim, this attribute is set to the port used to connect to the sim tracker. In combination with trackerhost, this forms the complete address of the simulation tracker. For example, trackerhost:trackerpor.

range

float3

The start, end, and increment frame range values associated with the work item. If the work item is in a batch, then these values refer to the range for the entire batch.

slice

integer

When running a distributed sim, this attribute is set to the slice number associated with the work item.

slicetype

integer

The type of distributed sim being used, set to either 0 for particles or 1 for volumes.

slicedivs

integer3

When using volume slices, this attribute is set to the x, y, and z slice divisions.

tileindex

integer

When using the ROP Fetch node to cook a tiled Mantra render, this value is set to the render tile index associated with the work item.

ifdoutput

file

When using the ROP Fetch node to generate an IFD instead of render a scene, this will be set to the value of Output Picture, which is the path to the image file that will be written when the IFD is rendered with Mantra.

Parameters

ROP Fetch

Generate When

Determines when this node will generate work items. You should generally leave this set to “Automatic” unless you know the node requires a specific generation mode, or that the work items need to be generated dynamically.

All Upstream Items are Generated

This node will generate work items once all of the input nodes have generated their work items.

All Upstream Items are Cooked

This node will generate work items once all of the input nodes have cooked their work items.

Each Upstream Item is Cooked

This node will generate work items each time a work item in an input node is cooked.

Automatic

The generation mode is selected based on the generation mode of the input nodes. If any of the input nodes are generating work items when their inputs cook, this node will be set to Each Upstream Item is Cooked. Otherwise, it will be set to All Upstream Items are Generated.

Cache Mode

Determines how the processor node handles work items that report expected file results.

Automatic

If the expected result file exists on disk, the work item is marked as cooked without being scheduled. If the file does not exist on disk, the work item is scheduled as normal. If upstream work item dependencies write out new files during a cook, the cache files on work items in this node will also be marked as out-of-date.

Automatic (Ignore Upstream)

The same as Automatic, except upstream file writes do not invalidate cache files on work items in this node and this node will only check output files for its own work items.

Read Files

If the expected result file exists on disk, the work item is marked as cooked without being scheduled. Otherwise the work item is marked as failed.

Write Files

Work items are always scheduled and the expected result file is ignored even if it exists on disk.

Cook Type

Determines how work items in the node should cook, e.g. if they should run in process, out of process, or using services.

Service Name

When Cook Type is set to the Service, this parameter is used to specify the name of the service. For more information on using services, see the PDG Service Manager documentation.

ROP Cook Order

When ROP Path points to a chain of ROPs or a ROP network, this parameter determines how the ROP nodes should be cooked. This parameter only affects cooking when the ROP Fetch TOP is configured to use batching or a single work item for all frames.

Frame by Frame

The entire ROP chain is cooked for the current frame before moving onto the next frame in the sequence.

Node by Node

Each ROP node in the chain is cooked for the entire frame range, one ROP at a time.

ROP Path

The path to the ROP node or network that is cooked by this TOP node.

Tip

When cooking as a batch, the target ROP should have the preframe and postframe callback parms.

Execute Parm Name

When on, specifies the name of the button parameter on the ROP Path that triggers a cook of that node. If this parameter is not specified, then the ROP Fetch TOP will first try to use the render method on the ROP, and then fallback to pressing the button if the target ROP node has no render method. By default, the button name is execute.

Reload Parm Path

When on, specifies the path to a parameter that should be be pressed before each frame cooks. This can be useful when cooking a ROP that references nodes that need to manually reload external resources with a Reload button.

Source Node Path

When on, specifies the path to the underlying node or network that the ROP Path will use when the ROP evaluates. This creates a dependency between that node and this TOP node, so that this TOP node is correctly marked as dirty when the source node changes.

For example, if you're using this ROP Fetch node to cook a Geometry ROP, then you would set the value of this parameter to the SOP network associated with your Geometry ROP.

If this parameter is not set, then no dependency is created. Likewise, if External HIP File is on, this parameter is not used since the ROP being cooked is in a different .hip file.

External HIP File

When on, specifies the path to the .hip file that contains the ROP node.

This is most commonly used when generating a .hip file procedurally as part of an earlier step in the TOP network.

When selecting a ROP in the current file, it is not necessary to specify this parameter.

Reset $HIP On Cook

When on, sets the $HIP variable in the job to the value of the $HIP on the submitting machine. This can be used to keep $HIP-based file references in the scene file valid when it is copied to the scheduler working directory at the start of the PDG cook. The original .hip file can then be safely modified while the cook proceeds.

Ignore Scene Load Errors

When this parameter is enabled, work items ignore any errors that occur when loading the .hip file. This can be useful when the scene has a load error that does not interfere with the nodes involved in cooking the ROP Path.

Cook Input ROP Nodes

By default, PDG will cook the target ROP node and any input nodes wired into it. You can disable this parameter to only cook the ROP node specified using the ROP Path and ignore its inputs.

This parameter only matters when cooking a ROP node and won’t have any effect on nodes that render using an “execute” button – for example, a File Cache SOP.

Unlocked Target ROP Node

By default, PDG will unlock the node specified using the ROP Path before cooking it, in order to ensure that frame range and progress reporting parameters are set correctly. You can disable this parameter to stop PDG from unlocking the node, but may see warnings in the work item log if PDG is unable to modify parameters on the ROP.

Output Paths From

Determines how PDG should detect output file cache paths for work items on this node.

ROP Node Parameters

PDG uses the output path(s) from the ROP node specified using the using the ROP Path parameter.

Custom Path List

PDG uses the list of custom output file paths specified by the Output Paths and Output Files parameters.

Output File Tag

Specifies a custom file tag for the output files. By default, the ROP Fetch node tags output files based on their file extensions. You can use this parameter to override that behavior.

This parameter is only available when Output Paths From is set to ROP Node Parameters.

Output Parm Name

Specifies the name of the output file parameter on the target ROP node. By default, for standard ROP nodes shipped with Houdini, PDG is able to automatically determine the output name based on the type of ROP. When using a custom ROP or a ROP that PDG fails to detect automatically, this parameter should be set to the string name of the output parameter on the ROP specified in the ROP Path field. If the output parameter is in a multiparm, you can specify the multiparm like outputfile#. You can also set the parameter to a space-separated list of output parm names.

This parameter is only available when Output Paths From is set to ROP Node Parameters.

Output Paths

Determines the number of custom output files.

This parameter is only available when Output Paths From is set to Custom Path List.

Output File

Specifies the path to a custom output file. This file is added as an expected output to all the work items on this node. The file path can be an expression or contain work item attribute references.

This parameter is only available when Output Paths From is set to Custom Path List.

Output File Tag

Specifies the file tag that should be associated with the corresponding Output File entry. If a tag is not specified, PDG will choose one based on the file’s extension.

This parameter is only available when Output Paths From is set to Custom Path List.

Output Log Parsing

By default, the ROP Fetch TOP looks for a toggle named pdg_logoutput on ROP nodes to determine if it should detect output files by parsing the standard output log when the ROP cooks. You can change that behavior so that PDG will always or never apply log parsing using this parameter.

Log messages that match the format OUTPUT_FILE:<file_name>;<file_tag> are matched and automatically added as work item outputs. The format string can be changed using the __Custom Log Format_ parameter.

None

No log parsing is done, regardless of the settings on the ROP itself

If Supported by ROP Node

Log parsing is only performed is the ROP node has a pdg_logoutput toggle. The toggle is enabled by the ROP Fetch job script.

Always

The output log while cooking work items is always parsed for output file messages.

Custom Log Format

When output log parsing is not disabled by setting Output Log Parsing to None, this parameter determines the regular expression used to match output files in the work item log.

The regular expression must have two capture groups – one for the file path, and one for the file tag. The second can optionally be empty/unset, and PDG will assign a tag based on the extension of the file instead.

Evaluate Using

Determines how the TOP node creates and executes work items.

When using the Single Frame option, the node creates one work item for each upstream frame. When using the Frame Range option, the node creates one work item per frame in the range for each upstream work item. When using the ROP Node Configuration option, PDG cooks the ROP without passing any frame range options and uses whatever configuration is set on the target ROP. When using the Automatic option, the node automatically configures the node based on whether or not it has any inputs.

Single Frame

The node creates a single work item per upstream item and sets the new work item’s frame to match the upstream item’s frame. If there are no input nodes, then the node creates a single work item set to frame 1. You can override the single frame value with the Override Frame parameter.

Frame Range

For each upstream work item, the node creates the full frame range of work items. If there are no upstream items, the node creates one set of items for the specified frame range.

Automatic

If the node has inputs, it generates one frame of work for each input. If the node does not have any inputs, it generates work items for the node’s full frame range.

ROP Node Configuration

For each upstream work item, the node creates a single work item that cooks the ROP Path and PDG does not pass any frame range settings to the ROP node. The number of frames to cook is determined by the parameters on the ROP itself.

Override Frame

When this parameter is enabled, it overrides the frame used with the Single Frame Evaluation mode.

Frame Range

The frame start, end and step size to use when Frame Generation is set to Frame Range.

Cook Frames as Single Work Item

Groups the work for all frames into a single work item.

This is most commonly used when cooking a ROP that produces an FBX, Alembic, or USD file which contains multiple frames of data in the same file.

This parameter is only available when Evaluate Using is set to Frame Range.

Expand Input Files Across Frame Range

Spreads the input file list across the frame range.

For example, if the upstream work item has 100 output files and the frame range on this node is set to 1 to 50, then each work item in this node will have 2 input files. The first work item will have file 1 and file 2, the second work item will have file 3 and file 4, and so on.

This parameter is only available when Evaluate Using is set to Frame Range.

All Frames in One Batch

When on, work items from the same frame range are grouped into a batch that is scheduled as a single job. This is used for simulations where each frame depends on the previous frame, and it is advantageous to keep everything in memory.

Frames per Batch

By default, when this parameter is at 1 each work item is cooked as a separate job. When this parameter is set to a value greater than 1, work items are grouped into batches.

Cook Batch When

Determines when batches of frames are scheduled.

By default, the batch is scheduled once the dependencies for all frames are cooked. However, it is possible to schedule the batch as soon as its first frame can run.

All Frames are Ready

The batch is scheduled only when all dependencies on all frames in the batch are satisfied.

First Frame is Ready

The batch is scheduled as soon as its first frame is ready.

This is possible because the ROP wrapper script used by PDG ensures that subsequent frames correctly wait for their dependencies. This is done by communicating back to PDG as the job is running to check the status of the dependencies before each frame. This incurs slightly more network overhead, and requires ongoing communication between the job and PDG.

This option is not available when cooking a Composite ROP node since COP file nodes must be able to load their entire input file sequence on the first frame.

Fixed Number of Frames are Ready

The batch is scheduled once a fixed number of frames have cook dependencies, starting from the first frame in the range. The number of frames is determined by the Number of Frames parameter.

For example, if a batch is cooking from frame 1 to frame 24 and the Number of Frames parameter is set to 6, the batch will be scheduler after the dependencies for batch frames 1 to 6 have cooked.

Percentage of Frames are Ready

The batch is scheduled once a percentage of its frames dependencies have been cooked. The percentage of frames is determined by the Percentage of Frames parameter.

Automatically Set Missing Frames

When on, the node will attempt to assign a frame and range value to work items that are missing those values. It determines the range based on the number of input work items, and sets the frame based on the work item’s position in the input list. If only some of the input work items are missing a frame, the behavior may be inconsistent.

Resume Partially Cached Batches

Determines whether or not batches produced by this node should resume cooking from partially cached output files. You should only enable this option for simulations if the DOP network itself is already configured to write sim cache files.

Distribution

Distributed Sim

When on, the node treats the target ROP as part of a distributed simulation. This enables the parameters for configuring a distributed sim, and disables the frame generation parameters.

Control DOP

The control DOP for the distributed sim.

Custom Tracker Port

When this parameter is enabled, the specified port is used for the sim tracker that transmits slice information between jobs. Otherwise, if this parameter is disabled, PDG will automatically use an available port.

Web Port

When this parameter is enabled, the specified port is used to host the web page for monitor distributed sim status. Otherwise, if this parameter is disabled, PDG will automatically use an available port.

Slice Type

The type of slicing used for the distributed sim.

Particles

Use the particle slicing method for the distributed sim.

Volumes

Use the volume slicing method for the distributed sim.

Slice Count

Specifies the number of slices for the distributed sim.

This parameter setting is only available when Slice Type is set to Particles.

Slice Divisions

Determines the x, y, and z volume divisions for the distributed sim.

This parameter setting is only available when Slice Type is set to Volumes.

Debugging and Performance

Enable Performance Monitor Logging

When on, the Performance Monitor is enabled when cooking the ROP. The Performance Monitor prints performance information to the work item log.

Performance File

When on, this parameter determines where the performance monitor output is saved to on disk. When off, the output is written to the work item’s output log instead.

If the specified file ends with a .csv extension, then it is exported as the CSV format. If the specified file is any other file type, then the file is saved using the Houdini performance monitor output syntax. A file in that format can be imported back into the Performance Monitor pane tab for viewing.

One file is written out for each process that performs a ROP cook. For example, if batching is enabled, then a single .hperf file is written out for each batch process. If every frame is cooked in a separate process, then one performance file is written out for each frame.

This parameter is only available when Enable Performance Monitor Logging is on.

Save Debug .hip File

When on, the state of the .hip file containing the ROP node is saved to the specified path after the cook has completed. This includes all wedge parameter and environment changes, or scene state modified by cooking the network. If the cook fails due to an error in the ROP evaluation, then the .hip file is still saved.

Report Debug Files as Outputs

When on, if the generation of the performance monitor output file (Performance File) or debug .hip file (Save Debig .hip File) is enabled, then they are reported (attached) as work item outputs. This provides easier access to the files.

Schedulers

TOP Scheduler Override

This parameter overrides the TOP scheduler for this node.

Schedule When

When enabled, this parameter can be used to specify an expression that determines which work items from the node should be scheduled. If the expression returns zero for a given work item, that work item will immediately be marked as cooked instead of being queued with a scheduler. If the expression returns a non-zero value, the work item is scheduled normally.

Work Item Label

Determines how the node should label its work items. This parameter allows you to assign non-unique label strings to your work items which are then used to identify the work items in the attribute panel, task bar, and scheduler job names.

Use Default Label

The work items in this node will use the default label from the TOP network, or have no label if the default is unset.

Inherit From Upstream Item

The work items inherit their labels from their parent work items.

Custom Expression

The work item label is set to the Label Expression custom expression which is evaluated for each item.

Node Defines Label

The work item label is defined in the node’s internal logic.

Label Expression

When on, this parameter specifies a custom label for work items created by this node. The parameter can be an expression that includes references to work item attributes or built-in properties. For example, $OS: @pdg_frame will set the label of each work item based on its frame value.

Work Item Priority

This parameter determines how the current scheduler prioritizes the work items in this node.

Inherit From Upstream Item

The work items inherit their priority from their parent items. If a work item has no parent, its priority is set to 0.

Custom Expression

The work item priority is set to the value of Priority Expression.

Node Defines Priority

The work item priority is set based on the node’s own internal priority calculations.

This option is only available on the Python Processor TOP, ROP Fetch TOP, and ROP Output TOP nodes. These nodes define their own prioritization schemes that are implemented in their node logic.

Priority Expression

This parameter specifies an expression for work item priority. The expression is evaluated for each work item in the node.

This parameter is only available when Work Item Priority is set to Custom Expression.

External Files

File Dependencies

A list of files that should be copied to the PDG working directory before the first work item in this node is executed. This can be used to ensure that supporting files like digital assets and custom scripts are available for the work item job.

The specified paths can be absolute or relative to HOUDINI_PATH.

Examples

ROPFetch Example for ROP Fetch TOP node

This example demonstrates how to use the ROP Fetch TOP node to create work items that cook a ROP node.

See also

TOP nodes