| On this page | |
| Since | 22.0 |
Overview ¶
This node converts COP nodes or image files into a dataset that’s
compatible with the 
ML Train Computer Vision TOP.
It prepares a .json file that uses the same format as the COCO dataset
to represent the images, sizes and masks. When using COPs, the node writes the
source image data to disk so that it’s available for out-of-process training
tasks.
Input Format ¶
This node generates a dataset from a set of images and per-object masks. The input images can be specified in a variety of ways using the parameters on the node. It’s also possible to specify a complete input dataset mapping as a dictionary attribute, which can be created using other TOP nodes or data processing networks.
Internally, the node uses a dictionary in the following format:
{
"<source_file_path>": {
"<class_id>": [{
"mask": "<mask_file_path>",
"keypoints" : {
"<keypoint_id>": [x, y],
"<keypoint_id>": [x, y]
}
},{
"mask": "<mask_file_path>",
"keypoints" : {
"<keypoint_id>": [x, y],
"<keypoint_id>": [x, y]
}
},
...
],
"<class_id>": [ {
"mask": "<mask_file_path>",
"keypoints" : {}
},{
"mask": "<mask_file_path>",
"keypoints" : {}
},
...
],
...
},
...
}The class_id and keypoint_id should correspond to entries defined in the
Object Classes and Keypoints multiparms on the node.
At the top level, the dictionary should have a key for each source image file that corresponds to a color image in the training set. Each source image file key maps to a sub dictionary that has object class IDs for keys, each of which map to an array containing the mask file(s) for the map for that object class.
By setting the Input Format parameter to Dict Attribute, it’s possible to pass in a custom dictionary to generate a final dataset. The node will still copy images found in the custom dictionary into the dataset output directory to make sure the dataset is self contained.
Parameters ¶
Base Directory
The base path that contains all of the inputs, outputs, and virtual environment used during the training process. Outputs are produced in specific subdirectories depending on the type of output file.
Base Name
The name of the subdirectory that contains all outputs produced by this node.
Dataset Name
The name of the subdirectory that contains datasets produced by this node
Output Directory
The directory to output preprocessed results. By default, this is relative to the Base Directory, Base Name and Dataset Name.
Generate When
Determines when this node will generate work items. You should generally leave this set to “Automatic” unless you know the node requires a specific generation mode, or that the work items need to be generated dynamically.
All Upstream Items are Generated
This node will generate work items once all of the input nodes have generated their work items.
All Upstream Items are Cooked
This node will generate work items once all of the input nodes have cooked their work items.
Each Upstream Item is Cooked
This node will generate work items each time a work item in an input node is cooked.
Automatic
The generation mode is selected based on the generation mode of the input nodes. If any of the input nodes are generating work items when their inputs cook, this node will be set to Each Upstream Item is Cooked. Otherwise, it will be set to All Upstream Items are Generated.
Cache Mode
Determines how the processor node handles work items that report expected file results.
Automatic
If the expected result file exists on disk, the work item is marked as cooked without being scheduled. If the file does not exist on disk, the work item is scheduled as normal. If upstream work item dependencies write out new files during a cook, the cache files on work items in this node will also be marked as out-of-date.
Automatic (Ignore Upstream)
The same as Automatic, except upstream file writes do not invalidate cache files on work items in this node and this node will only check output files for its own work items.
Read Files
If the expected result file exists on disk, the work item is marked as cooked without being scheduled. Otherwise the work item is marked as failed.
Write Files
Work items are always scheduled and the expected result file is ignored even if it exists on disk.
Object Classes
The number of object classes in the images.
Class Name
The human-readable name for the object class.
Class Color;
Class ID
A unique ID for the class.
Keypoints
The number of keypoints in the datasets. These values are common to all training images.
Keypoint Name
The human-readable name for the keypoint.
Keypoint Color
The display color of the keypoint in testing and visualization output.
Keypoint ID
A unique ID for the keypoint.
Input Source
Determines how the input image data is specified.
Dict Attribute
The input images and masks are passed in via a dictionary attribute in the appropriate format, created using other TOP nodes.
Custom File Map
The source images and masks for each object class are directly specified using a multiparm that maps images and class IDs.
Multiple Image Directories
Input images and masks are stored into separate directories with matching file names, with one mask directly per classs.
Multi-AOV Images
COP Nodes
Input images and masks are produced by multiple COP nodes.
Dict Attrib Name
When Input Format is set to Dict Attribute, determines the name of the dictonary attribute to use. The dictionary should match the format described above.
Input Files
When Input Format is set to Custom File Map, specifies the input file paths and their associated mask files.
Source File
A custom source file in the Custom File Map input. Each source file can map to multiple mask files for different classes.
Masks
The list of custom masks for the Source File. Each mask entry is for a specific object class.
Object Class
The object class ID for the mask.
Mask File
The mask file path.
Keypoints
The list of custom keypoints for the mask file.
Keypoint Class
The keypoint class ID for the keypoint entry.
Keypoint Position
The x and y position of the keypoint in the mask image.
Source Directory
When Input Format is set to either Multiple Image Directories or Multi-AOV Images, determines the directory that contains the source image files.
Masks
When Input Format is set to __Multiple Image Directories_, determines the mask file paths.
Mask Directory
The directory that contains the masks. Each mask should match the file name from the Source Directory.
Object Class
The object class ID for the mask.
Color AOV
When Input Format is Multi-AOV Images, determines the name of the color plane AOV in the source images.
Mask AOVs
When __Input Format_ is Multi-AOV Images, determines the names of the mask plane(s) in the source images.
Mask AOV
The name of the AOV for the current mask definition.
Object Class
The object class ID for the mask.
Cook Type
Determines how work items in the node should cook, e.g. if they should run in process, out of process, or using services.
Evaluate Using
When Input Format is set to COP Nodes, determines how the evaluation frame range should be specified when cooking the COPs.
Frame Range
The source and mask COP nodes are evaluated over the frame range specified in the Frame Range parameter.
Custom Range
The source and mask COP nodes are evaluated at the specific frames listed in the Custom Range parameter.
Source COP
The path to the COP that defines the source images.
Source Range
The frame range to use when rendering the COP nodes to disk.
Source Range
A Value Pattern that generates a custom list of frames to use when evaluating source COPs.
Mask COPs
The COPs that define masks for each object class.
Mask COP
The path to the COP that’s rendered to produce a mask for the specified object class.
Mask Range
The frame range to use when rendering the mask COP node to disk.
Mask Range
A Value Pattern that generates a custom list of frames to use when evaluating mask COPs.
Object Class
The object class ID for the mask.
Major Version;
Minor Version
The minor version for the dataset, affects the output directory name and
the version number in the generated .json file.
Output Name
The name for dataset directories produced by this node.
Write Dataset Metadata
Determines whether metadata fields like the version number, data, or
description should be written to the dataset’s .json file.
Version
Turns on writing the version number to the dataset file.
Creation Date
Turns on writing the creation date/time to the dataset file.
Description
Turns on writing a custom description string to the dataset file.
Environment Path
The path to the virtual environment on disk.
Python Bin
Determines which Python executable to use when creating the virtual
environment and installing packages. Also the version of Python
that will be associated with the venv.
Custom Python Bin
When Python Bin is set to Custom, determines the path
to the Python interpreter to use when creating the virtual environment.
Use Symlinks
Determines whether the virtual environment should be created with symlinks to the source Python binaries. This is recommended on Linux and Mac when creating a virtual environment using Houdini’s Python interpreter.
Never
Always fully copy the Python interpreter to the virtual environment, instead of creating a symlink.
Linux and Mac Only
Create symlinks on Linux and Mac, but copy the interpreter executable on Windows.
Always
Create symlinks on all platforms
Use Pip Cache
When on, pip will attempt to use cached packages on the local system instead of downloading them every time. This can improve the installation times when repeatedly installing the same Python package in different virtual environments.
TOP Scheduler Override
Overrides the TOP scheduler for this node.
Schedule When
When on, specifies an expression that determines which work items from the node should be scheduled. If the expression returns zero for a given work item, that work item will immediately be marked as cooked instead of being queued with a scheduler. If the expression returns a non-zero value, the work item is scheduled normally.
Work Item Label
Determines how the node should label its work items. This parameter allows you to assign non-unique label strings to your work items which are then used to identify the work items in the attribute panel, task bar, and scheduler job names.
Use Default Label
The work items in this node will use the default label from the TOP network, or have no label if the default is unset.
Inherit From Upstream Item
The work items inherit their labels from their parent work items.
Custom Expression
The work item label is set to the Label Expression custom expression which is evaluated for each item.
Node Defines Label
The work item label is defined in the node’s internal logic.
Label Expression
When on, specifies a custom label for work items created by this node. The parameter can be an expression that includes references to work item attributes or built-in properties. For example, $OS: @pdg_frame will set the label of each work item based on its frame value.
Work Item Priority
Determines how the current scheduler prioritizes the work items in this node.
Inherit From Upstream Item
The work items inherit their priority from their parent items. If a work item has no parent, its priority is set to 0.
Custom Expression
The work item priority is set to the value of Priority Expression.
Node Defines Priority
The work item priority is set based on the node’s own internal priority calculations.
This option is only available on the 
Python Processor TOP, 
ROP Fetch TOP, and ROP Output TOP nodes. These nodes define their own prioritization schemes that are implemented in their node logic.
Priority Expression
Specifies an expression for work item priority. The expression is evaluated for each work item in the node.
This parameter is only available when Work Item Priority is set to Custom Expression.
| See also |
ML Train Style Transfer
