| On this page | |
| Since | 22.0 |
Overview ¶
This node can be used with a dataset produced by the 
ML Preprocess Computer Vision TOP
to train a computer vision model for doing object detection and segmentation. The node can be configured to produe an .onnx
file to use with the 
ONXX Interface SOP .
This node makes use of a third party training system called Detectron2, and downloads pre-trained weights. It’s possible to change the Pre-Trained Weights menu to use a custom weights file or no weights at all.
Parameters ¶
Data ¶
Base Directory
The base path that contains all of the inputs, outputs, and virtual environment used during the training process. Outputs are produced in specific subdirectories depending on the type of output file.
Base Name
The name of the subdirectory that contains all outputs produced by this node.
Run Name
The name of the training run. If this node is being wedged out to perform different training runs with different settings, this parameter can create a per-wedge output directory.
Output Directory
The path to the directory that contains all ML models, tests, and plots for the current training run. By default, this parameter is derived from the Base Directory, Base Name, and Run Name parameters, but can be modified to a custom directory path.
Datasets to Train
Determines the path to the dataset(s) used for training. Each dataset entry
should be a directory produced by the ML Preprocess Computer Vision TOP .
Total Batch Size
The batch size when loading training images.
Number of Workers
The number of worker processes to use when loading dataset images. Set
this to 0 to allow the dataset loader to automatically choose the number of
workers at runtime.
Detection Type
Determines the detection type used for training.
Objects: Only detects the bounding boxes around object with no keypoints or masks.
Masks: Detects objects and their segmentation masks.
Keypoints: Detects objects and key tracking points in their bounding boxes.
Mask Format
When Detect Mask is on, specifies the format for the masks in the training dataset.
Image Format
The format of the images in the training datset.
Sample Style
Determines how training daatset images are sampled by the data loader.
Choice
Images are sampled from a specific list of possible sizes, determined by the Min Image Size parameter.
Range
Images are sample over a range of possible sizes, determined by the Min Image Size parameter.
Image Sizes
The space-separated list of possible image sizes to choose when sampling the dataset, if Sample Style is set to Choice.
Image Size Range
The range of image sizes to choose from when sampling the datset, if Sample Style is set to Range.
Max Size
The maximum sampling size.
Crop Inputs
When on, input images are randomly cropped in the X
and Y direction by the ratios specified in this parameter. For exmaple, a value of 0.25 means image dimension should be randomly cropped to any
size between 25% and 100% of the original size.
Horizontal Flip
When on, dataset images are randomly flipped in the horizontal axis using the specified probability.
Vertical Flip
When on, dataset images are randomly flipped in the vertical axis using the specified probability.
Network ¶
Pre-Trained Weights
Determines which pre-trained weights to use for the training process. Built-in weights must be downloaded to SHFS using the Download Weights button before they can be used for training.
Download Weights
Downloads all of the available pre-trained weight files to the SHFS path on the local machine.
Custom Weights
When Pre-Trained Weights is set to Custom, specifies the file path to a custom pre-trained weights file.
Training ¶
Config Dir
The directory where training config files and logs are written.
Max Epochs
The maximum number of training epochs.
Log Interval
The number of training epochs between writes to the main training log.
Use AMP
Turns on Automatic Mixed Precision while training, which switches some parts of the training process to 16 bit floating point precision for reduced memory usage.
Optimizer
Determines which optimization algorithm to use during the training process.
Adam
Uses the Adam algorithm, see pytorch documentation for more details.
AdamW
Uses the AdamW algorithm, see pytorch documentation for more details.
RMSprop
SGD
Uses the Stochastic Gradient Descent algorithm, see pytorch documentation for more details.
Learning Rate
The rate the model updates itself during the training process. A higher value can cause the model to train faster, but may also result in instability that causes the model a failure to converge.
Alpha
When Optimizer is set to RMSprop, determines the smoothing constant
used to configure the optimizer algorithm.
Beta
When Optimizer is set to Adam or AdamW, determines the beta values
used by the optimization algorithm. Higher beta values can result in faster
model convergence, but may also decrease stability.
Epsilon
When Optimizer is set to Adam, AdamW, or RMSProp, determines the
epsilon value passed to the algorithm to improve numerical stability.
Momentum
When Algorithm is set to RMSprop or SGD, determines the momentum
value. This value is optional but helps the training process converge faster
by allowing past gradient calculations to influence the optimization
process. The momentum value determines how heavily past results influence
the optimization process.
Weight Decay
The weight decay value passed to the optimizer algorithm.
AMSGrad
When Optimizer is set to Adam or AdamW, determines whether the
AMSGrad variant of the algorithm should be used.
Centered
When Optimizer is set to RMSProp, this parameter determines
whether gradients are normalized during the optimization process.
Nesterov
When Optimizer is seto SGD, this parameter determines whether
Nesterov momentum should be used during the optimization process. This
parameter only has an affect if the Momentum value is non-zero.
Learning Rate Multiplier
A scale factor applied to the learning rate. This should match the multiplier from the pre-trained weights selection.
Testing ¶
Test Model
Whether or not the model should validated and tested during the training process. Validation consists of evaluating the model and recording various numeric metrics, and testing involves generating actual image output from input images.
Validation/Test Split
Determines the ratio of the input dataset that should be used for model
validation and testing, respectively. For example, a value of 0.2 and 0.1
means that 20% of the input dataset is used for vaidation, 10% is used for
testing, and remaining 70% is used for training.
Number of Workers
The number of worker processes to use when loading testing and validation sample images.
Batch Size
The batch size to use when validating and testing.
Validation Output Dir
The output directory for .csv files produced during the validation step. These files will be picked up by the training monitor panel if they exist.
Validation Interval
The rate at which the model is validated, specified in training epochs.
Test Output Dir
The output directory that contains the image results from testing the model. The training monitor panel will pick up these files automatically if they exist.
Test Interval
The rate at which the model is tested, specified in training epochs.
Test Count
The number of test runs to perform during each testing step.
Test Threshold
The confidence threshold to use when processing test results. Detections that are below the specified score will be excluded from the testing output images.
Add Test Results as Outputs
Whether or not test output images should be added as work item outputs.
Checkpointing ¶
Model Dir
When on, the directory that contains PyTorch model checkpoint files.
Model Save Interval
The rate at which checkpoint files should be written to disk, specified in terms of training epochs.
Max Model Files
The maximuim number of checkpoint files to keep on disk. Files are deleted automatically to reduce the amount of disk space spent on checkpoints.
Resume Training from Model Checkpoints
Whether or not the training process should resume from checkpoint files if they exist.
ONNX Path
The format string for the .onnx output file path, which can include the
{epoch} token to create per-epoch model files.
ONNX Export Interval
The rate at which ONNX model files are written out, specified in terms of of training epochs.
Max ONNX Files
When on, limits the number of .onnx files on disk to reduce disk space
usage.
ONNX Opset Version
The ONNX opset version to use when exporting model files.
Input Tensor Type
The type of input tensor shape, i.e. either a fixed pre-determined size or dynamic.
Fixed Size
When Input Tensor Type is set to Fixed, determines the input size supported by the exported model.
Add ONNX Checkpoints as Outputs
Determines whether or not .onnx files should be added as work item outputs.
Execution ¶
Train as Single Work Item
By default, the training tasks are group into a batch with the same size as the Max Epochs. Turn this on to condense the training process into a single task instead.
Override Device
When on, override the device that PyTorch uses to train the model.
Set CPU Threads
When the device type is set to cpu, sets the maximum number of CPU threads used during training. It has no effect on the training process when using a GPU device.
Environment Path
The path to the virtual environment on disk.
Python Bin
Determines which Python executable to use when creating the virtual
environment and installing packages. Also the version of Python
that will be associated with the venv.
Custom Python Bin
When Python Bin is set to Custom, determines the path
to the Python interpreter to use when creating the virtual environment.
Use Symlinks
Determines whether the virtual environment should be created with symlinks to the source Python binaries. This is recommended on Linux and Mac when creating a virtual environment using Houdini’s Python interpreter.
Never
Always fully copy the Python interpreter to the virtual environment, instead of creating a symlink.
Linux and Mac Only
Create symlinks on Linux and Mac, but copy the interpreter executable on Windows.
Always
Create symlinks on all platforms
Use Pip Cache
When on, pip will attempt to use cached packages on the local system instead of downloading them every time. This can improve the installation times when repeatedly installing the same Python package in different virtual environments.
TOP Scheduler Override
Overrides the TOP scheduler for this node.
Schedule When
When on, specifies an expression that determines which work items from the node should be scheduled. If the expression returns zero for a given work item, that work item will immediately be marked as cooked instead of being queued with a scheduler. If the expression returns a non-zero value, the work item is scheduled normally.
Work Item Label
Determines how the node should label its work items. This parameter allows you to assign non-unique label strings to your work items which are then used to identify the work items in the attribute panel, task bar, and scheduler job names.
Use Default Label
The work items in this node will use the default label from the TOP network, or have no label if the default is unset.
Inherit From Upstream Item
The work items inherit their labels from their parent work items.
Custom Expression
The work item label is set to the Label Expression custom expression which is evaluated for each item.
Node Defines Label
The work item label is defined in the node’s internal logic.
Label Expression
When on, specifies a custom label for work items created by this node. The parameter can be an expression that includes references to work item attributes or built-in properties. For example, $OS: @pdg_frame will set the label of each work item based on its frame value.
Work Item Priority
Determines how the current scheduler prioritizes the work items in this node.
Inherit From Upstream Item
The work items inherit their priority from their parent items. If a work item has no parent, its priority is set to 0.
Custom Expression
The work item priority is set to the value of Priority Expression.
Node Defines Priority
The work item priority is set based on the node’s own internal priority calculations.
This option is only available on the 
Python Processor TOP, 
ROP Fetch TOP, and ROP Output TOP nodes. These nodes define their own prioritization schemes that are implemented in their node logic.
Priority Expression
Specifies an expression for work item priority. The expression is evaluated for each work item in the node.
This parameter is only available when Work Item Priority is set to Custom Expression.
| See also |
ML Train Regression
ML Train Style Transfer
