This is a general-purpose ML training node to apply to regression problems. In the context of deep learning, regression means training a feedforward neural network (a model) so it can closely approximate a continuous function from a fixed number of input variables to a fixed number of output variables. In Houdini, this function may be provided in the form of a procedural network. See Machine Learning documentation for more general information.

For example, regression ML may learn how to realistically deform a character based on its rig pose to improve over linear blend skinning. In this case, you use ML to learn the function that maps a rig pose to a skin deformation. See the ML Train Deformer recipe. Another example is to learn how to upres a pyro simulation. See the ML Train Volume Upres recipe.

ML Train Regression trains a model given a set of data set consisting of labeled examples, see ML Example. Each labeled example is a pair consisting of an input component and a corresponding target component. For regression, both the input component and the target component consists of continuous variables. This may include point coordinates, colors, or PCA components (among other types of values). The term regression is used because each target component consists of continuous variables, as opposed to discrete ones used in classification. Labeled examples are the basis for training. In the specific case of the ML Deformer, each input component is an (serialized) pose and the corresponding target component is the (PCA-compressed) skin deformation obtained by a flesh simulation. In the case of the ML Volume Upres, both inputs and outputs are 3D volume tiles (voxel grids).

ML Train Regression trains a model that predicts a target component given some user-specified input component. The goal is to have a well-fitted model that generalizes well, it should produce useful predictions for unseen inputs. This means inputs that were not included in the data set for training. ML Train Regression provides several methods that can help ensure that a trained model performs well on unseen inputs. These are called regularization methods.

The trained model produced by ML Train Regression can export to the ONNX format. It can also be saved as a pytorch model. However, Houdini has no built in nodes that can use this model. The easiest way to bring a model produced by ML Train Regression into Houdini is the ML Regression Inference tool.

ML Train Regression creates a feedforward neural network. The number of input units of this network must correspond to the number of variables in each input component of the data set. The number of output units of the network must correspond to the number of variables in each target component of the data set. When you use a standard kind of model, ML Train Regression automatically figures out the number of inputs and outputs of the network using the dimensions stored in the provided raw data set.

In between the input layer and the output layer, there may be several hidden layers. The number of hidden layers and their width can be controlled by the user. The behavior of the network is controlled by a set of parameters. For linear layers, these parameters may include weights (scaling factors) and biases (additive constants). The training aims to optimize these parameters for the regression task.

ML Train Regression performs multiple passes through the training portion of the data set. Each such pass is called an epoch.

The training process may be repeated multiple times, each time with different settings on the training node. Such training settings are commonly referred to as hyperparameters. Common examples of hyperparameters are settings such as the learning rate, the number of hidden layers of the network, and the width of the network.

The Wedge TOP allows the training process to be repeated with varying hyperparameters. For each setting in a wedge, the resulting model can be saved to a separate file by adding a TOP attribute name at the end of the Model Base Name.

ML Train Regression internally splits the data set provided by the user into two parts: a training portion and a validation portion. The relative sizes these portions can be controlled by the user. The training portion is used to optimize the network’s parameters during each training epoch. The validation portion is used to verify the network’s performance on data that is not used for the training.

Underfitting occurs when a model is not sufficiently complex to fit the training data well. When this occurs, the model cannot be expected to produce accurate results on test data either. In that case, a more complex network with more parameters (higher capacity) may be needed. This may be achieved, for example, by increasing the number of hidden layers or increasing the width of each hidden layer.

Overfitting occurs when a model produces outputs that are accurate for inputs in the training data but inaccurate for unseen inputs. To help reduce overfitting, the ML Train Regression TOP supports two simple regularization strategies: early stopping and weight decay. Early stopping checks whether the validation loss (the loss on the validation set) stops decreasing. If this is the case, then the training terminates. Weight decay adds to the loss function a term that penalizes large weights.

Instead of training a model all at once, a model can be partially trained bit by bit, over multiple passes, where each training pass performs only a limited number of epochs. This allows the TOP network to perform other tasks in between training passes. For example, fresh training data can be generated before each partial training pass, which may be applicable when the data set is generated within Houdini through a procedural network. Training on a stream of fresh training data compared to revisiting data points in a fixed-size data set may produce a model that generalizes better to unseen data. Another advantage is the amount of training data does not have to be decided upfront.

Partial training can also be a useful way of limiting the amount of work/time performed by a single cook of ML Train Regression, while allowing the training to be resumed from where it stopped at a later time. This may be relevant when training is performed on a farm, for example.

Note that starting in Houdini 21, partial training is no longer required in order to periodically save and/or export partially trained models. There are now separate options that allow you do do this without partial training.

ML Train Regression can be set up for partial training:

1. Turn on Enable Partial Training.

2. Place ML Train Regression inside a feedback block, see Feedback Begin.

3. Set Create Iterations From on Feedback Begin to Custom Count.

4. Set Iterations to the total number of partial training passes you want to perform.

5. Set Max Epochs per Cook on ML Train Regression to control the maximum number of epochs performed for each training pass. For example, you could set Max Epochs per Cook such that each partial training pass takes some approximate amount of time. The appropriate value for Max Epochs per Cook really depends on the size of the data set.

   • (Optional) To output and retain partially trained ONNX models, you can specify on the ML Train Regression node, in the Files tab, a Model Base Name that includes an attribute such as @loopiter.

   • (Optional) To generate fresh training data before each partial training pass, you can put a ROP Fetch that triggers the generation of a new data set in front of ML Train Regression inside the feedback loop. You should include the attribute @loopiter in the file name of the data set.

ML Train Regression accesses a variety of files. Each file name can be optionally modified by inserting one or more TOP attributes into a base name of a file. For example, @wedgeindex or @loopiter.

This enables various types of TOP ML training setups such as:

• Training for varying hyperparameters with the use of a Wedge TOP,

• Generating additional training data on demand in a TOP feedback loop.

Some of the files ML Train Regression accesses are read-only, both read from and written to, and only written to (or appended to).

In the read-only category, there are (custom network) hyperparameters and the data set. The Hyperparameters tab allows you to provide a dictionary that can be accessed from custom network snippets. For example, controlling the neural network structure. The Input tab allows you to specify the source of training data (and validation data if early stopping is on).

In the write-only category, there are trained (or partially trained) models, either saved or exported ONNX format, with file names determined by Models Folder and Model Base Name. These can be found under the Output tab. Also, diagnostic and progress information are controlled by the Log tab. If a log file exists, log information is appended to that existing log file. This logging allows the creation of a single log file if a single training is broken up into parts by invoking ML Train Regression multiple times. For example, in a TOP feedback loop.

In the read-write category, there are parameters under the State tab that allow you to specify the current state of training. These are used when partial training is on. The training state is read from at the start of a single invocation of ML Train Regression and written to at the end. The training state allows ML Train Regression to resume training where it left off in a previous invocation. This training state is only valid when the state name is the same as the previous invocation. The training state can act like a checkpoint, allowing an imcomplete training to resume.

The controls under the Save Frequency and Export Frequency tabs allow you to control how often models are saved and/or exported. Saving and exporting can happen multiple times during one cook of ML Train Regression, even when partial training is turned off.

ML Train Regression trains from scratch only if its expected output files are not found on disk. This behavior enables partial training setups. The expected output files include the exported ONNX model, training state, and training log. If these files exist, ML Train Regression will not re-train from scratch.

To make ML Train Regression re-train, remove the output files from disk. You can remove files through the TOPs UI:

• ML Train Regression node and select Delete this Node’s Output Files from Disk.

• For a single work item, the work item and select Delete File Outputs from Disk.

ML Train Regression allows you to easily use a standard kind of neural network, such as an MLP, without any coding. However, you can also program your own custom network structure if you want to. This requires writing one or more snippets consisting of a few lines of python code, making use of the PyTorch library.

An MLP consists of several fully connected layers. In theory, you could always use an MLP for your training. However, this is not always practical. For larger input and output sizes, your MLP model may end up having a very large capacity. Such a large model may require a very large number of training examples to achieve good generalization. Also, large models may exceed the amount of memory available on your machine.

When a standard model or a standard loss does not suffice, you can create your own model or loss function. This applies when Network Composition is set to Partitioned. However, to allow even even greater flexibility, you can switch Network Composition to Unified. In this case, you have full freedom to create a custom unified network that takes an input component and a target component and outputs a scalar value.

You can create your own model, loss function, or unified network using PyTorch network modules by writing python snippets. Each such snippet parameter has a default, which can be useful as a reference.

Each python snippet may access a kwargs dictionary and must set dictionary called result. You can write snippets as if the torch library is already included. See the pytorch documentation for more information about creating neural networks using PyTorch modules.

The kwargs dictionary stores the following values: a tuple of input shapes input_shapes, a tuple of target shapes target_shapes, and a dictionary of (custom network) hyperparameters hyperparameters. In a python snippet, these values could be extracted and put in separate variables like:

input_shapes = kwargs["input_shapes"]
target_shapes = kwargs["target_shapes"]
hyperparameters = kwargs["hyperparameters"]

Both input_shapes and target_shapes are determined by the data set. In Houdini 21, both the input components and target components are always flattened and concatenated, so that input_shapes and target_shapes contain only a single shape and this single shape has only a single dimension. In this flattened, concatenated representation, the total number of (scalar) elements of an input component is always input_shapes[0][0]. Similarly, in the flattened, concatenated representation, the total number of (scalar) elements of a target component is always target_shapes[0][0].

The dictionary hyperparameters is loaded from a JSON file. You can specify this file location using the Hyperparameter Folder and the Hyperparameter Base Name parameters. The contents of this hyperparameter dictionary can control the structure of the custom network.

A snippet that defines a custom network must always set the result dictionary. This dictionary must assign values to both module_class and module_kwargs. The value associated with module_class must be a python class object, for a class that is a subclass of torch.nn.Module. For example, in a python snippet, setting the result could look like

class MyModule(torch.nn.Module):
...
my_module_kwargs = ...
...
result = {"module_class": MyModule, "module_kwargs": my_module_kwargs}

Here my_model_kwargs should be valid keyword arguments for the construction of MyModel.

ML Train Regression and the ML Example family of supervised ML nodes in SOP and ROP provide an easy starting point for experimentation with supervised ML in Houdini. The applicability of ML Train Regression can be extended by making use of the script options that allow you to optionally create your own model networks and loss functions. The Unified option on Network Composition allows you to stretch the flexibility even further, allowing ML Train Regression to be applied for many more ML use cases.

Even with custokmizations, you may find that ML Train Regression is not general enough for your use case. In that case, you can extract and copy the underlying scripts of this node, located in $HHP/hutil/ml/regression and use these as a starting point for creating a custom training node. You may still be able to use many of the other tools of the supervised ML toolkit that exist in SOP and ROP to prepare your data set such as the ML Example operator family.