Houdini allows you to train using data sets from external sources such as mocap data, images, and videos. However, there are plenty of useful applications of ML where the data can be generated entirely synthetically using a procedural network. Some examples are the ML Train Deformer recipe and ML Train Volume Upres recipe. Both these training recipes send sample inputs into a simulation pipeline to produce a target for the ML.

When a procedural network in Houdini can already generate ground truth for a dataset, training a model on that dataset may still be useful. Although the procedural network could be applied directly to any input, a trained model can provide advantages such as function approximation and inversion.

Function approximation, such as regression, can be useful when the ground truth is generated by a slow process, such as a simulation. A model trained on that ground truth may closely approximate the desired results while running orders of magnitude faster than the original procedural network. The ML Train Deformer recipe uses this approach.

Inversion can be achieved by switching the inputs and ground truth generated by the procedural network before adding them to a dataset. The resulting model may provide an approximate inverse, allowing it to predict which input leads to a given output when run through the procedural network. The ML Sketching content library example uses this principle. In addition to function approximation and inversion, synthetic data generation in Houdini has many other potential applications.

Source data, whether obtained from an external source or generated synthetically in Houdini, requires preprocessing to support successful ML training. Preprocessing may include feature extraction, normalization, and augmentation.

Raw data sources are unstructured and may need to be converted into tabular data before they can be used for training. This process is called feature extraction. For example, each data point in a data set may need to be converted into a flat array of numbers, with each array having the same fixed size.

A concrete example is the ML Train Deformer recipe, introduced in Houdini 21. This recipe trains an ML model that maps each skeleton pose (input) to a deformed skin (target). Each data point in the data set consists of a skeleton pose as the input and a corresponding deformed skin as the target. However, the skeleton pose and deformed skin cannot be included in the data set directly because they are not yet in a tabular form suitable for training.

During preprocessing, each input skeleton pose is converted into an array of numbers by extracting the components of each joint transform. For the deformed skin,you can technically train on all point position coordinates of the skin mesh. However, this would make the model too expensive. Instead, the skin’s delta relative to linear blend skinning (LBS) is computed and encoded in a lossy way using principal component analysis (PCA). For this use case, the result is a relatively small array, typically with a size of 128 or 256.

Another type of preprocessing is normalization. Because of how model weights are typically initialized, model training works better on normalized data. For example, training a convolutional network on images may work better if the pixel values are normalized to mostly lie between -1 and 1.

For certain ML applications, dataset augmentation is useful. This means creating additional data points by transforming existing ones and inserting them into the dataset. For example, a computer vision data set may include many rotated copies of images to help ensure that the model behaves well on rotated inputs.

After feature extraction in Houdini, each data point in a data set may be represented as a piece of geometry, with tabular data stored in point attributes, detail attributes, or volumes of any dimension. Before a model can be trained on this data set, each data point must be serialized into one or more tensors the training stage can interpret. In ML, a tensor is a multidimensional array of numbers.

The conversion between attributes and volumes on one side and tensors on the other, may seem trivial. Attributes and volumes can be interpreted directly as tensors. However, these tensor interpretations may not match the tensor representations required by an ML model. Certain choices may need to be made when applying these conversions.

When working with volumes in Houdini, there are subtleties around the ordering of the X, Y, and Z axes and the handling of color components. For some models, the axis order must be changed before a volume is interpreted as a tensor. Other models require an image that stores an RGB triplet at each pixel to be reinterpreted as an array of three monochromatic images. In both cases, the raw volume data may need to be shuffled before it can be converted into a tensor. Additionally, one of the X, Y, or Z axes of a volume may need to be flipped. Because of these subtleties, you should be careful when passing images and 3D volumes into a model or when converting a model’s output tensors back into images and 3D volumes.

The training stage ingests the data set and produces a trained model. During inference, the trained model can run on any user provided inputs. A common goal of training is to obtain a model that produces useful outputs for unseen test inputs, which were not part of the trainig and validation data sets.

Training usually involves running a script with an underlying deep learning framework. Houdini uses PyTorch for its training solutions. The training script may read a data set that was written to disk as raw data at an earlier stage. Then, use PyTorch’s backpropagation mechanism with an approach like stochastic gradient descent (SGD) to find parameters (weights and biases) for a trained model.

However, not all training in Houdini requires a deep learning framework. In some cases, classic non-deep learning methods can train faster and more effectively. Examples include linear models, which may use nonlinear basis functions and kernel ridge regression models. These models can be quicker to train and just as effective as neural networks for low-dimensional problems or when little training data is available.

Hyperparameters are parameters that are chosen before training rather than learned during training. In Houdini ML setups, common hyperparameters include the number of hidden layers in an MLP network, the width of each hidden layer, the initial learning rate used during training, and regularization constants. You may not know the best values for these hyperparameters upfront. Houdini can help automate the search for optimal hyperparameters, for example by using TOPs.

After training, model evaluation helps determine whether training was successful. In ML, models are evaluated on unseen test inputs rather than the inputs in the training and validation datasets. This evaluation can sometimes be performed manually by inspecting the results. For discriminative machine learning tasks, such as function approximation or regression, evaluation can be automated by computing a loss value on a test dataset held separate from the training and validation datasets. A procedural network in Houdini can be set up for this evaluation.

Inference involves running a trained model on inputs to produce outputs, also known as predictions. In Houdini, ONNX is the main file format for storing ML models. Houdini ML nodes allow trained ONNX models to be used in various contexts such as SOPs and COPs. Houdini supports ONNX models that take one or more tensors as inputs and produce one or more tensors as outputs.

However, Houdini does not directly support tensors. Contexts such as SOPs and COPs work with Houdini specific data structures, such as point attributes and volumes. As a result, using trained ML models in Houdini requires conversions to and from tensors. Each model input must be converted into one or more tensors, and the output tensors produced by the model must be converted back into a form Houdini understands.

These conversions may repeat some steps performed during dataset preprocessing. For example, a regression model may be trained on labeled examples, where each example consists of an input and a target. The same conversion applied to the input components of the training data must also be applied to the model input. Similarly, the conversion used to represent the target as tensors must be inverted during inference to turn the model’s output tensors into something useful.

For example, to use a model trained with the ML Train Deformer recipe, a skeleton pose has to be converted into an input tensor. This uses the same procedure applied to the input components during preprocessing: extracting the components of the skeleton’s joint transforms. The model then produces an output tensor.

To convert this output back into a deformed skin, the target preprocessing step is reversed. The model outputs a tensor containing a small array of weights. These weights are multiplied by the previously computed PCA components to produce a delta, which is applied to an LBS-deformed skin. This results in the final deformed skin.

Houdini provides tools that support many of the workflows discussed earlier without requiring you to manage the underlying technical details. For some existing models, such as SAM2 and MoGe-2, Houdini provides assets that handle the extra steps required for inference. These assets allow existing models to be used as nodes that work seamlessly with other nodes. See Neural Nodes.

Houdini also provides convenient tools for preprocessing datasets, training models, and running inference without requiring you to work directly with tensors. See Building Blocks. For specific applications, Houdini includes tailored training nodes as part of Trainable ML Solutions, which reduce the need to understand the technical details of ML frameworks.

For workflows that require building a custom ML training solution outside the Building Blocks approach, or integrating a third-party model, use the Basic Utilities. These utilities are recommended if you have technical knowledge of deep learning frameworks such as PyTorch.