Houdini 22.0 Machine Learning Trainable ML Solutions

ML Train GSplats from Karma

On this page

Stages
Structure
Important nodes
Using this tool
Expanding on this tool

There are several machine learning recipes available through the tab menu. These are similar to shelf tools that put down networks of nodes. Some are meant for learning purposes or plug-and-play workflows, while others are fully fledged tools, as in this case.

The ML Train GSplats from Karma recipe automates the data generation and training process to create synthetic Gaussian Splats (“GSplats”).

The recipe automatically renders your scene from a surrounding camera array and pre-processes the results into a posed image data set. It then trains a Gaussian Splats model on that data set, producing .ply and .usd files you can bring back into Houdini.

Stages ¶

The generation of GSplats consists of two main stages:

data generation
training

Each of these stages is comprised of smaller steps, which are outlined below.

Data generation ¶

To train GSplats, it’s necessary to know how the scene looks from many different angles. The data generation stage produces this set of posed images and the data that accompanies it.

Camera setup ¶

A camera array is set up around the scene to determine which perspectives are rendered. The number of cameras and the shape used to cover the scene, are the most important choices here, because together they determine how thoroughly the scene is observed.

Camera array used to render the data set.

Rendering ¶

An automated TOP network renders the scene from every camera perspective. Each perspective is written out as an EXR image with its associated AOVs and camera metadata.

Pre-processing ¶

The rendered EXRs are passed to a ML Preprocess GSplats TOP, which converts them into a COLMAP-like data set. This data set contains the images, the cameras, and an initial point cloud that the training stage picks up.

Training ¶

The training stage optimizes the initial point positions based on the images and their corresponding camera positions, gradually refining the point cloud into a set of GSplats that reproduces the rendered views.

Results ¶

By default, the recipe writes a .ply and a .usd file to the output directory. You can sublayer the resulting GSplats on the stage with the .usd file. Alternatively, you can bring the .ply file into SOPs where you can convert it to Houdini attributes with the Bake GSplats SOP.

Final GSplats brought back onto the stage as a USD ParticleField3DGaussianSplat primitive.

Structure ¶

ML Train GSplats from Karma LOPs recipe.

IN_stage and layerbreak

This sublayer node lets you conenct your scene to the second input or point to a USD file on disk you want to turn into GSplats. The layerbreak node makes sure that all the upstream USD layers end up in their respective save paths and not in the gsplats.usd file created during rendering/data generation.

Setup_Camera_Array

This subnet lets you create a camera array around your input scene. You can select from a number of distribution algorithms and shapes to cover your scene.

karmarendersettings1 & karmastandardrendervars1

A default Karma render setup linked to the resolution and other render settings used for data generation. It already sets up the correct N and albedo AOVs and makes sure that the renders are denoised and don’t have any motion blur or depth of field.

ML_Train_GSplats

This subnetwork is a high-level wrapper around a TOPs data generation and training setup. It lets you configure how the data set should be pre-processed and provides several hyperparameters. Hyperparameters directly influence the training and subsequently the final result.

Important nodes ¶

ML Preprocess GSplats TOP

Pre-process rendered EXR images, cameras, and a point cloud into a COLMAP-like data set for GSplats training.

ML Train GSplats TOP

Train a 3D Gaussian Splats model from a set of posed images.

Using this tool ¶

To...	Do this
Train your own GSplats	Wire your scene into the right input on the `IN_stage` Sublayer LOP. Configure your camera array to cover your scene. The most important settings are the number of cameras and the shape used to cover the object. For environments, the Dome preset should deliver good result. For single assets like the Test Geometry: Rubbertoy, you may want to consider switching to Sphere. Click the Cook Output Node button to initiate the data generation and training process. Depending on your scene, the number of cameras, and your hardware, this can take a while. Once you see the green work item indicator dot show up on the subnet, you can head to the Monitoring tab. There, click the Open Viewer button to open a browser-based live training viewer.
Monitor the training progress	Repeat the steps from `Train your own GSplats` above to initiate the training. Open the PDG ML Training Monitor panel located in the TOPs menu. Select the `ML Train GSplats` subnetwork, which will link the monitor to the training TOP node. Monitor the training progress through the loss plot. Cycle to other plots to visualize individual loss components or the number of Gaussians over time. Switch to the test results page to inspect test renders and compare the ground truth image with the current state of the GSplats. You can see the current problem areas in the loss heat map visualization.
Change the custom features (AOVs)	Set up AOVs like you always do in Karma and specify them on the `ML Train GSplats` subnetwork under GSplats ▸ Features. Click the Cook Output Node button to initiate the data generation and training process.

Expanding on this tool ¶

To... Do this

To...	Do this
Replace the camera array	Change the distribution logic in the subnet itself or come up with an entirely new workflow instead. Some scenarios may require you to set up your own cameras to cover your scene sufficiently. You are free to do this in any way that comes to mind. Point the `ML Train GSplats` subnet’s Camera Array Prim parameter on the Data Generation` tab to any parent primitive that holds a number of USD cameras.
Replace everything	Come up with your own data generation logic entirely for extreme customization. Use the ML Preprocess GSplats TOP node to turn a set of EXR images into a COLMAP data set for training. The only requirement for those EXRs is that they store the camera metadata and contain a z-depth AOV. Pass the created data set to the ML Train GSplats node and cook the TOP network to initiate the training.

Replace the camera array

Change the distribution logic in the subnet itself or come up with an entirely new workflow instead. Some scenarios may require you to set up your own cameras to cover your scene sufficiently. You are free to do this in any way that comes to mind.
Point the ML Train GSplats subnet’s Camera Array Prim parameter on the Data Generation` tab to any parent primitive that holds a number of USD cameras.

Replace everything

Come up with your own data generation logic entirely for extreme customization.
Use the ML Preprocess GSplats TOP node to turn a set of EXR images into a COLMAP data set for training. The only requirement for those EXRs is that they store the camera metadata and contain a z-depth AOV.
Pass the created data set to the ML Train GSplats node and cook the TOP network to initiate the training.