Houdini 22.0 Nodes Copernicus nodes

Neural Layer to Mask (SAM2) Copernicus node

Creates a mask based on coordinates set through prompts.

On this page
Since 22.0

This node creates an image segmentation mask based on prompts using Meta’s SAM 2 model, which uses a learned neural network to predict object membership. Add prompts to tell the model which areas you want to mask in the original image. A prompt identifies an object, and the model predicts a mask that covers all pixels belonging to that object. Prompts are positive or negative coordinates, as well as an optional Bounding Box or mask input.

Note

To use this node, click Download Models after a fresh Houdini installation.

Use a mask input to provide additional values, which the model interprets as confidence values to find the related objects to mask. It’s not recommended to use a binary (0 or 1) mask, but to instead use a soft mask.

Add points in the points input, which are interpreted based on their world space location. Use this input for automated or procedural use cases (for example, when you reuse an existing object’s centroid to create a mask). Define the prompt type of a given point with a string point attribute prompt (for example, edit an Attribute String Edit SOP in the geometry). This point attribute can be positive, negative, bboxbl, or bboxtr. If you don’t specify a prompt point attribute, the node assumes positive by default.

Note

When you wire in a points input, the node uses these point values and then appends the Points parameters' values. If you set a Bounding Box on the node, it replaces the bounding box in the points input.

Tip

Reuse the points output by wiring it into another Neural Layer to Mask (SAM2) COP to input correctly labeled points.

You can also wire in an embeddings input with embeddings to reuse. This input stores the encoding of the image. Use this input when you create multiple masks based on the same image, since computing embeddings is the most computationally expensive step.

Note

The node can only reuse embeddings with the same model size used to compute them.

Set points in the Scene Viewer

Instead of setting Points multiparms in the parameter editor, you can click the Handles tool to set points directly in the Scene Viewer.

To...Do this

Add points

  • Hold ⇧ Shift and then to add a positive point.

  • Hold ⇧ Shift and then to add a negative point.

Remove points

Hold ⌃ Ctrl and then or a point to remove it.

Move points

Drag a point to another position.

Add a bounding box

Press F to turn on Bounding Box and then drag the corner handles of the box to set the boundaries.

Press F again to turn off Bounding Box.

Turn mask preview on or off

Press O to turn the mask preview on and off. Use Preview Color to set the color of the preview.

Parameters

Model

The SAM 2 model variant to use for segmentation.

SAM2 Large

The most accurate model, but slower and with a larger memory footprint.

SAM2 Tiny

A fast, but less accurate model.

Custom

Use custom encoder and decoder ONNX files.

Download Models

Downloads the pre-trained SAM 2 model files to the SHFS model location.

Encoder ONNX File

The path to a custom encoder ONNX file.

Decoder ONNX File

The path to a custom decoder ONNX file.

Reload Model

Forces a reload of the .onnx file.

Units

The coordinate system that specifies the point and bounding box positions. This controls the range of the values and whether they're in respect to the input’s display or data window.

Image

Specify coordinates in terms of the display window, with a range of -1 to 1.

Texture

Specify coordinates in terms of the data window, with a range of 0 to 1.

Pixels

Specify coordinates in pixels as integers.

Points

The number of point prompts, which are positive or negative. These tell the model which areas you want to mask in the original image.

Note

A prompt identifies an object, and the model predicts a mask that covers all pixels belonging to that object.

Tip

Click the Handles tool to set points directly in the Scene Viewer. See Set points in the Scene Viewer for more information.

Point Coords #

Coordinates for the prompt point. The Units parameter determines if this is in image space, texture space, or pixels.

Negative

Makes the point indicate an area to exclude from the mask.

Bounding Box

Defines a bounding box prompt.

BBox Lower Left

The lower-left corner coordinates of the bounding box. The Units parameter determines if this is in image space, texture space, or pixels.

BBox Upper Right

The upper-right corner coordinates of the bounding box. The Units parameter determines if this is in image space, texture space, or pixels.

Pre Processing

Resample Behavior

The method used to handle images that don’t match the model’s expected input size of 1024×1024.

Stretch to Fit

Stretch the image to fit the model’s input size (1024 x 1024).

Fit Minimally

Fit the image within the model’s input size (1024 x 1024) with minimal distortion. The node pads the image as needed.

Color Transform

The color transformation to apply to the input image before processing.

OCIO

Use OpenColorIO to transform colors from the Original Space to the target space (To Space). See Color management in Houdini for more information.

None (raw)

Don’t apply any color transformation.

Original Space

The space that the input color is in.

To Space

The output color space to which the input color transforms.

Output

Clean Up

Refines the output masks using Total Variation Denoising (TVD). These are post-processing cleanup operations.

TVD Iterations

The number of denoising rounds that the node performs on the mask. More iterations increase the smoothing of the mask but lose more fine detail. The run time of this operation is proportional to the number of rounds.

TVD Speed

The amount by which to multiply the TVD Iterations value. The result determines the total amount of diffusion to apply in each round of iteration.

Upsampling Filter

The filter to use when upsampling masks to the original image resolution.

See Filters for more information.

Point

Sets the pixel value to the pixel value of the nearest neighbor.

Bilinear

Sets the pixel value through bilinear interpolation .

Box

Sets the pixel value through a box filter, done by sampling all pixels within a box on the original image’s matching pixel.

Bartlett

Sets the pixel value through interpolating according to a triangular window or a Bartlett window , taking each pixel as a sample of this signal.

Catmull-Rom

Sets the pixel value through a bicubic spline known as the Catmull-Rom spline . This is a special case of the Mitchell-Netravali filter .

Mitchell

Sets the pixel value through a bicubic spline known as the Mitchell-Netravali filter.

B-spline

Sets the pixel value through a bicubic spline known as the B-spline . This is a special case of the Mitchell-Netravali filter.

Mask Threshold

The threshold value at which the node converts soft masks to binary masks. The node makes pixels above this value white (1.0) and pixels below this value black (0.0). When off, the node outputs the raw, soft mask values.

Visualization

Preview Color

The color that overlays the mask in the viewer state to preview the final mask output.

Tip

Click the Handles tool and then press O in the Scene Viewer to turn the mask preview on or off.

Advanced

Execution Provider

Determines which ONNX execution provider to use for inference. By default, the node attempts to pick the best available provider and prefers to use GPU acceleration.

Automatic

Chooses the best provider for the current system. This option prioritizes CUDA if installed, DirectML/CoreML as a fallback depending on the platform, and CPU inference if no GPU provider is available.

CPU

Perform inference using on the CPU.

CUDA

Perform inference using CUDA/cuDNN. CUDA and cuDNN must be installed using the packages provided by NVIDIA.

DirectML

Only available on Windows. Performs inference using the Windows Direct Machine Learning library.

CoreML

Only available on macOS. Performs inference using Apple’s Core ML library.

Device ID

This parameter can be used to specify which GPU to use for inference when multiple GPUs are available. It has no effect when using CPU inference.

Inputs

source

The source image for which to create the image segmentation mask.

Note

If you wire in an RGBA, UV, or mono input, they convert to RGB.

mask

An optional per-pixel mask amount that’s scaled by the Mask parameter.

embeddings

Optional precomputed embeddings, which store the encoding of the image, to reuse.

points

Optional points to use as a prompt.

Note

When you wire in a points input, the node uses these point values and then appends the Points parameters' values. If you set a Bounding Box on the node, it replaces the bounding box in the points input.

Outputs

mask

The predicted mask.

embeddings

The embeddings generated from the encoder.

points

The points generated from the Points parameters' values or passed through from the points input.

Copernicus nodes