ML neural style transfer learning

   924   2   1
User Avatar
Member
41 posts
Joined: Sept. 2021
Offline
Hi everyone!

Idea of this thread is to share our failed ML neural style transfer attempts, with some insights in what work, what doesn't. Also to share some learning resources that would help us understand better how to successfully train our own ML models.

I successfully trained ML style transfer example with noise generator given from SideFX.
After that I thought to train it on our own terrain example. Here is first result of that, posting them in the order they were trained:







First two were trained with very big range of the lowest and highest point in the terrain, while 90% of terrain was plateau. Moment I isolated only that, thing started working better. I trained with different data set size, resolution, training rate...

This kind of visualizing onnx generations alognside with graph seems to me like a good way to make a sense of the process.
Did anyone try to use this? If so, what would be best learning resources for us VFX artists to take, so we are not left behind
Edited by bogdana - Oct. 22, 2025 16:37:48
User Avatar
Member
41 posts
Joined: Sept. 2021
Offline
Here's another attempt. This time for predicting texture details:
This time turned on testing of the model, while it was running.
I have images that contain red and green channel from normal, and blue channel for height.
Here is one of these images (512x512):

For pair i reformat it to 32x32 pixels, then back to 512x512:

And here is generated image that comes out as testing of the model:


Here are the graphs for the ML training:




And lastly here is when I try to do inference in COPS:
That ONNX inference node works fine with grayscale images, but with this rgb gives me totally different result than testing of the model produces.


Here is the hip file:
Image Not Found


And here is one set of pair/ref images:

Attachments:
399.9.reference.png (12.2 KB)
399.9.input.png (55.0 KB)
399.9.generated.png (330.3 KB)
loss.399.0.png (51.8 KB)
score.399.0.png (56.3 KB)
ssim.399.0.png (44.3 KB)
ML_inference_cops.PNG (1.2 MB)
ML_textureUpres_FX_fx_v002.hip (719.1 KB)
norm_0.png (20.5 KB)
pair_0.png (243.1 KB)

User Avatar
Member
41 posts
Joined: Sept. 2021
Offline
Here are results of grayscale (height) dataset, after 400 generations:
Input:

Generated image:

Reference image:


And this is when I run it in cops with ONNX inference node:


Clearly, ONNX inference gives worse result than testing of the model.

Attachments:
399.0.input.png (38.6 KB)
399.0.generated.png (149.8 KB)
399.0.reference.png (10.7 KB)
ONNX_inference_problem.PNG (1.6 MB)

  • Quick Links