Search - User list
Full Version: Gsplats training stuck at ML Preprocess Gsplats TOP step
Root » PDG/TOPs » Gsplats training stuck at ML Preprocess Gsplats TOP step
terry_williams
I've been trying to train my first gsplat in Houdini from a render to no avail, it always gets stuck at this step without apparent progress, I can see the single core that is working at maximum capacity but yesterday I left the machine all night with this and when I came back it was in the same place.

Only on one of the tries I had the console output the message "Dataset not found: /path/to/file/images.txt"

I tried to debug it but couldn't find any clue as to why the file didn't get generated.

Also tried to ran the test on two different Linux machines, tried the daily and production built with both py3.11 and py3.13.
One with a 3090 and another one with a 5080.

Here is the test scene in case you guys want to give it a check
cgi_splats [www.dropbox.com]

terry_williams
Also gave it a try on a W11 machine with a 4090 and still the same.
cdordelly
I think I found the issue, it is on the ML PreProcess GSplats node, inside the TOPnetwork from the recipe.

The COPs pre processing are lefting some NaNs values on the @P attribute (and other attributes as well) which causes to freeze the Fuse SOP node. If you add a Clean node to remove the NaNs, it should work. There are other attributes with NaNs but looks like those one are not causing a conflict.



Maybe this should vi checked directly with the dev team to add something to handle any NaN value result from COPs.

Cheers!
terry_williams
Thanks a lot a I was able to get through the training now!

Here is one of the test that I made:
https://superspl.at/scene/de1f667a [superspl.at]
This is a "lo-fi" version of our main content. To view the full version with more information, formatting and images, please click here.
Powered by DjangoBB