I'm currently testing Karma (XPU) and working in LOPS. I stumbled upon something and wanted to make sure there aren't better ways.

If I understand it correctly, to actually see Motioblur I need time samples that usd can only access from native usd files (or adding a cache LOP). So if I import points from a simulation and instance geo on them using a Point Instancer, I don't see motionblur. But writing this Point Instancer to disk as a usd file gives me motion blur. (I instance in LOPs because it's a lot faster to instance random geo on points compared to for each loops)

Because I have millions of points the usd file is quite large and I need to resort to writing one frame at a time and using USD Stitch to combine them. Those are several steps more each taking quite some time and space.
I figured out a hacky(?) way of skipping this double step by putting a Configure Layer LOP with a Save Path before the USD ROP which is set to flush data each frame. It still only keeps the data of one frame in RAM but writes a single usd file for the Configure Layer LOP. Nice.

But while testing I noticed that writing a single usd file without flushing the data each frame will save massive amounts of data (as mentioned in the docs). My current file containing 300 frames is 40GBs. Having two separate files each containing 150 frames of the instances combine to roughly 20GBs. That is a massive reduction in file size and is probably good for I/O times as well.

I can use a sublayer LOP to import both of these files but of course, the second file's opinion is stronger and I don't see any animation before frame 151. I can manually animate the "mute" checkbox but I am wondering if there are better solutions?
Is this an intended workflow? Are there better ways to bring in massive amounts of data?

As far as I have seen, the USD Stitch ROP only accepts single-frame files, not files containing several frames.

In a perfect world, the USD ROP would automatically detect when RAM is running low and create a new file (and possibly combine them again). This would save the trial and error of finding out how much of the scene fits in your RAM and it would save the maximum amount of storage space.

No_ha
I figured out a hacky(?) way of skipping this double step by putting a Configure Layer LOP with a Save Path before the USD ROP which is set to flush data each frame. It still only keeps the data of one frame in RAM but writes a single usd file for the Configure Layer LOP. Nice.

This is not hacky. You shouldn't need the Configure Layer LOP before the USD ROP though. Just don't put "$F" in any of your Save File Paths, and then enable the "flush data after each frame" toggle on the USD ROP. This situation is exactly what that toggle is for.

As for the massive file size, I expect what that means is that you are not giving Houdini enough hints about what data coming from SOPs is animated, and what data is not animated. So things like topology that probably don't change frame to frame are being written out for each time sample. If you write a single USD file, the USD library automatically de-duplicates identical arrays of data, so you can end up with much smaller files if you have a lot of duplicate data in repeated time samples. It cannot perform this optimization if you are writing the data to disk after each frame. So you want to make sure that non-animated data is authored as a "default" value, not as a time sampled value.

Giving USD better information about what attributes on prims are animated can also make playback much faster.

Without seeint your hip file I'm not totally sure about how you should go about separating the animated data from the non-animated data, but we do provide a fair number of controls for this purpose on the SOP Import LOP.

Thank you for the detailed answer!

mtucker
Just don't put "$F" in any of your Save File Paths, and then enable the "flush data after each frame" toggle on the USD ROP.

Thanks! This is what I did in the Configure Layer. Don't know why I didn't try it in the USD ROP, too. :S

mtucker
As for the massive file size, I expect what that means is that you are not giving Houdini enough hints about what data coming from SOPs is animated, and what data is not animated.

I don't actually think the data is massive because of USD. I just saw how much smaller the file could be because of the de-duplication in single USD files.
The Vellum Grains sim I'm using is around 27GBs of data as a .bgeo.sc sequence. I upressed point count by 5 and deleted all attributes except Cd, pscale, orient, and v. Then I instanced 3 rocks randomly on those points. That's still a lot of data to write so I don't think that all that would be much smaller as a bgeo sequence. But I'll try to figure out how to set the non-animating attributes to default.

Right now I'm using the 40Gbs usd file and it's working fine. OpenGL performance is somewhat sluggish and I haven't found a way to display the instances as points, but Karma CPU is still very responsive. Karma XPU struggles a bit until the GPU starts to render but even with the 4 minutes it takes each frame to initialize the GPU it's still 80% faster than CPU only. (Honestly, I would never have thought XPU could deal with such a scene at all. At least not in an Alpha).


I might test splitting up the files again and animating the sublayer mute checkbox. It's not the prettiest way but saving 50% of the data makes this an interesting choice. (Or maybe I'll get to a similar small file size if I set more attributes to be not animated, although I suppose I won't be able to save as much as I can't set non-moving points in my sim to not get flagged as animated when others are still moving.)

If there are other people working with sims (or big scenes in general) in LOPs reading this, it would be interesting to hear how you are preparing the usd files.

I have tried 2 methods to import rbd simulation into lops and I think both are still far from optimal. The sim has around 70000 pieces.

In the first one I import geo as reference and the geo is not packed to lower primitive count. Then I use point instancer to transform each piece with simulation points without copying extra attributes. Each frame of the instancer is written out separately as value clips

The second method use transform point lop instead of instancer so the result layer have a lot of "over" to override each piece. Then again I export it as value clip.

I found that point instancer outputs 2-3x smaller clip file size and needs longer time to stitch. But viewport takes far more time to update (compose time?) than the second method and sometimes freezes.

With transform point lops, viewport is still usable but really slow. A workaround I use is to use Storm instead of Houdini GL (30 fps during tumbling in Storm compare to 1-2 in GL)

Right now I go with the second option but still looking for a better way.