mtucker
Indeed. Please give it a try and let me know if the performance has gotten to where you want it to be. Thanks!

Pretty sure that our need for performance will never be satisfied..and as soon things get faster, the workload gets bigger.
But this is very appreciated and I am going to test it today

The topology usd file seems to be large that contain geometry data. Could root USD references .bgeo only?

So what is the recommended way to handle large FX caches in Solaris then?

Right now when we want to render, we are required to load every single frame of the bgeo sequence into RAM (not all at once). With longer shots it also means longer caching of the USD ROP for the whole stage because of the loading of the primitives. There was the update of the Geometry Clip Sequence node with the All Clip Files Have Matching Scene Graph Structure checkbox. When recaching this geoclipsequence into its own USD, I have to sample on the node the whole timeline, but then the process is relatively fast (few seconds at most). But then I have a problem with referencing the usd back in stage as now this usd is loading all the timesamples at once (my session usually crashes before it loads, so this is just a hypothesis, not confirmed).

Back to my original question then. What is the recommended way to handle large or long bgeo caches in Solaris? How do I reference it back in stage, so the USD Stage is lightweight and only references the bgeos either in previous USD file or in itself without loading all the frames during the process?

Any hips or image is much appreciated!

The answer to your question is "the geo clip sequence LOP". If it's not working as expected, or not generating results that work for you, then please provide some kind of example/demo file that shows this happening. This is the intended and expected workflow, so if it's not working, we'd like to fix it.

Loading a properly structured value clip into a scene should not cause all the clip files to be opened at once. And the geo clip sequence LOP should always be authoring properly structured value clips.

I did notice you saying that you sublayered the value clip file into your scene, which is a bit unusual. Generally when working with value clips I've seen the referenced onto a stage rather than sublayered. I don't think that should matter, but it might be worth testing...

Does this workflow support a sequence of vdbs? Plugging a volume LOP into the 2nd input it correctly shows it as a value clip, but caching it hangs.
Alternatively directly pointing to the vdb sequence the clip only caches one frame, and lists the volume filePath as a Default Value instead of Value clip. Not sure where I'm going wrong.

luijee
Does this workflow support a sequence of vdbs?

For volume data you may want to check out the VolumeLOP.

Definitely use a Volume LOP, and using value clips buys you absolutely nothing in this situation. VDBs are "large geometry", but not in any sense that USD is aware of. USD handles VDBs like texture maps - simple paths to files on disks containing data that only the renderer cares about. Value clips are useful to split large amounts of USD data between multiple USD files on disk. But VDB files are already organized this way, so adding value clips into the mix adds complications to the authoring workflow and provides no advantages to the runtime or rendering.

To be clear, you _can_ use a volume LOP and value clips together, but there isn't enough information in your description to provide any useful analysis of why your setup isn't working, and I'm not bothering to ask additional questions about it because in the end I'll still tell you to stop trying to use value clips

what is the recommended way for deadling with a sequence of heavy animation .usd files that I need to layer over the static .usd file.
for anim I am caching only points and velocities while the static has the bulk of the data.
geoclip sequence is slow for me but I am using 20.0.547, should I upgrade or it wont matter if I am dealing with .usd files not .bgeo?

protozoan

tamte
Why would it need to load any file to memory?

It used to be like that, but that was a long time ago and now should not be necessary anymore. The value clip generation should zing just through, even for large datasets.

robp: A lot of people seem to struggle with this. Maybe this warrants a case-study tutorial like "Solarifying production caches efficiently from start to finish" without skipping any checkboxes etc.

1. Soo is there an example file or a tutorial available currently?
   
2. I saw the example file using bgeo above, but I have concerns with using bgeo. It doesn't seem practical because, after any simulation, there's usually a step where additional elements are added, requiring another round of caching. Using USD makes more sense since it's a more LOP-friendly format?
   
3. Is there a way to automate this process similar to the Labs File Cache? The current method feels quite tedious :S
   

Also there is this thread [www.sidefx.com] and there is an example in there using usdstitchclips [www.sidefx.com] but I guess it's not the option anymore?

1. I don't know.
2. Using USD for your value clips is definitely faster to load than using BGEO for your clip files.
3. Automate which process? There is a File Cache LOP you could try?

mtucker
1. I don't know.
2. Using USD for your value clips is definitely faster to load than using BGEO for your clip files.
3. Automate which process? There is a File Cache LOP you could try?

Thank you
3. Right now I have to manually fill all of these fields by hand. Yes I can create path presets but if there is a way to have this node to be included to a FIleCache somehow? What's the approach for that? I think we need a tutorial.

4. How do you actually cache this node? I tried to pass it to top rop fetch but it failed. Do we need to use lop file cache after it, if so, why do we need to recreate the output path in the filecache node if we already have it in the geometry sequence, it's very confusing

Geometry Clip Sequence authors the value clip in memory, like all other LOP nodes. It is not a ROP, as it never writes anything to disk itself. It requires a separate node (like a USD ROP or a File Cache LOP) to cause the value clip to be written to disk.

Value clips involve creating a number of separate USD files. None of these files are "active layer" (the layer to which the value clip metadata is written). It is this active layer that is controlled by the Output File parameter on the File Cache LOP.

mtucker
The answer to your question is "the geo clip sequence LOP". If it's not working as expected, or not generating results that work for you, then please provide some kind of example/demo file that shows this happening. This is the intended and expected workflow, so if it's not working, we'd like to fix it.

Loading a properly structured value clip into a scene should not cause all the clip files to be opened at once. And the geo clip sequence LOP should always be authoring properly structured value clips.

I did notice you saying that you sublayered the value clip file into your scene, which is a bit unusual. Generally when working with value clips I've seen the referenced onto a stage rather than sublayered. I don't think that should matter, but it might be worth testing...

I did some tests on storing valueclip, using the 20.5 version of the geoclipsequence node.
This is the problem I encountered many times in my tests:
If you directly use an existing file (bgeo or vdb) to link, when loading the valueclip, all the linked files will be read once before refreshing the data in the window.
Only after turning on Flatten Clip Files and regenerating the existing geo into a usd file, the read valueclip file can be read correctly according to the corresponding frame number. It will not read all the cache at once.
The same problem occurs when using sublayer and reference nodes
It is recommended to use a large cache to test, so that you can intuitively see that the hard disk reading speed is completely different.
I have also tested it on many machines and with the 20.0 version.
Is there any other way to solve this problem?

Have you tried turning on the (relatively new) "All Clip Files Have Matching Scene Graph Structure" parameter? Enabling this option should prevent the geo clip sequence LOP from having to open all the clip files in order to author the clip data (it normally has to check all the clip files in order to build the manifest and topology files). Turning off "Flatten Clip Files" (if you know this step isn't necessary, as it would not be for BGEO clip files because they don't support composition) can also help improve the performance of this node.

But to work this way, you are responsible for ensuring that the scene graph structure is not changing over time and that the clip files don't need to be flattened.

mtucker
Have you tried turning on the (relatively new) "All Clip Files Have Matching Scene Graph Structure" parameter? Enabling this option should prevent the geo clip sequence LOP from having to open all the clip files in order to author the clip data (it normally has to check all the clip files in order to build the manifest and topology files). Turning off "Flatten Clip Files" (if you know this step isn't necessary, as it would not be for BGEO clip files because they don't support composition) can also help improve the performance of this node.

But to work this way, you are responsible for ensuring that the scene graph structure is not changing over time and that the clip files don't need to be flattened.

Yes, I have taken care to ensure that these problems do not occur, the scene graph structure is not changing over time, the problem still exists

Here's my take on some of that stuff.

https://youtu.be/aLyO883mkoQ [youtu.be]

robp_sidefx

sekow
in the presentation you can see the lop went time dependent

Yes, one of the things I didn't show (but really should have) is that, as with many of our other LOPs, we can generate a range of time samples directly from this LOP and avoid introducing the time dependency.

Hey Rob, I realize this post is a bit old, but I just came across it (very useful thread). I’m wondering where the “/Geometry” path is coming from when sourcing from a bgeo sequence. Is this a known but undocumented path to the Houdini geo in the bgeo file? I see that anything else I put into that field will error out the LOP. Just curious how/where that came from.

Nevermind. I see - it is can be configured int he USD Configure SOP before you write out geo to disk.

Hello!

The hipfile attached shows some example workflows for caching .usd per frame files and stitching them together using the Geometry Clip Sequence LOP.

Workflows in the hip:
0. Caching usd in a single file. (for comparison )
1. Default workflow by Sidefx (?)
2. Hack 1 for caching in chunks
3. Hack 2 for caching in chunks


q1. Is this set up correctly? (1)
If so, this errors when trying to cache in chunks (on the farm) because:
A) race condition over .usda stitch single file.
B) each chunk will write a stitch file containing only the frames of that chunk, and only the last chunk written will remain.


q2. Are there other workaround a part from the hacks presented in the file? (2,3)


q3. When 'All Clip Files Have Matching Scene Graph Structure' is on, only the first frame is checked for manifest and topology, correct?







Thanks for helping!
Houdini 21.0.596

Davide