Hello. I am exploring usd and now I have some animated sequence (number of points does not change, only positions of them (69 frames, 100mb each). When I am exporting usd without flushing frames, animation weights around 150mb. Unfortunatelly around 40 frames, I ran out of memory. My solution was to split this sequence into two separate files but its nod ideal. I've tried exporting with flushing and then stich them into one file, but it's weight is just a sum of all single files. I would like to create one file with 69 frames, just like without flushing, reading this file is very fast and its lighweight (I've read someting that without flushing, usd saving only what changes, and in my animation, of 1mln points each frame, only about 200 of them changes position. I've also tried value clips, but it still loads slowly, not as reading file exported without flushing. Is this possible to solve this issue?

I don't think you're missing anything. USD's automatic de-duplication of data only happens when all the data is in memory. So if you flush data to disk after each frame, or try to stitch files together after the fact, the de-duplication doesn't happen and your file size will be larger.

The only other thing I would suggest is that when authoring the data, if there is some way you can split the animating data from the non-animating data at the SOP level, and import these two pieces separately, then you can get the best of all worlds. It will probably all fit in memory at once, so you can leave flushing off. Or if you do have to turn on flushing per frame, the size won't grow nearly as quickly because only the animating data will be added with each frame.

mtucker
USD's automatic de-duplication of data only happens when all the data is in memory

Why does all data have to be in memory?

Can't it just keep in memory the so far stitched stage and the next frame (or time sample) stage?

I don't really know how the deduplication works. I'm just basing this off the empirical evidence I've seen for how file sizes seem to indicate what data is getting de-duplicated. Then I made some assumptions about them not wanting to read data off disk just to avoid duplicating data when adding more data to an existing file. Maybe I'm wrong? Or maybe it could be improved, but that would have to be Pixar making any such improvements.