Hey everyone,

For the last few days, I was troubleshooting my system because I wasn't able to write a moderately sized USD File through a USD ROP in LOPs.

Long story short: After disabling everything I realized that Houdini was still crashing not because of faulty memory or reaching the physical RAM limit, but the virtual memory limit.

I haven't changed anything on my system for a couple of months but recently have been running into memory allocation issues constantly.
Now I wonder if this was a Windows issue (maybe it reduced the virtual memory size without me knowing) or if Houdini commits too much?

When writing something to Disk the committed memory starts out roughly double what's currently in the physical RAM. But then it seems to add a GB every one or two frames. This seems somewhat excessive as the actually used memory was growing by roughly 300 MB a frame and I wasn't able to utilize more than 63 GBs of my 128 GBs before crashing.

This is a screenshot from my task manager:

(This is with the increased virtual memory. The RAM is usually faster but I disabled X-AMP while troubleshooting)

Again, I am not sure where the issue is exactly but I remember being able to use all of my 128Gbs without having to increase the virtual memory just a couple of weeks ago and then it suddenly started with constant memory allocation errors.

Maybe the next time somebody writes something to disk they could watch their Task Manager to see if they notice the same behavior as I am? Or is your virtual memory bigger by default?

That's weird. Maybe you could take a look at the houdinifx.exe in the Resource Monitor and see the actual memory usage breakdown. Another thing to try maybe is to set the environment variable TBBMALLOC_PROXY_ENABLE to 0 prior to starting Houdini. Things will be MUCH MUCH slower because it disables use of the "fast" memory allocator. But maybe worth a try to see if it makes a difference at all.

Thank you for the answer Edward. I haven't tried to set the environment variable yet because I was hoping that with the increased virtual memory everything would be fine. But now I believe there is something wrong with the caching in Solaris. I was writing out a usd file which went fine but after that it didn't clear out the memory, resulting in another memory allocation error when I continued to work in Houdini.

Sorry for the photos but without memory I couldn't take screenshots:





There is little reason for Houdini to keep so much data in physical memory and absolutely no reason to keep so much data in virtual memory so that it crashes itself.
There isn't enough space on my System Disk to allow for even more virtual memory...

No_ha
Thank you for the answer Edward. I haven't tried to set the environment variable yet because I was hoping that with the increased virtual memory everything would be fine. But now I believe there is something wrong with the caching in Solaris. I was writing out a usd file which went fine but after that it didn't clear out the memory, resulting in another memory allocation error when I continued to work in Houdini.

I don't think you ever want to go into swap anyways so you better hope that you don't need more than what physical memory you have. The behaviour you're describing could be due to TBB Malloc so please try that first to rule out whether it really is something inside USD itself. If it is USD, then also try using the latest 19.0 daily build to see if it's still an issue.

Any news on this?

I am encountering same problems. When loading bunch of USD file caches for cloth, Solaris tries to consume large amounts of RAM. On 128 gb machines works fine, while on 64gb it crashes. When render actually kicks in, it consumes less than 64gb of ram, just that start is problematic.

Even in the original post, it isn't really clear what the root problem was... My best guess is that it related to a bug inthe USD library that was addressed in 19.5.332. But there are always going to be lots of ways to make USD consume lots of memory, bugs or no bugs. In the latest versions of 19.5 I am not aware of any bugs that are still contributing to this sort of problem.

So my suggestion would be that you generate a simple hip file that demonstrates the issue you're seeing using simple synthetic data (my favorite test case is a very high resolution torus with an animated Mountain SOP). If starting from the ground up doesn't let you reproduce the issue in a shareable way, try going top down by removing elements of your problematic hip file until the memory issue goes away. Sending a hip file into support that shows "these are the nodes that are causing the problem", even if you can't provide the raw USD source data, may give us enough information to figure out what the problem is (whether it's a USD bug or just something you're doing in your hip file that can be worked around). Thanks!

Update from myself:

I "fixed" the issue by massively increasing the available RAM swap in Windows. For me, it wasn't an issue of the actually consumed RAM, just the committed RAM that increased exponentially and eventually led to a crash.
I have monitored this a couple of times in the last few weeks and it seems like it doesn't overcommit as much anymore. It usually only commits a few GBs over the used RAM now.


Otherwise, I didn't experience any RAM usage out of the ordinary.

Htogrom
I am encountering same problems. When loading bunch of USD file caches for cloth, Solaris tries to consume large amounts of RAM. On 128 gb machines works fine, while on 64gb it crashes. When render actually kicks in, it consumes less than 64gb of ram, just that start is problematic.

I assume you just ran out of RAM the normal way. If the scene is too big there is no way around using "flush each frame".

I'll have to revive this topic.
Problem is still there on Houdnini 20.5

mtucker
So my suggestion would be that you generate a simple hip file that demonstrates the issue you're seeing using simple synthetic data (my favorite test case is a very high resolution torus with an animated Mountain SOP). If starting from the ground up doesn't let you reproduce the issue in a shareable way, try going top down by removing elements of your problematic hip file until the memory issue goes away. Sending a hip file into support that shows "these are the nodes that are causing the problem", even if you can't provide the raw USD source data, may give us enough information to figure out what the problem is (whether it's a USD bug or just something you're doing in your hip file that can be worked around). Thanks!

We've been experiencing the same issue at the studio. Haven't been able to pinpoint what causes this, but sometimes houdini seems to be increasing and accumulating RAM every time the stage gets loaded or small edits like transforms get dropped in, sometimes even if nothing on the stage itself has changed and the only thing that gets launched is Karma.

Example, we load in a USD asset made up of 50 other USD referenced assets, this consumes 15gb. After a few minutes switching viewports, launching KarmaXPU, moving some lights around. This can grow up to 100+ Gb until the system runs out of memory. Nothing we do seems to be able to flush the memory other than fully closing the houdini instance, not even deleting the entire scene or doing File-> New. We have to close the entire houdini session for it to release.

It is difficult to provide a hip file since (studio assets) but also it is hard to recreate since it happens 50% of the time we open a scene.

The most common thing I've seen leading to this kind of situation is (sometimes unintentional) caching within the LOP network. As an example, we recently got a bug submission where the Motion Blur LOP was causing memory bloat like this. It was an incorrect setting on the Cache LOP inside that HDA, which has now been fixed. So maybe look for Cache LOPs and Cache SOPs (especially ones buried inside HDAs).

Another interesting data point would be to know if you can reproduce without using Karma. This would help localize the problem to either being Karma causing the bloat or LOPs causing the bloat.

mtucker
The most common thing I've seen leading to this kind of situation is (sometimes unintentional) caching within the LOP network. As an example, we recently got a bug submission where the Motion Blur LOP was causing memory bloat like this. It was an incorrect setting on the Cache LOP inside that HDA, which has now been fixed. So maybe look for Cache LOPs and Cache SOPs (especially ones buried inside HDAs).

Another interesting data point would be to know if you can reproduce without using Karma. This would help localize the problem to either being Karma causing the bloat or LOPs causing the bloat.

Would you mind sharing what the incorrect setting on the Cache LOP is? We do have a couple of HDAs with cache LOPs inside, but none of those run during the session until we trigger an export. I'll try to make a simple scene to see if i can reproduce, I might be able to just make a loop that continuously swaps a usd reference to a new file (I noticed this same issue was happening when I was batch processing some assets in a loop)