FAQ FAQ      Search Search      Memberlist Memberlist      Usergroups Usergroups      Register Register      Profile Profile      Check private messages Check private messages      Log in Log in     

Memory Leak in H13 with Flips maybe ?

 
Post new topic   Reply to topic    Houdini Forum Index -> Technical Discussion
View previous topic :: View next topic  
Author Message
tricecold
Houdini Adept


Joined: 12 Jul 2006
Posts: 119
Location: Sofia

turkey.gif
PostPosted: Fri Dec 20, 2013 2:45 pm GMT    Post subject: Memory Leak in H13 with Flips maybe ? Reply with quote

Possible Memory Leak

H13

I seem to be having a very strange problem. Regular ocean surface 60 by 30 and 5 depth , passing velocities to a flip tank with similar size. Its like Igor Zanics coast beach Flip simulation

I have somehow detailed collision mesh on one side with like 200K faces and with a voxel size 0.1 it gives me sufficient detail almost 3mil VDB voxels. So VDBs as static collision( no deforming or translating collision objects here)

My machine has 32 gb ram and I am running out of memory with only around 20-25 mil points, which I believe is low as far as I can remember, Many videos on Vimeo are doing much more points with less ram.

The funny thing is I tried the same simulation on another PC with 190GB ram and 50 million particles were just passing 100 GB ram usage,

The other thing I noticed is that, the ram usage just continues to increase and even triples itself over a period of 500 frames, however the particle count pretty much (or the file size) doesn't change so much at all.

Please help me out here. I am about to reinstall a linux distro to see if there is sth wrong with my windows
Back to top
View user's profile Send private message Visit poster's website
johner
Houdini Expert


Joined: 11 Jul 2006
Posts: 428
Location: Austin, TX

usa.gif
PostPosted: Fri Dec 20, 2013 3:19 pm GMT    Post subject: Reply with quote

With H13 we moved to a new memory allocator under Windows that is much faster than the default one provided by Microsoft. Unfortunately it can fragment memory over time, which looks a lot like a leak, though technically speaking it is not.

Until we can work out a solution, we're recommending running very large simulations under Linux if at all possible.
Back to top
View user's profile Send private message
tricecold
Houdini Adept


Joined: 12 Jul 2006
Posts: 119
Location: Sofia

turkey.gif
PostPosted: Fri Dec 20, 2013 3:25 pm GMT    Post subject: Reply with quote

ok I see,thanks man. This has been driving me crazy, is this an official announcement, or is this from manual Smile .
Back to top
View user's profile Send private message Visit poster's website
Solitude
Houdini Expert


Joined: 27 Mar 2009
Posts: 272
Location: Vancouver

usa.gif
PostPosted: Fri Dec 20, 2013 6:54 pm GMT    Post subject: Reply with quote

johner works for sidefx, so he would know Smile That being said, I was running into the same thing recently, just wasn't sure exactly what was going on. It seems to fill up my available ram, but in terms of simming speed it held steady, and the computer was still quite responsive. Still, guess I'll be setting up dual boot this weekend. Wink
_________________
Ian Farnsworth
Back to top
View user's profile Send private message Visit poster's website
tricecold
Houdini Adept


Joined: 12 Jul 2006
Posts: 119
Location: Sofia

turkey.gif
PostPosted: Sat Dec 21, 2013 3:39 am GMT    Post subject: Reply with quote

I suppose this issue applies to other tools in Houdini 13 also, like Pyro, RBD multisolvers etc, on windows, please confirm.
Back to top
View user's profile Send private message Visit poster's website
johner
Houdini Expert


Joined: 11 Jul 2006
Posts: 428
Location: Austin, TX

usa.gif
PostPosted: Sat Dec 21, 2013 1:02 pm GMT    Post subject: Reply with quote

Solitude wrote:
IIt seems to fill up my available ram, but in terms of simming speed it held steady, and the computer was still quite responsive. Still, guess I'll be setting up dual boot this weekend. Wink


Yes, the allocator (tbbmalloc) allocates memory from the OS in pools and manages them itself for speed. When memory gets fragmented, it can't release these pools back to the OS, although it can still use the free memory within them. The Linux allocator (jemalloc) does a much better job of returning that memory to the OS.

Linux has always been the best OS for large scale simulation, but it's even more true at the moment.

tricecold wrote:

I suppose this issue applies to other tools in Houdini 13 also, like Pyro, RBD multisolvers etc, on windows, please confirm.


With Reseeding and the boundary layers in the OceanFX tools, the FLIP solver is often creating and deleting large numbers of particles each timestep, which puts a lof of pressure on the memory allocator. So it seems to show the problem the worst by far. Many other tools that don't create and delete memory as frequently won't be affected much (if at all.)
Back to top
View user's profile Send private message
Korhon
Houdini Expert


Joined: 16 Jul 2007
Posts: 321
Location: Oslo

norway.gif
PostPosted: Sun Dec 22, 2013 8:31 am GMT    Post subject: Reply with quote

Seems to me that simulating whitewater also suffers from this. Growing memory in windows and stable in linux Smile
_________________
www.gimpville.no
Back to top
View user's profile Send private message
Korhon
Houdini Expert


Joined: 16 Jul 2007
Posts: 321
Location: Oslo

norway.gif
PostPosted: Sun Dec 22, 2013 8:43 am GMT    Post subject: Reply with quote

Maybe it should be an option to disable the tbbmalloc and use the old one instead if possible.
_________________
www.gimpville.no
Back to top
View user's profile Send private message
tricecold
Houdini Adept


Joined: 12 Jul 2006
Posts: 119
Location: Sofia

turkey.gif
PostPosted: Wed Jan 22, 2014 4:34 am GMT    Post subject: Reply with quote

update,

The problem exists only in Windows but unlike hogging the machine to a halt, the simulation continoues, but you do start hearing the HD going bananas and eventually Windows returns not enough memory error, maybe crashing Houdini along the way, happened on the 192GB ram machine with 50 mil particles.

I tested the same scene on my Ubuntu, another scene with 25mil pts this time, simulation starts around 6gb ram usage and when the collisions happening, it peaks around 20-25 gb usage but I still find this amount of ram usage abnormal compared to previous versions or what Naiad was able to do with 8GB ram .

So far however at work and at home on linux this doesnt seem to be a big issue, but on windows , flips are a no go in my particular situation.
Back to top
View user's profile Send private message Visit poster's website
CeeGee
Houdini Adept


Joined: 24 Jul 2007
Posts: 181
Location: Serbia

serbia.gif
PostPosted: Wed Jan 22, 2014 7:13 am GMT    Post subject: Reply with quote

Not sure about your houdini settings, but i manage to run 80mil particle sim in houdini with only 20GB of ram on linux,
https://vimeo.com/74197060

And 55mil, with small grid scale ( use more memory) with only 18Gb of ram
https://vimeo.com/82942136

Be sure to optimize your scene before run some high res simulations.

Igor

_________________
WEB
http://www.igorfx.com/

Vimeo
http://www.vimeo.com/user1342200/videos
Back to top
View user's profile Send private message Visit poster's website
grayOlorin
Houdini Master


Joined: 05 Oct 2010
Posts: 1008
Location: Austin, TX

usa.gif
PostPosted: Thu Jan 23, 2014 1:06 am GMT    Post subject: Reply with quote

I can confirm that I have been having difficulties dealing with large geometries (scan data) and memory. I do have a machine with little ram though.. only 12. However, it seems that certain operations that were not as bad in 12.5 are getting a bit heavy in memory, and, as mentioned before, it is hard to get that memory back without killing houdini ):

Looking forward to a solution on the tbbmalloc... pretty please Smile

_________________
-G
Back to top
View user's profile Send private message
edward
Houdini Guru


Joined: 04 Jul 2002
Posts: 5675
Location: Toronto

blank.gif
PostPosted: Sun Jan 26, 2014 7:32 pm GMT    Post subject: Reply with quote

The immediate workaround is to use the VC11 Houdini build on Windows and then copy/replace the tbbmalloc_proxy.dll over from a H12.5 install into H13.0.

This will forcefully disable tbbmalloc_proxy.


Last edited by edward on Mon Jan 27, 2014 11:22 pm GMT; edited 1 time in total
Back to top
View user's profile Send private message
grayOlorin
Houdini Master


Joined: 05 Oct 2010
Posts: 1008
Location: Austin, TX

usa.gif
PostPosted: Sun Jan 26, 2014 11:29 pm GMT    Post subject: Reply with quote

thank you for the tip edward! I am going to give it a whirl for the short term
_________________
-G
Back to top
View user's profile Send private message
grayOlorin
Houdini Master


Joined: 05 Oct 2010
Posts: 1008
Location: Austin, TX

usa.gif
PostPosted: Mon Feb 03, 2014 7:06 pm GMT    Post subject: Reply with quote

but just to know, is there an ETA on a potential fix for this or even a way to officially switch off tbbmalloc?
_________________
-G
Back to top
View user's profile Send private message
edward
Houdini Guru


Joined: 04 Jul 2002
Posts: 5675
Location: Toronto

blank.gif
PostPosted: Tue Feb 04, 2014 12:45 am GMT    Post subject: Reply with quote

I think it be can fixed by upgrading Houdini to use the latest TBB version. The memory usage is still more than Linux but at least its growth is stable. There has been a few issues that have cropped up and we've had to work through them alongside with other tasks. There has been some question as to whether it is a good idea to do this for all platforms, or just Windows.

Unfortunately, there's no way to switch off tbbmalloc_proxy automatically at run time, it must be all or nothing.
Back to top
View user's profile Send private message
Korhon
Houdini Expert


Joined: 16 Jul 2007
Posts: 321
Location: Oslo

norway.gif
PostPosted: Tue Feb 04, 2014 6:23 am GMT    Post subject: Reply with quote

Thats good news Edward!

We are are almost forced to switch to linux atm because of this, but we do like windowsland alot better since we already have everything conifgured for windows.

_________________
www.gimpville.no
Back to top
View user's profile Send private message
Joker386
Houdini Guru


Joined: 12 Dec 2010
Posts: 1143

iran.gif
PostPosted: Tue Feb 04, 2014 9:24 am GMT    Post subject: Reply with quote

Yes ,I had some others problems about memory and RAM on the Windows.
But when I switch on the Linux ,My simulations worked really good !

I tested some different project about RAM usage on Windows and Linux ,RAM Usage on the Linux is must better than Windows ,Even Rendering time with Mantra on the Linux is faster than Windows !

If you didn't work with Linux and you like Windows Interface ,I suggest you to install Linux Mint ,It's have pretty lovely user interface exactly like windows .

Also I found another way to simulation large scale project (Pyro , Flip Fluid) on the Windows to prevent Ram Usage problems ,I set "Cache Memory (MB)" parameter in the AutoDopNetwork to zero and used "Explicit Cache Files" option to save simulation's caches on the disk ,I don't know But maybe this solution can be useful in your project too Confused

Also If you have SSD drive ,Format this drive as EX4 and save your cache files on this drive ,In my personal experience and projects ,SSD with ex4 on the Linux is 2.5 times faster that SSD with ntfs on the Windows ,Also HDD ex4 drive is about 1.5 times fastr than ntfs Cool

_________________
www.JKCompany.org
info@JKCompany.org

Rabiee.Sadjad@Gmail.Com
Back to top
View user's profile Send private message Send e-mail Visit poster's website
edward
Houdini Guru


Joined: 04 Jul 2002
Posts: 5675
Location: Toronto

blank.gif
PostPosted: Tue Feb 04, 2014 11:45 am GMT    Post subject: Reply with quote

Korhon wrote:
We are are almost forced to switch to linux atm because of this, but we do like windowsland alot better since we already have everything conifgured for windows.


IMHO, moving to Linux for production is the right thing to do. Linux is still much faster on many fronts than Windows. This is coupled with the fact that the majority of paying customers (and developers) are on Linux.

The new TBB version is not a panacea, the memory usage is still high enough on Windows that I expect that some people will still need to swap. And if you need to swap, then it's game over already.

Note also that if you turn off the use of tbbmalloc_proxy (on Windows only, Linux uses jemalloc), then you're again several times slower. Even in the best case scenarios with Windows using tbbmalloc_proxy vs Linux on jemalloc, Windows is still slower.
Back to top
View user's profile Send private message
edward
Houdini Guru


Joined: 04 Jul 2002
Posts: 5675
Location: Toronto

blank.gif
PostPosted: Thu Feb 13, 2014 1:21 am GMT    Post subject: Reply with quote

grayOlorin wrote:
but just to know, is there an ETA on a potential fix for this or even a way to officially switch off tbbmalloc?


On Windows, Houdini 13.0.322 now uses the latest TBB version (4.2 update 2), which contains the improved memory allocator.

EDIT: Corrected version


Last edited by edward on Thu Feb 13, 2014 9:02 am GMT; edited 1 time in total
Back to top
View user's profile Send private message
Joker386
Houdini Guru


Joined: 12 Dec 2010
Posts: 1143

iran.gif
PostPosted: Thu Feb 13, 2014 2:13 am GMT    Post subject: Reply with quote

edward wrote:
grayOlorin wrote:
but just to know, is there an ETA on a potential fix for this or even a way to officially switch off tbbmalloc?


On Windows, Houdini 13.0.323 now uses the latest TBB version (4.2 update 2), which contains the improved memory allocator.


That's very good ,I should test it later ,Because sometimes we have to use windows instead of Linux ,But not all Very Happy

_________________
www.JKCompany.org
info@JKCompany.org

Rabiee.Sadjad@Gmail.Com
Back to top
View user's profile Send private message Send e-mail Visit poster's website
Korhon
Houdini Expert


Joined: 16 Jul 2007
Posts: 321
Location: Oslo

norway.gif
PostPosted: Thu Feb 13, 2014 6:20 am GMT    Post subject: Reply with quote

Trying out latest build now and it works. My flip sim that previously would only sim in linux uses the same memory i windows now Smile

Woohooo

_________________
www.gimpville.no
Back to top
View user's profile Send private message
MartybNz
Houdini Guru


Joined: 26 Jun 2012
Posts: 2648
Location: New Zealand

australia.gif
PostPosted: Fri Feb 14, 2014 4:15 am GMT    Post subject: Reply with quote

edward wrote:

On Windows, Houdini 13.0.322 now uses the latest TBB version (4.2 update 2), which contains the improved memory allocator.



Out of interest are there particular nodes in simulations that benefit more than others with this?
Back to top
View user's profile Send private message Visit poster's website
edward
Houdini Guru


Joined: 04 Jul 2002
Posts: 5675
Location: Toronto

blank.gif
PostPosted: Thu Mar 20, 2014 9:34 pm GMT    Post subject: Reply with quote

MartybNz wrote:
Out of interest are there particular nodes in simulations
that benefit more than others with this?


Sorry, I missed this one. Er, I'm not sure to be honest. The patterns that the old TBB allocator had problems with were situations in which one thread would allocate memory that another thread would end up freeing.
Back to top
View user's profile Send private message
Friday
Lurker


Joined: 23 Jul 2013
Posts: 1

blank.gif
PostPosted: Tue Nov 18, 2014 3:09 am GMT    Post subject: Reply with quote

Having the same issues on Windows as everybody else and wanting to get into large scale Flip simulations I decided to install Ubuntu 14.04 Lts alongside my Windows 7 Installation. Thought this would be the key to solving all my problems. It took me a few days to get my head around all this and finally I could start a simulation which I already tested on Windows.

A simulation with a constant particle count of about 40 to 46 million starts occupying about 20 GB of my 48gb of ram fills up the memory completely within about 100 frames and crashes Houdini. (I did not activate a swap partition because I thought I would not need it for now). Now in Windows the Ram also fills up but to a far lesser amount then in Linux. With the exactly same sim it started with 15gb an went up to 35 GB at frame 160.

I also noticed that the .sim files written out on Linux are bigger then the ones written out on Windows and really the only thing I changed in the Houdini file is the location of the Explicit Cache.

I already set the Cache Memory to 0 and enabled Explicit cache.
As I am totally new to Linux I`m sure that I`m missing some settings.

Help would be appreciated.

Cheers.
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic    Houdini Forum Index -> Technical Discussion All times are GMT - 5 Hours
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


phpBB component v1.2.4 RC3 by Adam van Dongen. Based on phpBB © 2001, 2002 phpBB Group