Hi SESI.
I am experimenting some inestabilities using the File DOP in H 8.0.572 in Ubuntu Breezy.
Every time I use the File DOP to read a simulation, previously cached with th ROP Output Driver, Houdini comes unstable.
For example, if i try to make a flipbook from the File DOP or launch a render it fails. In the case of the flipbook Mplay isn't launched at the end of the flipbook generation. And if I try to launch a render, after using the File DOP I get the next error:
Fork failed for ‘exec mantra’: cannont allocate memory
Moreover if I use File DOP the kernel “eats” all the cpu time. If I cook for example a RBD Solver all goues well, but if then I go to a File DOP the kernel comes crazy a begin to use the cpu, so it almost impossible to use this operator for me.
I know is a starnge error, and I don't know if there are anybody with the same problem, but it only occurs with the File DOP so I think that is related with Houdini not with the system.
Thanks.

Hiya,

So, the error suggests you're running out of memory. Generally speaking, once a 32-bit Houdini process hits the ~2GB per thread memory limit it doesn't play well after that. Have you tried flipbooking/rendering just one frame?

How much memory do you have? run the unix command: "free“. It shows how much free memory and swap you have. Check the values in the free column are not all zero!

I'm assuming your unix shell isn't limiting things. Run ”ulimit -a“ to check. Most ”memory“ lines should say ”unlimited“. Ubuntu might set some limits. This is unlikely, but worth checking.

The huge CPU usage suggests your machine is paging (”swapping“) hard. This happens when you run out of physical memory. Kernel eating all CPU time = swapping.

Oh, the CPU can also seem very busy if you're writing lots-and-lots of data to a slooow harddisk. If you have a coloured CPU meter (I run ”ktimemon“ in my KDE Panel. Recommended!) you tend to see yellow for swapping, and red for writing. But that's just a convention. Run ”top“ to see which.

Aside: If you've crashed a few times (esp. with the Houdini Web Browser running) chances are you might have some shared memory lingering. That will chew up your available memory. run ”ipcs“ to see it. Use ”ipcrm" to remove them. I won't explain further as ipcrm is a risky task (ie. if you remove the wrong segment!), so RTFM. It might be better to just reboot in this particular case (i hate saying this).

I found some issues writing out simData back in December, but that was fixed in 8.0.452. Maybe there's still a problem there?

Maybe try turning off stuff in the DOP Object's Simulation Tab (at the top level), like the options: “cache simulation” and maybe “automatic recook” etc. That might gain some memory.

Meantime, to get things done™, instead of using the FILE DOP at the moment, you might do better to just write out .bgeo geometry via an Object Merge Sop and Geometry ROP. Then read those .bgeos back in with a FILE SOP and render from there. Actually, I think the Geometry ROP can directly reference a DOP path, so you don't need the Object Merge SOP.

cheers, ben.

ben simons
Aside: If you've crashed a few times (esp. with the Houdini Web Browser running) chances are you might have some shared memory lingering. That will chew up your available memory. run "ipcs“ to see it. Use ”ipcrm" to remove them. I won't explain further as ipcrm is a risky task (ie. if you remove the wrong segment!), so RTFM. It might be better to just reboot in this particular case (i hate saying this).

Message queues (on Unix) haven't been used in Houdini since March 2004 (ie. 2 years). To make a long story short, don't bother with ipcs/iprm.

Is this problem new to 8.0.572? In other words, are you doing something with the File DOP that you did in a previous version, without problems, and now you've installed 572 and there are problems? Or is this the first time you've tried to do whatever it is you're doing?

Thanks for your replies.
Ben:
I cheked the Houdini process with top the first time i saw this strange behaviour, and it reports that houdini was using almost 1.2 GB of vurtual memory, and almost 800 MB of physical memory, this is one of the reasons due to I think that it was a problem from Houdini. I have had processes using almost 2 GB of virt mem without problems. The question is that when i read sim files from File DOP the kernel is using all the CPU reading from disk and when I use the File SOP all works fine. I have tried ulimit -a (thanks I didn't know this command) and it reports unlimited limits.
I have had the problem with and without the help pane opened.
Yes I am almost sure that the problem is caused by the File DOP because it provokes many memory paging. But why if I play the simulation from the RBD Solver node all goes well and if I play the simulation from the File DOP then the system begins to page memory? Both are using the same caches (both are in the same RBD Network).
This behaviour leads me to thinking that the File DOP is provoking memory leaks.
Yes, I have the problem solved passing the simulation usinf a DOP Transformation SOP and the generating a bgeos sequence, and reading them using File SOP. the bgeo File SOP due is always a safe solution.

Craig:
Ihave test with 8.0.572 and 8.0.539, same behaviour.

After making many test I think that I know what is the problem.
When you use the File DOP for read is very easy to use a great amount of memory quickly due to the in/out buffers used by the filesystem subsystem.
Usually this buffers lives in RAM, this is the reason because they are filesystem buffers, it allows to read recently hard disk data from ram instead to go to the disk for them. And Linux make very intensive use of this buffers.
For example make a huge copy of data, 1GB or more, and see, using free, your buffers.
Well here come the problem, you normally are caching your simulation, and this cache lives in ram too, so if your ram is almost full by the dop cache your buffers will go to swap, and the you have a beauty bottleneck in your system.
I think that it doesn't happens with File SOP because .bgeo are smaller that their corresponding sim files, at least for the scene I am working on, and then you don't need the same ram for the buffers, and like you aren't in dops you don't have the dop cache so you have all of your ram for your buffers, this is the reason because File SOP works fine.
So I think that the best workflow is:
1-Write sim files using ROP Output Driver, then use a File DOP to recover them.
2-Go to SOP and there put the DOP in SOP using and writeObject Merge SOP or DOP Transform SOP,
3-Write a bgeo sequence of your dynamic object, ensure that in the RBD Network the display flag is in the File DOP and disable the simulation cache in the RBD Network. You can reuse your sim for every object you want to put in SOP but without hanging your system. And in this stage the dop cache is useless.
4-Read your final objects with dinamics using a File SOP.

And … That's all