Hello,

Here is something I can't say I quite comprehend…

I have a fairly simple jpg sequence (1-420 @ 720x540 @ ~30kb each), process through a few COPs (File, Color, scale, crop, over, rop_output) and write out a new jpg (tried tiff too) sequence (at 720x480). Now, what took me by surprise is that the RAM usage here is just constantly climbing up and beyond 1GB usage as the render is progressing frame by frame… (Opteron 270 w/ 2GB RAM on XPsp2). I've gotten insufficent memory crash earlier.


Is this some kind of a bug? This kind of RAM usage is not normal for such a simple network, is it?


Thanks.
A

Is this with 8.1.697 or later? If not, try a build later than that. If that doesn't help, post a .hip file. Just make any simple image that matches the same characteristics as your test file. (ie. run iinfo from the command shell on your image).

Hi e.d.w.a.r.d.

Ooops. My bad. I had completely forgotten to mention that I am using Houdini 8.1.705… ops:

I can't post the actual img sequence I am using here. However, I grabbed some sufficiently big jpg off from the net, append a Transform COP after it to simulate my File COP issue. This pretty much get really close to the problem I was running into.

Again, I don't know if this is normal, but, for a 13mb image sequence to consume close to 1gb of RAM is a bit abnormal, I think… :?

I think you're confusing *compressed* image size vs uncompressed memory footprint. Note also that .jpg's are a *lossy* compression. Looking at your example file, your image is 1024x753 at 8 bits per component, RGBA, 420 frames. So raw memory size is 1024x753x4x420 ~= 1.2 GB. Plus multiply that on a per COP basis and you easily reach high memory limits. So trying to look at it from a file size standpoint is irrelevant.

What *is* relevant is whether the memory hits the cache limit and stays there. Your example file appears to work fine. The memory for my houdini session right now is at a steady 1.28 GB VM Size after several times through the playback sequence. I have my COP viewer display cache ('d' in the COPs viewer) set to 128 MB, and from the Cache Manager window, I see that I've maxed out my COP Cook Cache at 768 MB. So this totals 1 GB. Add to that UI and various housekeeping, my Houdini memory usage is reasonable.

You should confirm that your COP display cache and cook cache doesn't exceed 1 GB together. If you still run into problems with your hip file, then perhaps there is a memory leak. However, you will need to provide your exact .hip file. Again, it's not necessary to provide your real images, just your exact image info as obtained via iinfo.

Hi Ed.

Thanks for the bit of explanation. I guess image processing is probably one of the many areas I have yet to learn.

Though I can't say I fully comprehend what's going on behind the scene, but as long as it's not bug, I guess it's all cool. It just that the amount of RAM it took to process a very simple network (coupled with my lack of understanding) kinda vexed my mind a bit, as the similar operation on the same image sequence in Nuke took far less RAM and faster.


Thanks again, Ed.

Last time I evaled Nuke, it worked off disk, doesn't it? It's memory usage barely budges, but it exercises your disk drive. Might have been an option, dunno, but that's the way it worked out of the box. Houdini is trying to let you store as much in memory as possible, like Shake.

Cheers,

J.C.

Yeah that's true. RAM are cheap these days. I am still thinking about throwing another 2gb in there eventhough I don't absolutely need that much..

Huh… I think my lightbulb just went up. I think I get what you're saying now. Thanks Ed! I guess basically what's going on is that since jpg is essentially compressed image, so on disk file size will appear to be much smaller than an uncompressed image where every pixel information is stored on disk. And since COP has to process every single pixel, thus the size on disk is irrelevant, and thus the memory usage hike because it needs to allocate that much space for every pixel. Sighed… I AM slow today…. I need to get some sleep. : P


So, when COP info shows “Total Byes per pixel: 4”, does this means “4 bytes per pixel per channel per plane”? Or “4 bytes per pixel for RGBA per plane”?


Thanks again!

It means that it takes 4 bytes to represent a pixel, in total. Each pixel in your image has 4 components: R, G, B, and A. Each component is 8 bits, or 1 byte. So 4 components X 1 byte each component = 4 bytes per pixel.

Again, please check your COP cook and display cach limits. You shouldn't be running into crashes unless your total exceeds 1 GB. If they total 1 GB (or less), then it should not be crashing when rendering the sequence. It's important that we check this so we know whether you're encountering a memory leak.

I'll add a few points:

1) You can use the compopts command to set the cook size, so if you're running from hscript and don't want to use much RAM, you can put “compopts -c 10” in your script (use 10MB for the cache). This is just cached memory, so setting it low will not cause your cook to fail. However, this setting is *terrible* for interactive use, so don't save it as the default (I recommend at least 120MB). Also, the display cache is not used when rendering to files.

2) The “Bytes per pixel” returns the entire size of a pixel. So, if you're got a COP with C{r,g,b} (8bit), A (8bit) and Pz (32bit), it would return 8 bpp (3x1+1x1+1x4). The memory usage of a COP isn't xres x yres x bpp, though, as unused planes aren't stored in the cache, and some operations do not use the entire image (and, with large canvas, the image might actually be larger or smaller than xres x yres). Constant tiles are also compressed almost 99%, though all tiles have a little overhead. Cooked results for a COP may get partially bumped out of the cache as well if it gets full. So, instead of estimating the memory usage (which would be very wrong), COPs instead reports the bytes per pixel.

3) The transform code we're using in the Transform COP is the same code that's used to apply transformed, AA'd texture maps in mantra. It has a very nice quality, but loses out in performance to almost every compositing package I know of. That's probably why COPs is taking much longer than nuke.

twod
3) The transform code we're using in the Transform COP is the same code that's used to apply transformed, AA'd texture maps in mantra. It has a very nice quality, but loses out in performance to almost every compositing package I know of. That's probably why COPs is taking much longer than nuke.

If that's the case could the Transform COP have two modes? “Fast & Dirty” and “Slow & Pretty?”

The Fast mode could be used when doing quick test comps, and the set to the high quality mode when its time to do a final render. Or if you are just moving around elements which are going to be blurred you could use the Fast mode.

Wolfwood
If that's the case could the Transform COP have two modes? “Fast & Dirty” and “Slow & Pretty?”

Gets my vote.

I'm all for have COPs work fast. I'd prefer speed over a slight quality hit when interactively comping just as a workflow thing. It makes it more bearable. Then when things are looking good check the niceness box and grab a pint while it renders.

Yep, it's been on my todo list for ages. Never quite gotten around to it, though - always been other more-important things to do.

The fast version would just transform the point, and then do a 4 pixel bilinear lerp on the input. Could also do a nearest pixel lookup, but the speed gain from bilinear to nearest is probably not worth the quality drop.

You could always open up your “list” for voting like vim.

http://www.vim.org/sponsor/vote_results.php [vim.org]

he,he,he,he :twisted:

And the winner is …..

Model SOP? (ducks)

Cheers,

J.C.

JColdrick
Model SOP? (ducks)

heh heh.

I was one of the biggest whingers when they took the model SOP away, but there definitely seems to be some inertia towards reinstating a new improved version of it. But that's another thread/can of worms..

Jerry

Thanks for the bits of explanation, Mark. It's all making more sense now.

Hey Alex, you've never answered my questions throughout this thread ….