Hi all,
I'm trying to set up mantra with backburner to get some rendering done over our farm. That's the end goal anyway.
However, to do so I need to create IFDs for Mantra to render, and this process is a really significant bottleneck in the whole process because 1. The IFD creation is very slow and apparently single threaded, and 2. The IFD file size is quite significant.

While I can live with the file sizes, the real killer here is the workflow of having to have one machine create IFDs, which on my current shot takes approximately thirteen hours. Does anyone know why this extremely important step seems to be single threaded? To make this process feasible at all I had to use hbatch to manually open up ten copies of the scene (each eating RAM) and split up the IFD creation between them.

So, is there something I'm missing here? Are there some tools to make this process less painful? This is the first thing in Houdini that has frustrated me

Edit: I don't think I can make the IFD creation any quicker, as the geometry is dynamic, so different on every frame.

13 hrs to submit a sequence of Ifds !!! . Ive worked on some giant shots and never had that, normally IFD creation is fast. Maybe using Backburner as a render manager is the issue. I take it you bought Engine when you acquired Houdini, maybe try using HQ and setting 1 machine on your farm.


rob

Hi rob,
I have no idea about Engine…I have a license for Houdini only (not Houdini FX), but in any case it's locked to one workstation. I can use hbatch to open the hip and export, but it still suffers from the same poor output speed. This is why I had to use hbatch to open about ten copies of the scene and render out simultaneously.

As for backburner, I've yet to get it to work over that, but in principle it's quite simple (just calling mantra.exe ifd_file).

In regard to teh output time, I think I'm just using Houdini for unusual shots. My geometry is constantly changing so the opportunities to use packed disk primitives are quite rare, so IFD generation usually has to evaluate quite a few nodes

I don't know about backburner, but most render managers allow you to have multiple jobs on one machine. This would automate the process you do manually now.

Even if your geometry changes per frame, as soon as you do more than one testrender with the same geo you are wasting time. Cache it out, load as packed disk and then do the rendering. Also cache stuff down the chain of your setup to speed up evaluation of the whole thing.

blackpixel
I don't know about backburner, but most render managers allow you to have multiple jobs on one machine. This would automate the process you do manually now.

Yup, but as I only have the one license, I cannot give that task to a farm node - they can only use IFDs, not create them
I'll have to figure out a way to get the IFDs written as efficiently as possibe, be that caching geo to disk and loading a packed priomitive, or scripting the use of hbatch on my one licensed machine to start multiple copies and generate IFDs

My geometry is constantly changing so the opportunities to use packed disk primitives are quite rare, so IFD generation usually has to evaluate quite a few nodes

If that's case I wonder what will happen to your velocity values > with a changing point count

Rob

circusmonkey

My geometry is constantly changing so the opportunities to use packed disk primitives are quite rare, so IFD generation usually has to evaluate quite a few nodes

If that's case I wonder what will happen to your velocity values > with a changing point count

Rob

They are completely borked.
I either have to use reelsmart motion blur in post and just hope it does a good job, or calculate my own consistent unique id attribute to figure out the vel. The former can mess up completely sometimes and there's nothing you can really do about it, and the latter takes some not insignificant planning.
Fun!

As a test, definitely do what blackpixel recommended. Save the geo out first, load it back in as packed, and save out the ifds. If it takes more than a minute to save out the IFDs, you have some other serious issues.

If it's the geo creation time, are you loading something weird from disc as a source? An obj sequence maybe, (horribly slow)? A good first step is to convert the source to something native to Houdini.

Also make sure that your sop is set to load Packed primitive and not full geometry. This should greatly speed up, as well as reduce file size on the .ifd. I just finished rendering scenes with 150+ million poly's, all packed, some animating and the .ifd size was generally around 35 mbs.

That certainly does reduce the ifd size, yes. However, here's an example of the kind of problems I'm having…
I saved out a bunch of geometry in one geo file that contains groups. It was a particle simulation…sort of that lasted about 250 frames. My final animation requires that simulation to start at about frame 200, making the whole length 450 frames. I also wanted to displace things slightly based on groups, so I cannot load it as a packed primitive. So from what I've learn it seems I have no choice but to write out another sequence of geo files costing another 30gb and time
I've attached an imagine in case my ramblings above were incoherent.

So…was there no way to do this without writing out geo again for every one of the 450 frames?

You are correct, you'll have to write out your geometry as a final step. Note that if you don't, you're still essentially writing out your geo into IFD files anyway so it's not like you're saving disk space. What I would do is use a delayed-load procedural right after the file cache node and your IFD-gen should be very fast because they'll contain links to the geometry on disk rather than containing the geometry itself. I'm not sure what the advantage of using packed primitives are, but delayed load procedural works very well in our pipeline

moogtastic
You are correct, you'll have to write out your geometry as a final step. Note that if you don't, you're still essentially writing out your geo into IFD files anyway so it's not like you're saving disk space. What I would do is use a delayed-load procedural right after the file cache node and your IFD-gen should be very fast because they'll contain links to the geometry on disk rather than containing the geometry itself. I'm not sure what the advantage of using packed primitives are, but delayed load procedural works very well in our pipeline

Hmm, I must have done something wrong as I tried that before. If I check the ‘delay load geometry’ on the file SOP shown in the previous image then the geo is just written to the IFD. Unless that's expected behaviour in that situation?

Edit: I've just looked into delay load procedural and all I can find is this: https://www.sidefx.com/docs/houdini14.0/nodes/shop/vm_geo_file [sidefx.com]
Which appears to be a shader. And now I'm just confused.

That's correct, it's a shader inside a shopnet. We use our own OTLs for this so I don't recall the steps to set it up manually but it looks like the help doc will get you going. One note, if you're applying materials inside your geometry context you'll need to write out a material description file and reference that in your delayed load procedural. You'll use a Mantra Archive node in the output context to save a material description into a single IFD file that contains your shader links

Thanks for the tips Just a few more questions if I may…
So the ‘Delayed Load Geometry’ checkbox in the file SOP isn't anything to do with the delayed load procedural that you're referring to?

Secondly, going back to the IFDs…if I have a mesh, say a complex character for example, and that is rigged with bones and other deformations. In order to render that I have no choice but to save out the same piece of geometry (albeit deformed) for every frame? Or is that precisely what the delayed load geometry does?
I'm having issues getting a proper handle on this as it is completely different from any other application I've used

The Delayed Load Procedural is a different method than the Delayed Load Geometry feature in the file sop, as far as I know. I can't really say if one is better than the other, folks that are a little more well versed in the differences will have to chime in here, but I can say I've rendered billions of point primitives and 9-10gig per-frame pyro cache files on our farm using Delayed Load Procedural so it works very well for us.

Yes, if you're planning on any kind of distributed rendering you'll need to cache your geometry at every frame as a final step in the process. This may sound counter-intuitive if you're used to rendering locally straight out of your work hip, but keep in mind Mantra is essentially generating a render description every time you render a frame anyway. Generating delayed-load geometry caches is a very stable and efficient method because the renderer only needs to load the geometry it needs for the current bucket and it can discard everything else.

Thanks again for the reply. I'll have to poke around a bit and see if I can find out what the difference is. It may be a case that they're essentially the same, but the packed primitives are a relatively new feature.

As for the distributed rendering/geo caching thing, it's a bit of a shock that I'm now going to have to have hundreds of gigs of drive space for something that previously took none D: I understand that Mantra and Houdini are sort of separate entities here. Houdini processes all the inputs and internally creates an IFD, then passes it to Mantra to produce the image. The way other apps I've used work is that there is essentially a version of the application without an interface which performs the distributed rendering, which takes all the inputs (models, animations, deformations, and everything else that was constructed in the app) and processes it again, on the fly, to produce the image. I guess this will take some getting used to

Anti-Distinctlyminty
The way other apps I've used work is that there is essentially a version of the application without an interface which performs the distributed rendering, which takes all the inputs (models, animations, deformations, and everything else that was constructed in the app) and processes it again, on the fly, to produce the image. I guess this will take some getting used to

Well, you can indeed work that way if you wish, you don't *have* to create IFDs first. But everytime you render, you are essentially creating everything at render time. If you are iterating on a render and nothing is changing but a shader parm, then you are re-creating the same data over and over, probably greatly slowing everything down.

The advantage of saving IFDs first is that if done right, (and I know sometimes they can't be done “right”), they are done in seconds, and then you have got a million frames that are able to start up immediately on your farm with no cooking. Also, by separating IFD creation from rendering, you can optimize the right machine for the right task, (multi-threading, memory, CPU, etc) as rendering and geo cooking often have different requirements.