hello forum,

I am building a pipeline for next years major project in Uni (ncca). I am interested about network rending as with have 3 labs with almost 200 computer in the same network with Houdini license.

The idea is to save out each scenes as IDF sequence and then using a python script, render each of these sequence. I did some test and look much faster than rendering in Houdini. The I was think that using SSH I could connect to the other computers and spilt the renderings.

I like to know props and cons from people who have already did that. Other option maybe to make the process faster.

Also I am not sure if I have to assign to each computer a small different sequences or I can assign the bigger sequence to many computers and let them decide which frame they gonna be render. The last looks more efficient but I don't know how to do this.

The OS in all the computer is Linux.

Any help will be really appreciated.
Christos

Hey Christos,

when I was there I wrote a script that does what you describe.
- it was a modified version of a script originally written by Lars Vanderbijl
I have no idea if my user account as demonstrator still exists, but it might still be in my bin folder…

It basically sends commands to different machines through threads in python. That way it could launch Maya renders as well as houdini renders (through ifd or simply from batch). I've also run houdini over the network to do high res fluid sims. It worked with renderman as well.

Every year somebody reinvents the wheel it seems… Good luck with it. I don't have the files myself anymore, but like I said perhaps they are still floating around.

ssh would be the way if you were planning to do it just through shell scripting,

Don't underestimate the amount of space and time it can take to generate the ifds. I was dealing with massive particle simulations and my datauseage at the time got me using 10 extra accounts of classmates that were doing smaller projects. It is faster to render ifd's, but because of the data restrictions it might help you to render straight through hrender/hbatch.

Thanks peter for your reply

The idea about using network rendering with ssh came after reading your major project report about the Jam.

We render out ifd's and each spawns it's own process. You're quite correct that some renders the ifd generation process can easily exceed the render time, and I/O is brutal. We just make sure there's a mechanism in there for splitting up the hbatch processes. It's still handy to keep the ifd tasks separate from the hbatch…

Cheers,

J.C.

When I was there, I installed Gridengine on all the machines and used that

Jon Macey had a R+D render queue based on Python, not sure how that went. Houdini's Hqueue is Python based too, you might be able to use that?

I don't recommend using SSH for that type of thing, the overhead is kind of killer.

Basically, unless your actual project at school is writing/doing a render queue, don't waste time doing it yourself, it can be a rabbit hole.

Cheers,

Peter B

thanks guys for your replies, but now I am more confused than before. I have not match experience with remote rendering in Houdini.

I started two days ago reading everything online related to that. All I got is how to generate .idf sequences for each render. Then I wrote a simple script in python which takes all the idf files in a directory and does the rendering, work in windows and linux, and you can set a frame range. I did some test but with one machine, not network rendering.

In Houdini's documentation says that you can start a remote render on any host computer on the network that has the hserver license administration server installed. But Im not sure how thats works.
http://www.sidefx.com/docs/houdini11.0/rendering/network [sidefx.com]

@pbowmar I am gonna sent a email to Macey about his render queue.
About Houdini's Hqueue, I had a discussion for simulation with Macey during this summer and his words were:
It is unlikly we will have HQueue not year as it is a standalone system which will not fit into the existing infrastructure however the farm should be able to handle simulations once it is up and running again.

Again thanks all of you about your responds, I am gonna do little more research and I will be back with more specific question.

I wised that this kind of things could be more easy, though I like to learn.
But as I am not gonna be r+d, I hope to find a solution soon in order to concentrate to the effects of the project.

cheers,
Christos

i9089303
In Houdini's documentation says that you can start a remote render on any host computer on the network that has the hserver license administration server installed. But Im not sure how thats works.

Well, hserver and Houdini/mantra installed. This is network rendering though as also mentioned here:
http://www.sidefx.com/docs/houdini11.0/rendering/batchrendering [sidefx.com]

See also
http://odforce.net/wiki/index.php/MantraRemoteRendering [odforce.net]

I assumed this was an educational experience. Peter is correct, it's a rabbit hole…*huge* big topic. We use SGE here for our solution, there are other solutions out there of course and it sounds like you already have something in place. The guts of scheduling, resources, rules of engagement - that's the hard stuff if done correctly. Personally, I have zero interest in re-inventing that particular wheel. Wrapping a particular software product into such a system, that's typically more straightforward, that's just implementation and a bunch of work.

I'm unclear which exactly you're trying to do here. Whatever you do, don't go the route of ‘mantra -H foobar’ - that particular feature is very limited and quite old, it won't give you the flexibility you need. There's plenty of flexibility with Houdini to fit it into pretty much any scheduling system out there - it spits out discrete chunks, you can edit the process of writing it out plus pre-process it before running, all quite trivially. Houdini is designed for maximum scriptability.

Cheers,

J.C.

edward wrote:

Well, hserver and Houdini/mantra installed. This is network rendering though as also mentioned here:
http://www.sidefx.com/docs/houdini11.0/rendering/batchrendering [sidefx.com]

See also
http://odforce.net/wiki/index.php/MantraRemoteRendering [odforce.net]

Thanks edward for the link, the odforce one I have already read it.

JColdrick wrote:

Personally, I have zero interest in re-inventing that particular wheel.

I absolutely agree with you on that, I just want to do my job fast and as efficient as possible. I had a talk with my tutor and seems that next year will have a renderfarm which can I use, so I guess one problem is solves.
But in general, I like learn stuff and how things work, so I am gonna try few things and I am gonna let you know if I have something cool.

Thanks again all of you who tried to help in any way, I hope the topic be filled with more information about this subject which is quit important and not so well documented.

cheers
Christos