Hqueue farm size

   3670   5   3
User Avatar
Member
7046 posts
Joined: July 2005
Offline
So, finally getting Hqueue going at home, I'm quite impressed with the API and feature set.

Before I invest too much time writing up some more complex dependency dispatching and a PySide UI, has anyone scaled Hqueue up to a farm size with 1000's of clients and workstations?

Obviously you'd need the MySQL database activated on a dedicated server with boatloads of RAM and networking, but the xmlrpc interface worries me a bit from a performance standpoint. Just curious if anyone has really hammered it successfully?

Cheers,

Peter B
Cheers,

Peter Bowmar
____________
Houdini 20.5.262 Win 10 Py 3.11
User Avatar
Member
19 posts
Joined: March 2010
Offline
during our test (granted 3 or so years ago) we found it very slow on submissions.

a render with 2000 frames would take several minutes to submit because of the way created json objects to submit to the database. a other issue that a task could not have multiple parents meaning you couldn't do a proper DAG submission.

I hope this is different now, but we decided to go for gridengine.
User Avatar
Member
7046 posts
Joined: July 2005
Offline
Hey Lars,

Good info, the slow job submission would be killer for sure.

Looking at the API (but not yet trying it) it looks like multiple parents are possible. That would be a dealbreaker for sure of course.

BTW, I notice “parent” and “child” are used backwards in the docs. In all render queues I've ever used, a Parent job must complete before a Child, but the API docs make it sound like the opposite in Hqueue. Perhaps I'm reading it wrong though
Cheers,

Peter Bowmar
____________
Houdini 20.5.262 Win 10 Py 3.11
User Avatar
Staff
4195 posts
Joined: Sept. 2007
Offline
pbowmar
BTW, I notice “parent” and “child” are used backwards in the docs. In all render queues I've ever used, a Parent job must complete before a Child, but the API docs make it sound like the opposite in Hqueue. Perhaps I'm reading it wrong though

I too found it backwards
I'm o.d.d.
User Avatar
Member
41 posts
Joined:
Offline
Hi guys,

I used Hqueue at work on a small farm ( 3 nodes only ) and mostly to launch python jobs and I find that yes, with big jobs ( a lot of frames ) it can be quite slow to submit job …

And the parent / child job relationship is quite tricky as well, there is not clear dependence between 2 different jobs, you have to use one job and create children job on that particular job, which is sometimes what you don't want.
As far as I know there is no way you can link 2 jobs with a parent / child relationship easily.

For simple tasks, Hqueue is a perfect tool, with a great python API, for big farms and complex jobs, I wouldn't go for it …
My portfolio : www.guillaume-j.com
User Avatar
Member
120 posts
Joined: July 2005
Offline
Cetras
For simple tasks, Hqueue is a perfect tool, with a great python API, for big farms and complex jobs, I wouldn't go for it …

Out of interest, what would you go for ? What are people using these days ?


Thanks,
Drew
  • Quick Links