So, finally getting Hqueue going at home, I'm quite impressed with the API and feature set.
Before I invest too much time writing up some more complex dependency dispatching and a PySide UI, has anyone scaled Hqueue up to a farm size with 1000's of clients and workstations?
Obviously you'd need the MySQL database activated on a dedicated server with boatloads of RAM and networking, but the xmlrpc interface worries me a bit from a performance standpoint. Just curious if anyone has really hammered it successfully?
Cheers,
Peter B
Hqueue farm size
3670 5 3- pbowmar
- Member
- 7046 posts
- Joined: July 2005
- Offline
- larsvdb
- Member
- 19 posts
- Joined: March 2010
- Offline
during our test (granted 3 or so years ago) we found it very slow on submissions.
a render with 2000 frames would take several minutes to submit because of the way created json objects to submit to the database. a other issue that a task could not have multiple parents meaning you couldn't do a proper DAG submission.
I hope this is different now, but we decided to go for gridengine.
a render with 2000 frames would take several minutes to submit because of the way created json objects to submit to the database. a other issue that a task could not have multiple parents meaning you couldn't do a proper DAG submission.
I hope this is different now, but we decided to go for gridengine.
- pbowmar
- Member
- 7046 posts
- Joined: July 2005
- Offline
Hey Lars,
Good info, the slow job submission would be killer for sure.
Looking at the API (but not yet trying it) it looks like multiple parents are possible. That would be a dealbreaker for sure of course.
BTW, I notice “parent” and “child” are used backwards in the docs. In all render queues I've ever used, a Parent job must complete before a Child, but the API docs make it sound like the opposite in Hqueue. Perhaps I'm reading it wrong though
Good info, the slow job submission would be killer for sure.
Looking at the API (but not yet trying it) it looks like multiple parents are possible. That would be a dealbreaker for sure of course.
BTW, I notice “parent” and “child” are used backwards in the docs. In all render queues I've ever used, a Parent job must complete before a Child, but the API docs make it sound like the opposite in Hqueue. Perhaps I'm reading it wrong though
Cheers,
Peter Bowmar
____________
Houdini 20.5.262 Win 10 Py 3.11
Peter Bowmar
____________
Houdini 20.5.262 Win 10 Py 3.11
- goldleaf
- Staff
- 4195 posts
- Joined: Sept. 2007
- Offline
- Cetras
- Member
- 41 posts
- Joined:
- Offline
Hi guys,
I used Hqueue at work on a small farm ( 3 nodes only ) and mostly to launch python jobs and I find that yes, with big jobs ( a lot of frames ) it can be quite slow to submit job …
And the parent / child job relationship is quite tricky as well, there is not clear dependence between 2 different jobs, you have to use one job and create children job on that particular job, which is sometimes what you don't want.
As far as I know there is no way you can link 2 jobs with a parent / child relationship easily.
For simple tasks, Hqueue is a perfect tool, with a great python API, for big farms and complex jobs, I wouldn't go for it …
I used Hqueue at work on a small farm ( 3 nodes only ) and mostly to launch python jobs and I find that yes, with big jobs ( a lot of frames ) it can be quite slow to submit job …
And the parent / child job relationship is quite tricky as well, there is not clear dependence between 2 different jobs, you have to use one job and create children job on that particular job, which is sometimes what you don't want.
As far as I know there is no way you can link 2 jobs with a parent / child relationship easily.
For simple tasks, Hqueue is a perfect tool, with a great python API, for big farms and complex jobs, I wouldn't go for it …
My portfolio : www.guillaume-j.com
- drew
- Member
- 120 posts
- Joined: July 2005
- Offline
-
- Quick Links