Hi,
I realise this is the PDG forum, but with PDG + Hqueue, Hqueue is really starting to find its feet IMO.
A couple of questions, that might be PDG or might be Hqueue or both:
1.
I have, let's say, 3 computers. 2 of them are roughly the same spec, but one is older with slower CPU and less RAM. All have (for this example) 4 real cores + 4 “hyperthreading” which add a teeny bit of extra processing. For argument I will just say 8 cores.
On each of the “new” computers, I want to run 2x PDG Tasks, i.e. 4 cores per task _except_ on the older one, I want to run a 1x PDG tasks, to compensate for the crappier CPU.
How might I approach this via either PDG or Hqueue? I'm not adverse to modifying the Hqueue config for the slower one, which is what occurred to me first, i.e. “pretend” it's only 4 cores.
The downside with that is that in some cases I want to actually use all the cores on the slower one for low-CPU tasks such as comp or image processing, file copies, etc etc. So in some cases I actually want PDG to be able to access and use all “8” cores.
2. Probably more of an Hqueue question but: Is there any way besides editing the config for a client, to temporarily reduce the available CPUs on a machine. Ideally, while Hqueue is in the midst of running jobs.
Scenario: PDG has set up a bunch of static tasks, and sent them to Hqueue. It's humming along. I then discover that on one machine I want to reduce the available CPUs to only use half of that particular machine.
3. Have you thought about having Hqueue use the more industry-standard concept of Slots? I.e. a Slot can run a single PDG task such as a sim or Mantra render.
The Slots could (and likely would) be different per client, and possibly (ideally) be dynamic as well. So sometimes a slot is a full machine (all CPU and RAM) and other times, there could be 2-4 slots on that same machine, with corresponding adjustment of cores and RAM.
In an even more perfecter world, PDG could adjust these things on the fly depending on what it needed vs what the Hqueue config offered as available resources.
Cheers,
Peter B
Hqueue questions/discussion
1878 3 2-
- pbowmar
- Member
- 7024 posts
- Joined: July 2005
- Offline
-
- chrisgreb
- Member
- 603 posts
- Joined: Sept. 2016
- Offline
pbowmar
I have, let's say, 3 computers. 2 of them are roughly the same spec, but one is older with slower CPU and less RAM. All have (for this example) 4 real cores + 4 “hyperthreading” which add a teeny bit of extra processing. For argument I will just say 8 cores.
On each of the “new” computers, I want to run 2x PDG Tasks, i.e. 4 cores per task _except_ on the older one, I want to run a 1x PDG tasks, to compensate for the crappier CPU.
How might I approach this via either PDG or Hqueue? I'm not adverse to modifying the Hqueue config for the slower one, which is what occurred to me first, i.e. “pretend” it's only 4 cores.
The downside with that is that in some cases I want to actually use all the cores on the slower one for low-CPU tasks such as comp or image processing, file copies, etc etc. So in some cases I actually want PDG to be able to access and use all “8” cores.
You might be able to use the ‘Assign To’ job parm in combination with setting CPUs per Job, to achieve this. For example if the slow machine has 8 hqueue cpus, but is in a client group ‘low-CPU’, you can target it specifically with those tasks and not others.
As well there is the ‘conditions’ in the job spec which can do more logic with groups and hostnames.
2. Probably more of an Hqueue question but: Is there any way besides editing the config for a client, to temporarily reduce the available CPUs on a machine. Ideally, while Hqueue is in the midst of running jobs.I don't see a way to do that. It sounds like a good RFE.
Also note that you can use the tag list to ensure that for example a heavy GPU task will never run with another one on the same machine.
-
- pbowmar
- Member
- 7024 posts
- Joined: July 2005
- Offline
Hi,
Good advice thanks!
Sadly, Hqueue is steadfastly refusing to work for me and I can't figure out why
If I look at the debug info, I run this command in Bash and it fails:
In fact, just trying to run Hython gives me:
Any hints appreciated!
Good advice thanks!
Sadly, Hqueue is steadfastly refusing to work for me and I can't figure out why
The Houdini 17.5.257 environment has been initialized. Can't open dophints.cmd Traceback (most recent call last): File "/mnt/suse42_3/opt/data_root/hqShared/hqShared/pdgtemp/31863/scripts/rop.py", line 35, in <module> import hqueue.houdini as hq ImportError: No module named hqueue.houdini # ---- Running onError ---- # ---- Finished running onError ----
If I look at the debug info, I run this command in Bash and it fails:
export HFS="/opt/software/hfs17.5" && cd $HFS && source ./houdini_setup && "$HFS/bin/hython" "/mnt/suse42_3/opt/data_root/hqShared/hqShared/pdgtemp/31863/scripts/rop.py" The Houdini 17.5.257 environment has been initialized. Can't open dophints.cmd Traceback (most recent call last): File "/mnt/suse42_3/opt/data_root/hqShared/hqShared/pdgtemp/31863/scripts/rop.py", line 35, in <module> import hqueue.houdini as hq ImportError: No module named hqueue.houdini
In fact, just trying to run Hython gives me:
export HFS="/opt/software/hfs17.5" && cd $HFS && source ./houdini_setup && "$HFS/bin/hython" The Houdini 17.5.257 environment has been initialized. Can't open dophints.cmd /opt/software/hfs17.5/python/bin/python: No module named houdiniInterpreter
Any hints appreciated!
Edited by pbowmar - June 9, 2019 13:28:28
-
- pbowmar
- Member
- 7024 posts
- Joined: July 2005
- Offline
-
- Quick Links