HQueue distribute simulation is too slow

   1393   2   1
User Avatar
Member
6 posts
Joined: 6月 2019
Offline
Hello

After a long struggle, I was finally able to set up HQueue. So I tried the distributed simulation.
But the results were slower on three computers than on one.

I'm using Houdini 18.5.408 (Indie) and Slave is using Engine license.

Main PC is 36core Slave PCs are 8core each.

CPU usage during simulation is about 20% each. (Why!

Is there a better setting?

Attachments:
HQPYRO.hiplc (1.4 MB)
hqsimtest.jpg (98.5 KB)

User Avatar
Member
146 posts
Joined: 9月 2011
Offline
Well done on getting HQueue set up!

Re your sim: it'll be a number of factors.

Assuming your render is multithreaded, it's understandable that 3 x 8 core machines will take longer than 1 x 36 core machine even if all else was equal. On top of that, though, there's the network overhead as all the machines are trying to read and write to the server, and there's the overhead of having to spin up the Houdini instances.

Network overhead: if you can afford it, it's worth running a 10GbE network (cost me around £300 for the switch, then £50 each for second hand network cards). It's made a huge difference to renders; I can actually comp footage directly from the server, no need to copy it locally. And sims and geo processing is significantly faster. *Everything* is faster.

Houdini spin-up time can be significant as well; I started with plain old HDDs in the render nodes, and when I eventually replaced them with cheap and small SSDs (after all, you don't need much space) they fire up Houdini incredibly fast now. With rendering (my main farm task) picking the right batch size is a matter of balance; small batches mean lots of time spent loading and unloading Houdini, big batches can leave you waiting for that last one to render while the other machines sit around doing nothing.

I'm digressing.

That 20% CPU usage thing - it could be that it's averaging out the usage, so it may be counting the time the machine's waiting for data to arrive or files to be written. Does the sim max out the cores on your main machine?
User Avatar
Member
6 posts
Joined: 6月 2019
Offline
Thanks reply howiem

Not much info on HQueue, so thanks for the reply!

I'll think about installing a 10Gb environment.
I already have all my PCs on SSD.

We are simulating a 36-core main machine and three 8-core subs simultaneously.

"Does the sim max out the cores on your main machine?"

What do you mean by that? Is there somewhere I can look sim core ?
Is the part of the attached photo important?

It is a test file Pyrosim resolutin0.07, so it is not heavy data.

Attachments:
hqcpu.png (36.9 KB)

  • Quick Links