Distributed Fluid Sims Published: Friday, May 29, 2009

Video [115MB QT - 43 Min.]

In this blog, I have posted a video from a recent Toronto technical evening conducted by Senior Developer, Jeff Lait about Distributed Fluid Simulations in Houdini 10. This presentation takes you deep into how Distributed Fluid simulations work and teaches you how to create simulations using a farm of computers.

After learning how to distribute the simulations using HQueue, a python-based job manager, Jeff conducts a manual simulation to show how the underlying pieces work in practice. This will show you how you can add distributed simulations to your in-house job manger and integrate this workflow into your production pipeline.

Distributed Fluid Sims without HQueue

While Side Effects is providing a job management solution with HQueue, we realize that many studios will want to use their existing job management tools. Jeff mentions this in his talk and to make it easier for you to implement, I am adding more detailed notes to this blog:

First Create a Smoke Sim

• Click on Shelf: Smoke Container
• Click on Shelf: Torus
• Click on Shelf: Smoke from Object.
• Select the torus as the source geometry.

Next Up-res the Smoke Sim

• Click on Shelf: Up-Res Fluid Sim
  • Select the cage for the low res smoke
  • A dialog will pop-up. Choose “Same Network”.

Now Slice the Sim and Distribute

• In this example, we will distribute 2 segments (or slices) on a single machine as a simple test.
• Make sure you are on frame 1.
• In your AutoDopNetwork, select the switch_simulationmode node. Then set the Simulate parameter to Both so both sims are present.
• Click on Shelf: Distribute.
  • Select the upressmoke object by selecting it in its dopnetwork.
  • A new DISTRIBUTE_upressmoke_CONTROLS node will be created in AutoDopNetwork. This will be called the control node. The control node is used to slice and distribute the simulation.
  • An Hqueue Simulation ROP will also be created. You can ignore this node.
• Now we need to slice the simulation on say two machines.
  • In the control node, set the Slice Divisions parameter to 2x1x1. This divides the grid in half in X.
  • Note the density cube is now 1/2 the original size - the upresing will only be computed inside there.
  • When we simulate across multiple machines, each machine will simulate a single slice, designated by a slice number. In the control node, the Slice parameter is set to $SLICE. This is done for convenience in case you want to automate a distributed sim (i.e. In your automation, you can simply set the slice number on each machine using a single hscript command, i.e. “setenv SLICE=0”). You can temporarily override the Slice parameter to see other slices. But set Slice back to $SLICE when you are ready to distribute.
• Tracker Process Setup
  • To distribute a simulation, all the machines need to be able to share data with each other. To locate the other machines simulating the slices, each machine contacts the tracker process.
  • To run the tracker process on a machine (say yours), open a console and run:

    cd $HFS/houdini/scripts/python python tracker.py 8000 9000

  • Now that the tracker is running, you can view it in a webbrowser. Go to:

    http://mymachinename:9000

  • This should give you a webpage listing active, barriers, done list, etc. You'll get some spam from the tracker as well once the simulation is running.
• Back in Houdini, go to the control node. Set the tracker name to mymachinename.
• Look inside /obj/distribute_upressmoke. Note that there is an output driver called saveslices. The file name is set by the loadslices node. Note that it has ${SLICE} in the variable name and writes to $HIP/geo
• Save the .hip file as foo.hip
• Setup Houdini sessions to simulate slices
  • Load foo.hip into a new Houdini session. You now have two houdini's active, both at frame 1.
  • In one copy of Houdini, set the control node's Slice parameter to 0, in the other Houdini session, set it to 1. You can do this by deleting the channel and setting it to 0/1, or by using a hscript textport with setenv SLICE=1 and a varchange.
  • Go forward one frame in each Houdini

    Note that each Houdini session will freeze until the other one catches up.

  • Look at the webbrowser. Note that while one Houdini was frozen, you had an active job that moved to the done list when the second Houdini caught up.
  • If you have stale jobs after you close Houdini new sessions may get confused - you have to restart the tracker in this case.

Automating the Distribution

• Close all instances of Houdini. Make sure the tracker process is running with no active jobs.
• Make sure you have a geo subdirectory to store the output
• In the console, create a shell script file invoke.sh with the following contents.

  #!/bin/csh -f
  
  echo Start a specific slice and connect to the tracker.
  
  echo Hipfile: $1
  echo ROP: $2
  echo Slice: $3
  
  setenv ext_SLICE $3
  setenv ext_ROP $2
  
  hscript $1 << "EOF"
  
  setenv SLICE=$ext_SLICE
  varchange
  render $ext_ROP
  quit
  
  "EOF"

Splicing Slices - Putting it all Together

• $HIP/geo now should have slice_${SLICE}_$F.bgeo.gz. We want to render these. Return to our foo.hip houdini session
• Remove the dependence on the simulation. Read from the *.bgeo.gz files instead.
  • Unlink static and source objects from the sim - ie, dive into torus_object1 and set the display to the torus rather than dopimport
  • Turn off display of the smoke output nodes.
  • Alternatively, just hit the Brain Icon to turn off all simulation, but this can easily be forgotten and you may end up trying to send to the render farm something that wants to distribute-sim.
  • Display the distributed_upressmoke node

Additional Notes

• Splitting the Sim
  • You will note that each distributed sim re-simulates the base fluid sim. This is a bad thing as it is wastes cycles and makes it slower to try different variations of the upresing stage.
  • In the dopnetwork, set the switch_simulationmode to Low Res
  • In smokeobject1 go to smokefields. Got to the Export To File tab Set frame range to 1 to 20 for testing
  • Render it - this should be fast and not invoke the upresing.
  • Turn on Load From Disk.
  • In the dopnetwork, set the switch_simulationmode to Up Res
  • Save the foo.hip and you should be able to re-run invoke.sh, this time smidgen faster as we aren't resimulating the base simulation.

It will be wonderful to see distributed fluids being applied by Houdini users in real-world situations. Be sure to touch base and let me know how things are going.

Please login or register to add comments.