Houdini 19.0 HQueue

Installation

How to set up a basic HQueue farm.

On this page

Before you begin

Note

This guide assumes the server and client machines all share the same operating system (Linux, macOS, or Windows). It is possible to set up an HQueue farm with mixed operating systems, but that is not covered in this quick guide.

The following items are required before setting up an HQueue farm:

Network connectivity

  • Machines must be able to “see” or ping each other by their domain (DNS) names or by their IP addresseses.

  • Client machines must be able to connect to the server machine through port 5000. The server machine must be able to connect to client machines through port 5001. This may require opening ports in any local firewalls.

Shared network folder

  • The servers and clients must be able to access a shared network filesystem, for example an NFS mount of SMB share.

  • The shared folder must be mounted at the same location on every machine, for example /nfs/storage/ on Linux, or \\bigdisk\hq on Windows.

Houdini

  • Install Houdini on every client machine in the same install directory.

    Mac

    On macOS, the default Houdini installation path is /Applications/Houdini/HoudiniX.Y.ZZZ.

    Windows

    On Windows, the default Houdini installation path is C:/Program Files/Side Effects Software/Houdini X.Y.ZZZ.

    Linux

    On Linux, the default Houdini installation path is /opt/hfsX.Y.ZZZ.

  • Houdini licensing must be set up correctly on each client machine as well, so the client software can launch licensed executables such as hython and mantra.

Installing HQueue

Setting up an HQueue farm involves installing the HQueue server, registering a network folder with HQueue and adding client machines to the farm.

Install the HQueue Server

  1. Choose a machine, say serverMachine, to run the HQueue Server.

  2. On the machine, download and run the Houdini Launcher.

  3. In the Launcher, select HQueue on the left-hand side and then click on the Install HQueue button on the right.

  4. When prompted to choose a version, select Choose Another Version.

  5. Select a version that has a py2 Build Type (i.e. Python 2) and then click Install.

    Note

    A Python 3 version of the HQueue Server is currently not available. You can submit jobs from a Houdini Python 3 build and run Houdini Python 3 in jobs on an HQueue Python 2 farm.

  6. In the HQueue Install Options dialog, check on the HQueue Server component and uncheck the HQueue Client component.

  7. Complete the installation in the Launcher.

    Mac

    On macOS, the Launcher installs the server to /Library/HQServer and creates a new com.sidefx.hqserver.plist service file in /Library/LaunchDaemons.

    Windows

    On Windows, the Launcher installs the server to C:/HQueueServer and creates a new HQueueServer service in Windows Services.

    Linux

    On Linux, the Launcher installs the server to /opt/hqueue.

  8. You can open a web browser and navigate to http://serverMachine:5000 to confirm that the HQueue server is running.

    Note

    If DNS is not configured for serverMachine then you can connect to HQueue using the server machine’s IP address.

    For example, http://X.X.X.X:5000.

Register the network folder with HQueue

  1. Open a web browser and go to the HQueue web interface. For example, http://serverMachine:5000.

  2. Hover the mouse over the “triple bars” icon in the top-right corner and then select Network Folders.

  3. In the Network Folder 1 box, set the fields that apply to the desired operating system.

    Mac

    For macOS, set the on macOS field to where the network folder is mounted on macOS machines (i.e. /Users/Shared/myShare). Leave all other fields at their default values.

    Windows

    For Windows, set the Host field to the machine name or IP address hosting the network folder. Then set the on Windows field to the network folder’s share name and finally set the Map to field to the drive letter that the network folder is mapped to on Windows machines.

    For example, suppose your network folder is served on a machine named myFileServer, its share name is myShare, and your machines map X: to \\myFileServer\myShare, then set Host to myFileServer, set on Windows to myShare and set Map to to X:.

    Leave all other fields at their default values.

    Linux

    For Linux, set the on Linux field to where the network folder is mounted on Linux machines (i.e. /mnt/myShare). Leave all other fields at their default values.

  4. Save the changes.

    Mac

    References to $HQROOT in HQueue jobs and Houdini parameters will now resolve to the value of the on macOS field on macOS machines.

    Windows

    References to $HQROOT in HQueue jobs and Houdini parameters will now resolve to either the drive letter specified by the “Map to” field or the UNC path produced by the Host and “on Windows” fields on Windows.

    Linux

    References to $HQROOT in HQueue jobs and Houdini parameters will now resolve to the value of the on Linux field on Linux machines.

Add a client machine

  1. Run the Houdini Launcher on the machine that you want to add.

  2. In the Launcher, select HQueue on the left-hand side and then click on the Install HQueue button on the right.

  3. When prompted to choose a version, select Choose Another Version.

  4. Select a version that has a py2 Build Type (i.e. Python 2) and then click Install.

    Note

    A Python 3 version of the HQueue Client is currently not available. You can submit jobs from a Houdini Python 3 build and run Houdini Python 3 in jobs on an HQueue Python 2 farm.

  5. In the HQueue Install Options dialog, check on the HQueue Client component and uncheck the HQueue Server component. Then set the HQueue Server URL field to point to your HQueue Server. For example, http://serverMachine:5000.

  6. (optional, Linux and macOS only) Change the user account that runs the HQueue Client and jobs by changing the Client User field. By default, the Launcher creates a new hquser account to run the HQueue Client.

  7. (optional) Change the HQueue Client install location by changing the Install Directory field. This is required on Linux if you also changed the user account. By default, the Launcher installs the HQueue Client into the hquser's home directory on Linux.

  8. Complete the installation in the Launcher.

    Mac

    On macOS, the Launcher installs the HQueue Client to /Library/HQClient and creates a new com.sidefx.hqclient.plist service file in /Library/LaunchDaemons.

    Windows

    On Windows, the Launcher installs the HQueue Client to C:/HQueueClient and creates a new HQueueClient service in Windows Services.

    Linux

    On Linux, the Launcher installs the HQueue Client to the client user’s home directory, $HOME/hqclient. For example, the HQueue Client is installed to /home/hquser/hqclient by default.

  9. Go to the Clients page in the HQueue web interface to confirm that the machine was successfully added to the farm.

Note

Repeat these steps for each additional client machine you want to add to the farm.

Note

For macOS, if adding the machine failed with a “Connection Refused” error then check out this FAQ.

Note

For Windows, the HQueue Client by default runs with a generic Windows network service account. The account has limited network access and may not have permission to access your network folder.

It is recommended that the HQueue Client be changed so that is run with a real Windows user account to avoid possible file permission errors in jobs. To change the HQueue Client’s user account, please follow this FAQ.

Testing the Farm

To test that the HQueue farm is configured correctly you can submit a simple Houdini render job.

Here’s how:

  1. Launch Houdini.

  2. Create a simple scene. At a minimum, put down some geometry, a light, a camera and a Mantra ROP for rendering.

  3. Navigate to the /out node network and put down an HQueue Render ROP.

  4. In the Parameters pane, set the Output Driver parameter to your Mantra ROP.

  5. Point the HQueue Server parameter to your HQueue farm. For example, http://serverMachine:5000.

    Note

    If DNS is not configured for serverMachine then you can alternatively use the server machine’s IP address in the URL. For example, http://X.X.X.X:5000.

  6. Then set Target HFS to where Houdini is installed on the client machines.

    Mac

    For macOS, set Target HFS to:

    /Applications/Houdini/HoudiniX.Y.ZZZ/Frameworks/Houdini.framework/Versions/Current/Resources

    Windows

    For Windows, set Target HFS to:

    C:/Program Files/Side Effects Software/Houdini X.Y.ZZZ

    Linux

    For Linux, set Target HFS to:

    /opt/hfsX.Y.ZZZ

  7. Save the .hip file somewhere in your network folder.

    Mac

    On macOS, the network folder is defined by the hqserver.sharedNetwork.mount.macosx key in the HQueue Server configuration file. For example, save the .hip file to /Volumes/myShare/test.hip.

    Windows

    On Windows, the network folder is defined by the hqserver.sharedNetwork.mount.windows key in the HQueue Server configuration file. For example, save the .hip file to X:/test.hip.

    Linux

    On Linux, the network folder is defined by the hqserver.sharedNetwork.mount.linux key in the HQueue Server configuration file. For example, save the .hip file to /mnt/myShare/test.hip.

  8. Click Submit Job at the top of the Parameters pane.

    • A new job will appear in the HQueue web interface.

    • You can click on the job’s id to dive into the job and see more details.

Troubleshooting

When a job fails, you can find out why by clicking on the job’s id in the HQueue web interface. This will take you inside of the job where you can see more details. If any of the job’s Child Jobs failed then click on the child job’s id to further dive inside. Eventually you will see an Output Log field with the job’s output. The output will contain an error message indicating why the job failed.

Check out the Troubleshooting FAQs for information on common job failure errors and possible solutions.

Upgrading the HQueue Server

Before upgrading the HQueue Server, create backup copies of the HQueue Server database and configuratiion file.

Mac

On macOS, the HQueue Server database is located at /Library/HQServer/db/hqserver.db and the configuration file is located at /Library/HQServer/hqserver.ini.

Windows

On Windows, the HQueue Server database is located at C:/HQueueServer/db/hqserver.db and the configuration file is located at C:/HQueueServer/hqserver.ini.

Linux

On Linux, the HQueue Server database is located at /opt/hqueue/db/hqserver.db and the configuration file is located at /opt/hqueue/hqserver.ini.

To upgrade an HQueue Server previously installed by the old Houdini installers, perform a fresh HQueue Server installation with the Houdini Launcher. See Install the HQueue Server.

To upgrade an HQueue Server previously installed by the Launcher:

  1. In the Launcher, select HQueue on the left-hand side.

  2. Click on the gear icon menu button next to HQueue Server and then select Modify.

  3. In the Modifications dialog, change the Version drop-down menu to the desired version.

    Note

    Select a py2 (i.e. Python 2) version because an HQueue Server Python 3 version is currently not available.

  4. Click Modify to complete the upgrade.

Upgrading the HQueue Client

Before upgrading the HQueue Client, create a backup copy of the HQueue Client configuratiion file.

Mac

On macOS, the HQueue Client configuration file is located at /Library/HQClient/hqnode.ini.

Windows

On Windows, the HQueue Client configuration file is located at C:/HQueueClient/hqnode.ini.

Linux

On Linux, the HQueue Client configuration file is located at /home/hquser/hqclient/hqnode.ini.

To upgrade an HQueue Client previously installed by the old Houdini installers , perform a fresh HQueue Client installation with the Houdini Launcher. See Add a client machine.

To upgrade an HQueue Client previously installed by the Launcher:

  1. In the Launcher, select HQueue on the left-hand side.

  2. Click on the gear icon menu button next to HQueue Client and then select Modify.

  3. In the Modifications dialog, change the Version drop-down menu to the desired version.

    Note

    Select a py2 (i.e. Python 2) version because an HQueue Client Python 3 version is currently not available.

  4. Click Modify to complete the upgrade.

Scripted Installations

The Houdini Launcher supports scripted, or silent, installations of the HQueue Server and HQueue Client. The Houdini Launcher installation can also be scripted, see Scripting Launcher installations.

To install the HQueue Server from the command line:

Mac

Open a Terminal app and run:

cd /Applications/Houdini Launcher.app/Contents/MacOS
sudo ./houdini_installer install --product "HQueue Server" --version X.Y.ZZZ --settings-file /path/to/launcher_settings.ini --build-option py2 --accept-EULA=SideFX-20XX-XX-XX

Windows

Open a Windows command prompt and run:

cd C:/Program Files/Side Effects Software/Launcher
houdini_installer.exe install --product "HQueue Server" --version X.Y.ZZZ --settings-file C:/path/to/launcher_settings.ini --build-option py2 --accept-EULA=SideFX-20XX-XX-XX

Linux

Open a terminal and run:

cd /opt/sidefx/launcher/bin
sudo ./houdini_installer install --product "HQueue Server" --version X.Y.ZZZ --settings-file /path/to/launcher_settings.ini --build-option py2 --accept-EULA=SideFX-20XX-XX-XX

To install the HQueue Client from the command line:

Mac

Open a Terminal app and run:

cd /Applications/Houdini Launcher.app/Contents/MacOS
sudo ./houdini_installer install --product "HQueue Client" --version X.Y.ZZZ --settings-file /path/to/launcher_settings.ini --build-option py2 --accept-EULA=SideFX-20XX-XX-XX

Windows

Open a Windows command prompt and run:

cd C:/Program Files/Side Effects Software/Launcher
houdini_installer.exe install --product "HQueue Client" --version X.Y.ZZZ --settings-file C:/path/to/launcher_settings.ini --build-option py2 --accept-EULA=SideFX-20XX-XX-XX

Linux

Open a terminal and run:

cd /opt/sidefx/launcher/bin
sudo ./houdini_installer install --product "HQueue Client" --version X.Y.ZZZ --settings-file /path/to/launcher_settings.ini --build-option py2 --accept-EULA=SideFX-20XX-XX-XX

See Houdini Launcher - Scripting installations for more information on command line installations using the Houdini Launcher.

Optional: hquser account

On client machines you can optionally set up an hquser user and hqgroup user group to run Hqueue as. By convention the user and group ID are both 601, and the user’s password is hqpass, but you can change these.

Linux

Your Linux distribution may come with a user interface for creating user accounts, otherwise use the adduser command.

Mac

You can create users and groups in the Users and Groups panel of System Preferences.

Having consistent user accounts across clients has benefits:

  • Don’t have to keep track of different user accounts on different machines when you need to log into a particular client.

  • All files created by the client software are owned by a single consistent user.

Once you have created the user and group, you should start the HQueue server/client as hquser.

HQueue

Getting started

  • About HQueue

    HQueue is a general-purpose job management system. You can use it to distribute renders, simulations, and other work to remote clients.

  • Installation

    How to set up a basic HQueue farm.

  • Configuration

    How to set configuration options for the HQueue server and clients.

  • How to submit jobs

    How to put work on the farm.

Managing the farm

  • Managing clients

    How to use the web interface or local logins to add, remove, restart, and manage client machines.

  • Managing jobs

    How to view and manage jobs on the farm.

  • Managing client groups

    How to use the web interface or local logins to create and manage groups of client machines.

  • Notes

    Each client and job can have informational notes attached.

  • Resources

    You can specify what resources (such as licenses) are available on each client, so jobs can be scheduled on clients where they can run.

  • Network Folders

    How to use the Network Folder management page.

Next steps

  • Logging

    Hqueue stores separate logs for errors and scheduling events, and each client also generates a log.

  • Uninstalling

    How to uninstall the HQueue server or client software.

  • FAQs

    Answers to frequently asked questions.

Guru level

  • Remote API

    You can connect to the HQueue server over the network and call functions to manipulate and query jobs and other information.

  • Jobs specification details

    More detail on the internals of a job specification for users who want to submit custom jobs.

  • Switching to MySQL (Linux only)

    You can optionally configure HQueue to use MySQL instead of SQLite on Linux.