Houdini 20.5 HQueue

About HQueue

HQueue is a general-purpose job scheduling system. You can use it to distribute renders, simulations, and other work to remote clients.

On this page

Overview

HQueue is a general-purpose job scheduling system that distributes, monitors and manages tasks across a collection of computing nodes, or client machines. It specializes in managing render and dynamic simulation jobs submitted from Houdini, but you can customize it to work with any job from any application.

Components

The main components in an HQueue farm are the:

  • HQueue Server

    This machine runs the HQueue server process, which is the system’s engine. The server schedules and distributes jobs to the other machines, hosts the web interface, and stores the job scheduling data.

  • HQueue Clients

    These are the machines that run the HQueue client processes. They receive and execute jobs assigned by the server. The HQueue clients are also known as client machines, compute nodes, or render nodes.

  • Workstations

    These are the machines that users submit jobs from using interactive Houdini, non-graphical Houdini or another appliciation. The machines submit jobs to the server and the server distributes the jobs to the clients.

  • Shared Folder Server

    This machine hosts the storage backend for the farm, typically a shared network folder or drive accessible by all machines in the farm. The workstations transfer input files onto the shared folder that are then read by the client machines when executing jobs. The client machines also write the output files onto the shared folder that are then accessed by the workstations.

  • Houdini License Server

    This machine hosts the required licenses to run interactive Houdini on the workstations and non-interactive Houdini, Mantra, and Karma on the client machines.

How It Works

A simple way to describe how HQueue works is to go through a typical workflow – distributing a Mantra render job:

  1. The workstation submits a job to the HQueue server.

    The workflow starts on the workstation. The artist creates a scene in Houdini and when the scene is ready for rendering, the artist uses an HQueue Render ROP to submit a render job to the HQueue server. The HQueue Render ROP also copies the scene (.hip) file and its file dependencies to the shared folder if the files aren’t on the shared folder.

    The artist can now open the HQueue web interface in a web browser to monitor the job’s progress.

  2. The HQueue server assigns the job to a client machine.

    The HQueue server analyzes the job and assigns it to the next available client machine. The client machine contacts the HQueue server and retrieves the job.

  3. The client machine executes the job and breaks the work into smaller tasks.

    The client machine begins executing the job and reads the input files from the shared folder. Specifically for Mantra render jobs, the client reads the .hip file and generates IFD (scene description) files in preparation for the renders. It also creates tasks or child jobs that do the actual rendering work.

    The number of child jobs created is typically based on the number of frames requested to be rendered. For example, the render job is for frames 1 to 240, then 240 child jobs are created with each job instructed to render a single frame from the corresponding IFD file.

  4. The smaller tasks are sent back to the HQueue server and distributed to the rest of the farm.

    The client machine submits the new child jobs back to the HQueue server where they are queued up and assigned to other available client machines on the farm. This is how multiple machines can work on the same render job.

    At the same time, the first client machine continues to generate IFDs for the remaining frames and create new child jobs for those frames.

  5. The smaller tasks are executed.

    The other client machines execute the child jobs in parallel and write the output images to the shared folder.

  6. The job is finished.

    When all tasks are complete, the final rendered images appear in the shared folder. The artist can view the rendered images and re-submit the child jobs for any failed frames. If scene changes are made, then the artist can re-submit the entire render job.

Licensing

The HQueue server does not require a license to operate.

The HQueue clients do require licenses to run Houdini jobs. Houdini simulation jobs and IFD generation jobs require Houdini Engine (or equivalent, i.e. Houdini FX) licenses. Mantra render jobs require Render licenses. Karma render jobs require Karma licenses. All other render jobs that cook .hip files require Houdini Engine (or equivalent) licenses.

HQueue

Getting started

  • About HQueue

    HQueue is a general-purpose job scheduling system. You can use it to distribute renders, simulations, and other work to remote clients.

  • Installation

    How to set up a basic HQueue farm.

  • Configuration

    How to set configuration options for the HQueue server and clients.

  • How to submit jobs

    How to put work on the farm.

Managing the farm

  • Managing jobs

    How to view and manage jobs on the farm.

  • Managing clients

    How to use the web interface or local logins to add, remove, restart, and manage client machines.

  • Managing client groups

    How to use the web interface or local logins to create and manage groups of client machines.

  • Network Folders

    How to use the Network Folder management page.

  • Resources

    You can specify what resources (such as licenses) are available on each client, so jobs can be scheduled on clients where they can run.

  • Notes

    Each client and job can have informational notes attached.

Next steps

  • Logging

    HQueue stores separate logs for server errors and scheduling events, and each client also generates a log.

  • Uninstalling

    How to uninstall the HQueue server or client software.

  • FAQs

    Answers to frequently asked questions.

Guru level