Houdini 18.0 Nodes TOP nodes

HQueue Scheduler TOP node

Schedules work items using HQueue.

On this page

This node schedules work items using HQueue in order to execute them on remote machines.

For more information on configuring HQueue, see Getting Started with HQueue or PDG For Design Work Pt. 3 - Setting Up Distributed PDG.

Cook Modes

This scheduler can operate in two different cook modes:

  • The normal cook mode connects to your HQueue scheduler and creates jobs for work items as they become ready to execute, and the jobs then communicate back to the submitting machine with status changes. This means that the submitting Houdini session must remain open for the duration of the cook.

    This mode is used whenever you select Cook from any of the menus or buttons in the TOP UI.

  • The standalone job mode cooks the entire TOP network as a standalone job. In this mode, the submitting Houdini session is detached from the cooking of the TOP network, the .hip file is copied if necessary, and a Hython process executes the TOP network using the default scheduler for that topnet. You will also not see any updates to your current Houdini session. To check the progress of your job when using this mode, you will need to use the HQueue web portal.

    This mode is used whenever you click the Submit Graph As Job > Submit button in the HQueue Scheduler’s parameters.

Network Requirements

As part of the cook, a message queue (MQ) job will be submitted, which is used to communicate information from executing jobs back to the submitting machine. For this reason the farm machines must be able to resolve the hostnames of other farm machines.

Tip

This may be as simple as editing the /etc/hosts (Linux / macOS) or C:\Windows\System32\Drivers\etc\hosts (Windows).

In addition, farm machines must either not have firewalls between them, or you must use the Task Callback Port to specify the open port to use.

When the cook starts, the submitting machine will connect to the farm machine that is running the MQ job, so farm machines must either not have firewalls between themselves and the submitting machine, or you must use the Relay Port to specify the open port.

Warning

Windows Services cannot use mapped drives. Since HQueue jobs on Windows are executed by a Windows Service, you should only use UNC paths. For example, use //myserver/hq/project/myhip.hip instead of H:/project/myhip.hip. Also be careful with backslashes in paths, as they are interpreted as escape sequences when evaluated by Houdini or the command shell.

TOP Attributes

hqueue_jobid

integer

When the scheduler submits a work item to HQueue, it adds this attribute to the work item in order to track the HQueue job ID.

Parameters

Scheduler

These are the global parameters for all work items using this scheduler.

Submit Graph As Job

Submit

Cooks the entire TOP network as a standalone job. Displays the status URI for the submitted job. The submitting Houdini session is detached from the cooking of the TOP network. The .hip file is copied if necessary and a Hython process executes the TOP network normally using the default scheduler for that topnet.

Tip

You can restart a finished standalone jobs using the HQueue Web UI. However, you should restart the child job named TOP Cook instead of the parent job.

Job Name

The name of the top-level HQueue Job for submitted cooks.

Data Layer Server

Enable Server

When on, turns on the data layer server for the TOP job that cooks on the farm. This allows PilotPDG or other WebSocket clients to connect to the cooking job remotely to view the state of PDG.

Server Port

This parameter is only available when Enable Server is on.

Determines which server port to use for the data layer server.

Automatic

A free TCP port selected by the node.

Custom

A custom TCP port specified by the user.

This is useful when there is a firewall between the farm machine and the monitoring machine.

Auto Connect

This parameter is only available when Enable Server is on.

When on, the node will try to send a command to create a remote visualizer when the job starts.

If successful, a remote graph is created and it automatically connects to the server executing the job. The client submitting the job must be visible to the server running the job or the connection will fail.

Working Directory

Specifies the directory where the cook generates intermediate files and output. The intermediate files are placed in a subdirectory named pdgtemp.

If you are opening your .hip file in Houdini from the shared network path (for example, from H:/myproj/myhip.hip), you can use $HIP here (the default). However, if you are opening your .hip file from a local directory (for example, from C:/temp/myhip.hip), you have to copy it to a shared network before it can be accessed by farm machines. In this case, the Working Directory should be an absolute or relative path to that shared network location (for example, //MYPC/Shared/myproj).

Job Name

The name of the job that cooks the TOP network.

Job Description

The description of the top-level HQueue job. This can be seen in the Job Properties for the job.

Override Local Shared Root

When on, the location of the local shared root directory is overridden by the Local Shared Root Paths parameters.

Local Shared Root Paths

These parameters are only available when Override Local Shared Root is on.

The HQueue farm should be configured with a shared network filesystem and the mount point of this shared file system is specified for each platform.

Load from HQueue

Queries the HQueue server to retrieve the local shared root paths for each platform and fills the parameters below.

Windows

The local shared root path on Windows machines. For example, I:/.

macOS

The local shared root path on macOS machines. For example, /Volumes/hq.

Linux

The local shared root path on Linux machines. For example, /mnt/hq.

HQueue Server

URL of the HQueue server. For example, http://localhost:5000.

Executable Paths

Universal HFS

When on, a single path to the $HFS directory (the Houdini install directory) is used by all platforms. You can use $HQROOT and $HQCLIENTARCH to help specify the directory path.

Linux HFS Path

This parameter is only available when Universal HFS is off.

$HFS path for Linux.

macOS HFS Path

This parameter is only available when Universal HFS is off.

$HFS path for macOS.

Windows HFS Path

This parameter is only available when Universal HFS is off.

$HFS path for Windows.

Python

Select which python interpreter should be used for python jobs. This python can be specified in a command using the PDG_PYTHON token.

From HFS: Use the python interpreter that is installed with Houdini.

From HQClient: Use the same python interpreter that HQClient is using on the farm machine.

Custom: Use the executable path specified by Python Executable

Python Executable

The full path to the python executable to use for python jobs.

Message Queue

Create Queue

Determines whether or not Message Queue jobs are created for interactive cooks.

The Message Queue is useful when the submitting Houdini session is not reachable by farm machines due to networking restrictions or firewalls.

Automatically

The Message Queue is created for interactive cooks.

Jobs on the farm communicate their status changes to the Message Queue, which in turn is connected to the submitting Houdini session.

Never

The Message Queue is not created for interactive cooks.

Jobs on the farm communicate their status changes directly back to the submitting Houdini session.

Task Callback Port

Sets the TCP Port used by the Message Queue Server for the XMLRPC callback API. This port must be accessible between farm clients.

Relay Port

Sets the TCP Port used by the Message Queue Server connection between PDG and the client that is running the Message Queue Command. This port must be reachable on farm clients by the PDG/user machine.

Submission Timing

Tick Period

The minimum time in seconds between calls to the onTick callback.

Max Items Per Tick

The maximum number of ready item onSchedule callbacks between ticks.

Job Parms

These job-specific parameters affect all submitted jobs, but can be overridden on a node-by-node basis. For more information, see Scheduler Job Parms / Properties.

Note

Many of these parameters correspond directly to the HQueue Job Properties.

Scheduling

Job Priority

The job’s HQueue priority.

Jobs with higher priorities are scheduled and processed before jobs with lower priorities. 0 is the lowest priority.

Assign To

Which clients to assign priority to.

Any Client

Assign to any client.

Listed Clients

Assign to specified clients.

Clients from Listed Groups

Assign to specified client groups.

Clients

This parameter is only available when Assign To is set to Listed Clients.

Names of clients to assign jobs to separated by spaces.

Select Clients

This parameter is only available when Assign To is set to Listed Clients.

Selects clients from HQueue to populate the Clients list.

Client Groups

This parameter is only available when Assign To is set to Clients from Listed Groups.

Names of client groups to assign jobs to separated by spaces.

Select Groups

This parameter is only available when Assign To is set to Clients from Listed Groups.

Selects client groups from HQueue to populate the Client Groups list.

CPUs per Job

The maximum number of CPUs that will be consumed by the job. If the number exceeds a client machine’s number of free CPUs, then the client machine will not be assigned the job.

Note that you can control the multi-threading of some jobs with Houdini Max Threads. You can also use the Tags parm to control if this job needs a dedicated machine.

Job Description

Description property for the job.

Tags

Space-separated Job Tags, which can be used to indicate if this job requires a dedicated machine.

Allowed Host

The host name of the machine that the job should execute on.

Resources

A space-separated list of HQueue Resources that the job consumes. For example, sidefx.license.render sidefx.license.engine.

Non-Zero Exit Code Handling

Handle By

What to do when the command fails.

Reporting Error

The work item fails.

Reporting Warning

The work item succeeds and a warning is added to the node.

Retrying Task

The work item is retried by HQueue for the number of Retries remaining.

Ignoring Exit Code

The work item succeeds.

Handle All Non Zero

When off, you can specify a particular exit code.

Exit Code

This parameter is only available when Handle All Non Zero is off.

Specifies the exit code that you want to handle using Handle By. All other non-zero exit codes will be treated as a failure as normal.

Retries

This parameter is only available when Handle By is set to Retrying Task.

Number of times to retry the job when the command fails.

Task Environment

Houdini Max Threads

When on, sets the HOUDINI_MAXTHREADS environment to the specified value.

If CPUs per Job is on, then Houdini Max Threads is set to its value. If CPUs per Job is off, then Houdini Max Threads is set to 0 by default, which means that the job will use all available processors.

Positive values limit the number of threads that can be used. For example, a value of 1 disables multi-threading entirely by limiting the job to one thread. Positive values are also clamped to the number of available CPU cores.

If the value is negative, the value is added to the maximum number of processors to determine the threading limit for the job. For example, a value of -1 uses all CPU cores except 1.

See limiting resource usage.

Inherit Local Environment

When on, the environment variables in the current Houdini session are copied into the job’s environment.

Environment Variables

Adds custom key-value environment variables to each task.

See also

TOP nodes