12 Python API

HQueue comes with an API that allows you to submit and query jobs on the server using Python. To access the API, you first need to connect to the HQueue server. To do this, use the xmlrpclib module:

import xmlrpclib

# Connect to the HQueue server.
hq_server = xmlrpclib.ServerProxy( "http://hq_server_hostname:5000")

In the example above, a connection to the server is made a proxy reference is stored in the hq_server variable. Using the proxy reference, you can invoke any of the methods described below.



addNote(note_text, thread_id, job_id, client_id)

Adds a note to a thread or creates a new thread if there is no corresponding thread to either thread_id, job_id or client_id. Only one of job_id and client_id may be not nil. If one of those two values are not nil then they are used to determine the id of the thread to either edit or add a note to. Otherwise thread_id is used.

getAllJobStatusNames()

Return a tuple containing all the possible job statuses.

getClient(client_id [, attribs])

Convenience function for getClients(). Return a dictionary of attributes for the specified client.

getClientByHostname(hostname [, attribs])

Convenience function for getClients(). Return a dictionary of attributes for the specified client.

getClientGroups([group_ids [, attribs]])

Return a list of dictionaries where each dictionary contains the attributes for a client group specified in group_ids. group_ids must be a list of group id numbers. If it is not specified or is set to None, then attributes for all registered client groups are returned.

You can optionally set the attribs argument to a list of client group attributes that will be returned. This can help reduce the execution time of the function and also reduce the amount of data returned.

Here are the list of attributes:

  • clients   - list of ids for clients that are members of the group.
  • id   - group id
  • name   - group name

getClients([client_ids [, attribs]])

Return a list of dictionaries where each dictionary contains the attributes for a client specified in client_ids. client_ids must be a list of client id numbers. If it is not specified or is set to None, then attributes for all registered clients are returned.

You can optionally set the attribs argument to a list of client attributes that will be returned. This can help reduce the execution time of the function and also reduce the amount of data returned.

Here are the list of attributes:

  • available   - whether the client is available to process jobs at the current time
  • availabilityRules   - list of rules outlining the client's availability on the farm
  • cpus   - number of cpus
  • cpuSpeed   - processor speed
  • disabled   - whether the client has been disabled on the farm
  • groups   - list of group ids that the client is a member of
  • hostname   - client machine name
  • id   - client id
  • idleTime   - number of minutes that the client has been idle (no activity from the logged-in user). Only applies to workstations. Is set to -1 if the idle time is unknown.
  • ip   - client's IP address
  • lastHeartbeat   - time when the client last contacted the server. The time is in UTC.
  • latestNote   - the text of the latest note from the thread corresponding to the client. An empty string is returned if there is no corresponding thread.
  • load   - client load
  • memory   - amount of physical memory in kilobytes
  • needsUpgrade    - whether the client's code is out-of-date and requires an update
  • platform   - client's operating system and architecture
  • runningJobs   - ids for the jobs currently running on the client
  • threadId   - the id of the note thread corresponding to the client. null is returned if there is no corresponding thread.
  • version   - hash string representing the version of the HQueue code installed on the client

getClientsByHostname([hostnames [, attribs]])

Convenience function for getClients(). Return a list of dictionaries where each dictionary contains the attributes for a client specified in hostnames. hostnames must be a list of client hostnames. If it is not specified or is set to None, then attributes for all registered clients are returned.

getclientinfo(hostname)
Deprecated. Use either getClient(), getClients(), getClientByHostname() or getClientsByHostname() instead.

Return a dictionary of information for the given client machine. The keys in the dictionary are "disabled" and "jobs". The "disabled" value indicates if the client machine has been disabled. The "jobs" value is a list of ids for jobs currently running on the client machine. Return None if the specified hostname is not registered on the farm.

getConfig(key)

Returns the configuration value for the given key or None if the key does not exists in the configuration file.

getFailedJobStatusNames()

Return a tuple containing the names of what is considered a failed job status.

getFinishedJobStatusNames()

Return a tuple containing the names of what is considered a finished job status.

getFinishedRootJobIds()

Return a list of job ids for all finished, root jobs. Root jobs are ones that have no parent.

getHQRoot(platform , is_client = True)

Return the value of the $HQROOT environment variable as it would appear on a machine of the given platform. platform must be either "linux", "windows" or "macosx". Return None if platform is none of those values. Returns the UNC path if platform is "windows" and is_client is True.

getinfo(id)
Deprecated. Use either getJob() or getJobs() instead.

Return a dictionary of information for the given job. The keys in the dictionary are "startTime", "endTime", "cmdStartTime", "cmdEndTime", "status", "progress", "root", "children", "parents". "startTime", "endTime", "status", "priority", "progress" and "conditions". The "root" value contains the id of the top-most ancestor (i.e. job with no parent) of the specified job. Return None if id is not a valid job id.

getJob(job_id [, attribs])

Convenience function for getJobs(). Return a dictionary of attributes for the specified job.

getJobHistory(client_id [, limit])

Return a list of job ids that recently ran on the specified client. The jobs are ordered by end times with the job that last finished on the client appearing first in the list. You can optionally specify the limit argument to cap the maximum number of jobs that are returned. The default limit value is 5.

getJobIdsBySatus(statuses)

Returns a list of ids of the jobs that have a status in the list of given statuses. Valid status names are "abandoned", "cancelled", "failed", "succeeded", "paused", "pausing", "queued", "resuming", "running", "runnable", and "waiting".

getJobOutput(job_id [, max_last_chars])

Return the output log of the given job. If max_last_chars is specified, then return at most the last so many characters of the log. If max_last_chars is less than one or is not specified, then the entire output log is returned.

getJobs(job_ids [, attribs])

Return a list of dictionaries where each dictionary contains the attributes for a job specified in job_ids. job_ids must be a list of job id numbers.

You can optionally set the attribs argument to a list of job attributes that will be returned. This can help reduce the execution time of the function and also reduce the amount of data returned.

Here are the list of attributes:

  • availabilityRules   - list of rules outlining the client's availability on the farm
  • children   - list of child job ids
  • clients   - list of clients that ran or are currently running the job
  • clientsOnHold   - list of clients that are currently on hold (or reserved) for the job. Clients can be on-hold for, but not officially assigned to, a job if the job requires multiple clients to execute but only some of the clients are available at the moment. Once the job has enough clients on-hold, HQueue takes those clients and assigns them to the job.
  • conditions   - list of scheduling conditions imposed on the job
  • endTime   - time when the job ended. The time is in UTC. Is set to None if the job has not finished.
  • ETA   - estimated number of seconds before the job is finished. Is set to -1 if the ETA cannot be determined. Note that calculation of this attribute requires HQueue to query the job and all of its descendents which can significantly slow down the execution of this function.
  • id   - job id
  • logs   - list of entries for the job's status change log
  • name   - job name
  • latestNote    - the text of the latest note of the thread corresponding to the job. An empty string is returned if there is no corresponding thread.
  • overallProgress   - overall progress percentage (between 0.0 and 1.0) of the job and its descendents. Note that calculation of this attribute requires HQueue to query the job and all of its descendents which can significantly slow down the execution of this function.
  • parents   - list of parent job ids
  • priority   - job priority
  • progress   - progress percentage (between 0.0 and 1.0) of the job command. If the job has no command to execute, then the progress will always be set to 0.0.
  • queueTime   - time when the job was submitted, or queued, in HQueue. The time is in UTC.
  • startTime   - time when the job or one of its descedents started. The time is in UTC.
  • status   - internal job status. HQueue queries only the job itself (and not its descendents) so requesting this attribute is extremely fast. If you are interested only in whether a job is finished or not, then request for this attribute instead of uiStatus. Possible values of this attribute are "abandoned", "cancelled", "failed", "paused", "pausing", "resuming", "running", "succeeded", "runnable" and "queued". "runnable" is the equivalent of the "waiting for machine" UI status. "queued" indicates that the job is waiting for descedent jobs to finish before it can run.
  • threadId   - the id of the thread corresponding to the job. null is returned if there is no corresponding thread.
  • tags   - list of tags, or custom properties, attached to the job. Of note, the 'single' tag indicates that the job can only run on a machine that has no other jobs running on it.
  • uiStatus   - job status which appears in the HQueue web interface. A list of the possible UI job statuses is listed at 8.7 Job Statuses . Note that this attribute requires HQueue to query the job and all of its descendents which can significantly slow down the execution of this function.

getLatestNotes([thread_ids , attribs])

Return a list of dictionaries where each dictionary contains the attributes for the latest note of a thread specified in thread_ids. thread_ids must be a list of thread id numbers or null. If it is null then the latest notes for all of the jobs are returned.

You can optionally set the attribs argument to a list of note attributes that will be returned. This can help reduce the execution time of the function and also reduce the amount of data returned.

Here are the list of attributes:

  • clientHostname   - The client hostname associated with the thread.
  • clientId   - The id of the client associated with the thread.
  • date   - The date and time that the note was last added.
  • jobId   - The id of the job associated with the thread.
  • text   - The contents of the note.
getNoteThreadIds()

Return a list of the all the thread ids.

getNoteThreads([thread_ids, thread_attribs, note_attribs)

Return a list of dictionaries where each dictionary contains the attributes for a thread specified in thread_ids unless thread_ids is null in which case the dictionaries for all of the jobs are returned.

The thread_attribs may be set to a list of thread attributes that will be retured. If the notes attribute is to be returned, then note_attribs may be set to a list of note attributes that will be returned. These arguements can help reduce the execution time of the function adn the amount of data returned.

Thread Attributes:

  • clientHostname   - The client hostname associated with the thread.
  • clientId   - The id of the client associated with the thread.
  • jobId   - The id of the job associated with the thread.
  • notes   - The notes that belong to the threads.
Note Attribues:
  • clientHostname   - The client hostname associated with the thread.
  • clientId   - The id of the client associated with the thread.
  • date   - The date and time that the note was last added.
  • jobId   - The id of the job associated with the thread.
  • text   - The contents of the note.

getoutput(id)
Deprecated. Use getJobOutput() instead.

Return the output of the executed commands specified in the given job. Return None if id is not a valid job id.

getPausedJobStatusNames()

Return a tuple containing the names of what is considered a paused job status.

getPriorityChangedEventNames()

Return a tuple contining the names of the priority change events.

getprogress(id)
Deprecated. Use either getJob() or getJobs() instead.

Return two numbers representing the numerator and denominator to use when calculating the job progress as a percentage (i.e. progress = numerator / denominator). If the returned denominator is None, then use a denominator of 100.0. If the returned numerator is None, then the progress is 0.0. Return None if id is not a valid job id.

getRescheduledEventNames()

Return a tuple containing the names of the reschedule events.

getResumedEventNames()

Return a tuple containing the names of the resume events.

getRunningJobStatusNames()

Return a tuple containing the names of what is considered a running job status.

getServerTime()

Return the current time reported by the HQueue server. The time is in UTC.

getStartedJobEventNames()

Return a tuple containing the names of what is considered a start job event.

getstatus(id)
Deprecated. Use either getJob() or getJobs() instead.

Return the status of the given job. Return None if id is not a valid job id.

getSucceededStatusNames()

Return a tuple containing the names of what is considered a succeeded job status.

getUnfinishedJobStatusNames()

Return a tuple containing the names of what is considered an unfinished job status.

getUnfinishedRootJobIds()

Return a list of job ids for all unfinished, root jobs. Root jobs are ones that have no parent.

getVersion()

Returns the version of the HQueue server.

getWaitingJobStatusNames()

Return a tuple containing the names of what is considered a waiting job status.

newjob(jobspec, parent_id=None, child_ids=[])

Submit a new job to the server. jobspec can be either a JSON string, a dictionary or a list of dictionaries which represents the job specification(s). For more information on the structure of a job specification, see 8.2 The Job Specification . parent_id is an optional argument which takes the id of a job that the new job should be a child of. child_ids is an optional argument which takes a list of ids for jobs that should be children of the new job. A list of job ids is returned.

ping()

Test whether the HQueue server is active. Always return True.

removeNotes(note_ids)

Removes the notes with the specified note ids from the server. An array of note ids that could not be deleted is returned. When deleting many notes, you must put the note id of the first note in a thread after the note ids of all the other notes of that thread.

removeSingleNote(note_id)

Removes the note with the specified note id from the server.

removeSingleThread(thread_id)

Removes the thread with the specified thread id, and all of the notes belonging to that thread, from the server.

removeThreads(thread_ids)

Removes the threads with the specified thread ids, and all of the notes belonging to those threads, from the server. An array of thread ids that could not be deleted is returned.

runcmd(cmd)

Executes the given command on the HQueue farm. This is accomplished by creating a new anonymous job with its command property set to cmd. The job is assigned to a client machine and processed. The id of the anonymous job is returned.

setdisabled(hostname, disabled)

Disables or enables the given client machine depending on the value of disabled. disabled must be either True or False. Return True if the method succeeds. Return None otherwise.

threadExists(thread_id)

Returns a boolean representing whether a thread with the given thread_id exits or not. If there are errors while attempting this, a null value (e.g. None for python) is returned instead.

Scripting Examples

This example shows how to connect to the HQueue server and test if the connection is valid:

import xmlrpclib

hq_server = xmlrpclib.ServerProxy( "http://localhost:5000")
try:
         hq_server.ping()
except:
         print "HQueue server is down."

This example shows how to submit a new HQueue job and output its progress:

import time
import xmlrpclib

# Connect to the HQueue server.
hq_server = xmlrpclib.ServerProxy( "http://hq_server_hostname:5000")

# Define a job which renders an image from an IFD using Mantra.
job_spec = {
         "name": "Render My Image",
         "shell": "bash",
         "command":
                  "cd $HQROOT/houdini_distros/hfs;"
                  + " source houdini_setup;"
                  + " mantra < $HQROOT/path/to/ifds/some_frame.ifd"
}

# Submit the job to the server.
# newjob() returns a list of job ids (in case multiple jobs are passed in at once).
job_ids = hq_server.newjob(job_spec)

# Periodically check the job progress and status.
while True:
         # Get the job status.
         job_details = hq_server.getJob(job_ids[0], [ "progress", "status"])
         status = job_details["status"]

         # Check if the job is finished.
         if status in ("succeeded", "failed", "cancelled", "abandoned"):
                  break

         # Job is not finished. Output its progress.
         progress = job_details["progress"]          print "Progress: %.2f" % (progress * 100)

# Output final status.
print "Status: ", job_details["status"]