Search - User list
Full Version: tractorscheduler and "Reset $HIP on Cook" output location?
Root » PDG/TOPs » tractorscheduler and "Reset $HIP on Cook" output location?
davidoberst
I'm trying to submit to our Tractor farm (Houdini 17.5.460), and have a .hip file from production for testing. Call the value of $HIP for this file “ORIGHIP”, and the working directory set in the tractorscheduler node as “WORKINGDIR” (so that when the farm runs the copy of the .hip file from the working directory, $HIP would be WORKINGDIR). To submit, I programatically create a “topnet” node, containing a “tractorscheduler” node (filled in with appropriate parms to go to our farm) and a “ropfetch” node, which has “Reset $HIP on Cook” turned on. The ropfetch is pointing to a filecache node (part of a “geo” node“) which creates ”.vdb“ files, with the ”file“ parameter (”Geometry File“) being ”$HIP/geo/frame.$F4.vdb“, and they can generate those geo .vdbs fine (without the topnet) locally.

I've deleted the .vdb file for a single frame (say, ”frame0001.vdb), so the job should be forced to recreate that. I can programmatically do a submitGraphAsJob(), and it spools out to our farm and starts up. Because we have turned on “Reset $HIP”, all uses of $HIP in node parms should point back to the ORIGHIP directory (which the farm can see)? And indeed, the farm job (on centos blades) sees that frame0001.vdb doesn't exist in the appropriate subdir of ORIGHIP, and creates a task to cook it up. The ropfetch task succeeds, and the main cooking task outputs success like this:
Finished Cook
Work Item States:
ropfetch_farm_1                             workItemState.CookedSuccess
But the output frame0001.vdb file has been placed in the WORKINGDIR tree, not the ORIGHIP tree. The Tractor log for the task ends with something like this:
PDG_RESULT: ropfetch_farm_1;-1;'__PDG_DIR__/geo/frame.0001.vdb';;0

Shouldn't the output honour the “Reset $HIP” setting and be placed in ORIGHIP/geo/frame.0001.vdb, and not relative to __PDG_DIR__ (which is WORKINGDIR)? Am I missing something on how to get the output to go back to ORIGHIP as desired?

I tried doing the same thing with 18.0.348, but there's something in my .hip file that doesn't like 18, and I'll have to get the production people to give me a fixed copy before I can try spooling it out.
davidoberst
I got them to make me a small test file, and got the current 18.0.460 production build installed on our farm. 18 still does the same thing where the output is created in our working directory, not in the original $HIP location as the “Reset $HIP” setting suggests it should be.

What should I be expecting here?
chrisgreb
Could you attach a hip here or to a support ticket so we can take a look? In my test the hip file is written to WORKINGDIR along with pdgtemp, but the output files which are relative to $HIP are written to ORIGHIP.
chrisgreb
Oh I misread your question and was testing with an interactive cook.

Because you are using `submitAsJob`, the hip file is being first copied to the working directory and then the top network is cooked. So at this point the ‘real’ $HIP is considered to be WORKINGDIR as far as the ropfetch node is concerned. When those jobs cook, the hip is not re-copied because $HIP == WORKINGDIR, and `Reset $HIP` does nothing, so $HIP continues to be WORKINGDIR.

Logged as issue #105717
davidoberst
Oh, so even aside from the case of the tractorscheduler copying the .hip file to a working directory and running from there, there's some sort of .hip copy concept possibility in ropfetch itself, and the Reset $HIP is internal to that? I hadn't realized that - I'm a non-Houdini type that's coming n to try and add support for farming out their existing workload. So, the scheduler working directory concept is another level of $HIP-changing that isn't being taken into account? The documentation reads so perfectly like what we were looking for that it didn't occur to me.

So most workflows don't rely on $HIP-relative paths in this manner, unless it is for intermediate stuff that will be generated and used as part of the cook, or for later processes that can deal with things in the working directory? Unfortunately, I think a lot of our stuff is producing product like geometry or .vdbs or whatever for a Maya scene that is located above the original $HIP location and is referencing the files. We want to offload that to the farm to free up their workstations, but it now runs in a different $HIP context than intended.

What I may be able to do for our purposes, when I programmatically create the topnet/ropfetch/tractorscheduler bundle, is go through the node that is being attached to the ropfetch, convert $HIP-relative parms to be absolute, spool the job, then undo the changes. I'll give that a shot.
davidoberst
Or, actually, could one arrange to alter $HIP once the .hip file has been opened from the working directory? Houdini itself wouldn't be using the value of $HIP internally and have problems if it didn't match the actually open .hip file?

The docs say that Houdini will run $HOUDINI_PATH/scripts/456.py whenever a scenefile is loaded, and $HIP is on $HOUDINI_PATH once the file is loaded. However, Houdini does not seem to run $HIP/456.py when a .hip file is opened. I'm guessing $HIP is only added to $HOUDINI_PATH after the 456 mechanism has executed? Otherwise I could have arranged to have a little 456.py file copied along with the working directory, and have it do something like os.environ = “/orig/hip/path” to cleanly alter $HIP at startup.
chrisgreb
I fixed the issue in the next build of 18.0, but it sounds like you're on 17.5?

> there's some sort of .hip copy concept possibility in ropfetch itself

No, there's only the general mechanism that when the TOP graph cooks, it will copy the hip file to the scheduler working directory if it's not already there. The problem here is that submit-as-job doesn't work with it, which is what I've fixed in 18.
davidoberst
We have a current production that is using 17.5, but all upcoming stuff will use 18, so if it is changed in the next production release of 18.0 that will be good. For the last 17.5 stuff I'll find some way of trying to adjust $HIP when it is opened from the working directory copy.

I should be able to stash the original $HIP somewhere, and restore it when the file is opened in the working directory? Perhaps something like mytractorschedulernode.setUserData(“orighip”, OrigHIP)? Our crew are on Windows, while the farm is Linux, and our working directories would be created on a separately named share. So I can probably use a global HSITE/scripts/456.py to check if a file is being opened under the specific Tractor/workingdir conditions, and do something like os.environ = mytractorschedulernode.userData(“orighip”)?
chrisgreb
probably yes - you'd want to do something like:

hou.hscript('set HIP={}'.format(original_hip))
hou.hscript('set HIPFILE={}/{}'.format(original_hip, hip_basename))
hou.hscript('varchange')
This is a "lo-fi" version of our main content. To view the full version with more information, formatting and images, please click here.
Powered by DjangoBB