Forums PDG/TOPs Missing /scripts/rop.py with custom scheduler

Missing /scripts/rop.py with custom scheduler

   2077   2   3
User Avatar
vfxwork
Member
2 posts
Joined: July 2019
Offline
 
Hello!

I am writing a custom scheduler and I am getting pretty close to success. Right now I am able to communicate with our farm, get job/task statuses and logs all working.

The issue that I am having is that whenever a farm machine picks up the “rop geometry output” task, Houdini does not write a rop.py to the /pdgtemp/<process_id>/scripts/ folder. Hence giving me the error: /opt/sidefx/hfs-17.5.258/bin/hython-bin: can't open file ‘/PATH/TO/PDG FOLDER/pdgtemp/<process_id>/scripts/rop.py’: No such file or directory

The local scheduler works just fine, I see the bgeo end up where it needs to be. However during this process I don't see a “pdgtemp” being created and thus no scripts folder.

I have tested the permissions on the folder, they are fully writeable by both (and any other) machines. From any machine on our network I can also view/write the folder just by it's path. My onStartCook looks like this:

        wd = self["pdg_workingdir"].evaluateString()
        self.setWorkingDir(wd, wd)
        if not os.path.exists(self.tempDir(True)):

            os.makedirs(self.tempDir(True))

        if not os.path.exists(self.workingDir(True)):
            
            os.makedirs(self.workingDir(True))

        if not os.path.exists(self.scriptDir(True)):
            
            os.makedirs(self.scriptDir(True))

        if not self.isCallbackServerRunning():

            self.startCallbackServer()

        self.tick_timer = TickTimer(0.25, self.tick)
        self.tick_timer.start()

        return True

Just a part of my onSchedule (due to sensitive code):

            temp_dir = self.tempDir(False)
            work_dir = self.workingDir(False)
            script_dir = self.scriptDir(False)

            item_command = item_command.replace("__PDG_ITEM_NAME__", item_name)
            item_command = item_command.replace("__PDG_SHARED_TEMP__", temp_dir)
            item_command = item_command.replace("__PDG_TEMP__", temp_dir)
            item_command = item_command.replace("__PDG_SHARED_ROOT__", work_dir)
            item_command = item_command.replace("__PDG_DIR__", work_dir)
            item_command = item_command.replace("__PDG_SCRIPTDIR__", script_dir)
            item_command = item_command.replace("__PDG_RESULT_SERVER__", self.workItemResultServerAddr())
            item_command = item_command.replace("__PDG_PYTHON__", "python")
            item_command = item_command.replace("__PDG_HYTHON__", "hython")

            # Ensure directories exist and serialize the work item
            self.createJobDirsAndSerializeWorkItems(work_item)

My onSchedule contains self.createJobDirsAndSerializeWorkItems(work_item). The SideFX changelog [www.sidefx.com] tells me: createJobDirsAndSerializeWorkItems: creates job directories (log, data), and serializes work items. This indeed creates the log and data folder but not scripts, that's why I create them myself in the onStartCook.

I think I am missing something somewhere, I just can't figure it out?

Thanks for future replies.
 Report
User Avatar
chrisgreb
Member
603 posts
Joined: Sept. 2016
Offline
 
rop.py is a file dependency of the ropfetch TOP node, which is inside ROP Geometry Output.

The scheduler function transferFile(self, file_path) should be called automatically for rop.py before the first such rop workitem is scheduled. Can you confirm that the function is being called? It's implemented in pdg/scheduler.py
Edited by chrisgreb - July 19, 2019 12:47:37
 Report
User Avatar
vfxwork
Member
2 posts
Joined: July 2019
Offline
 
Thanks for pointing me to pdg/shceduler.py, should have found that earlier actually.

I have found the issue. When first starting to code up the scheduler I used the “Save to Python Script” button on the Python Scheduler node to start with a template. This .py file will have the transferFile() function included in the template script like so:

def transferFile(self, file_path):
      # Custom transferFile logic. Returns True on success, else False.
      #
      # The following variables are available:
      # self          -  A reference to the current pdg.Scheduler instance
      # file_path     -  Path to file that should be moved

      return True

This was still floating around in my Custom Scheduler python script and this was being called instead of the function inside pdg/scheduler.py. By removing it in my own script, of course it now works.

Thanks so much for the insight!
 Report
  • Quick Links