DjangoBB LoFi version

Wolrajh

May 3, 2019 07:31:13

So I've been getting this error way to often these days, to the point I -still can't- submit wedges to Deadline.

2019-05-03 13:19:16:  0: STDOUT: Loading .hip file V:/POUET/WORK/film/000_preprod/Test_Roto/scenes/fx/test_Roto_Track_FireODF_v03a/test_Roto_Track_FireODF_v03a.hip.
2019-05-03 13:19:20:  0: STDOUT: Traceback (most recent call last):
2019-05-03 13:19:20:  0: STDOUT:   File "V:/POUET/WORK/film/000_preprod/Test_Roto/scenes/fx/test_Roto_Track_FireODF_v03a/pdgtemp/20068/scripts/rop.py", line 500, in <module>
2019-05-03 13:19:20:  0: STDOUT:     cooker.cookBatchFrames(args)
2019-05-03 13:19:20:  0: STDOUT:   File "V:/POUET/WORK/film/000_preprod/Test_Roto/scenes/fx/test_Roto_Track_FireODF_v03a/pdgtemp/20068/scripts/rop.py", line 255, in cookBatchFrames
2019-05-03 13:19:20:  0: STDOUT:     hq.setParmValueInRopNodeAndInputs(cb_node, 'preframe', poll_cb)
2019-05-03 13:19:20:  0: STDOUT:   File "X:/_MTC_C~1/_CONFI~1/HOUDIN~1.229/houdini/python2.7libs\hqueue\houdini.py", line 87, in setParmValueInRopNodeAndInputs
2019-05-03 13:19:20:  0: STDOUT:     hq.houdini.setParmValue(parm, val)
2019-05-03 13:19:20:  0: STDOUT:   File "X:/_MTC_C~1/_CONFI~1/HOUDIN~1.229/houdini/python2.7libs\hqueue\houdini.py", line 62, in setParmValue
2019-05-03 13:19:20:  0: STDOUT:     parm.getReferencedParm().deleteAllKeyframes()
2019-05-03 13:19:20:  0: STDOUT:   File "X:/_MTC_C~1/_CONFI~1/HOUDIN~1.229/houdini/python2.7libs\hou.py", line 44977, in deleteAllKeyframes
2019-05-03 13:19:20:  0: STDOUT:     return _hou.Parm_deleteAllKeyframes(*args)
2019-05-03 13:19:20:  0: STDOUT: hou.PermissionError: Failed to modify node or parameter because of a permission error.  Possible causes include locked assets, takes, product permissions or user specified permissions
2019-05-03 13:19:20:  0: INFO: Process exit code: 1

This also seems to be happening with custom HDA with embedded Python scripts that directly alter fields inside the HDA ( like, a python script at root level that would drive what file a file node is supposed to read from ). Some HDA that were working perfectly fine in 17.0 broke with 17.5 unless I set a * to editable nodes.

ANYWAY, this can't be a solution when working at a company with locked assets and HDAs, so maybe, juuust maybe, the deadline schedulder could actually, at last, work as expected ? I've been struggling with paths not being set correctly, variables not being passed on to Deadline, DHCP failures that forced me to edit the command line after the job was submitted, and now this.
Bugs and crashes ( and god they are legion ) are enough of a pain as it is, but not having been able to use Deadline for wedging since 17.5 have been released is a problem.

Oh, this is using 17.5.229, the latest production build.
Also, I'm passing a @divSize attribute to the division size of a smoke object.

kenxu

May 3, 2019 10:53:28

Hi Wolrajh,

I think the paths and variables issue with Deadline has been solved. DHCP you seem to have a work-around for, and we are working on a longer term solution that we may be able to deliver in a few weeks (have a “tracker” run on the farm, so everything talks to the tracker, and the tracker talks to PDG thus bypassing the DHCP issues).

The asset unlocking - we've been working with the assumption that parameters that need to be set are exposed at the top level, or nodes are appropriately marked as editable. However, we see this is not the case here. We'll try to put in a change to “auto-unlock” these things on the farm. We can probably do this fairly quickly. If there are other issues with Deadline that we're not yet aware of, please point them out.

Help is on the way…thanks for your patience - bear with us here

Wolrajh

May 3, 2019 11:33:37

Yes, it's more a case of “that bug, then this bug, then that other bug” chaining up that had me snapping a bit here ! I don't mean to hurt feelings, it was just an expression of my frustration at the situation. Sorry !

I must say though, I don't think I ever saw an HDA with actual editable nodes properly set up inside, and my confusion here comes from the fact that I'm not even sure what node is causing the issue. I suppose it's our custom deadline submitter, filled to the brim with Python code : we basically set up an asset name, a version, and the python code sets up the actual render path according to our production pipeline - as well as doing a bunch of other stuff. Making anything editable inside of such an asset is quite risky : anyone, myself included, could just break it by accident.

And to be honest, I'm not entirely fair to the PDG system here: HDA with custom python just stopped working altogether when 17.5 came along. Again, for my own assets, or “small”, non-essential assets from third party, it's annoying but one can just put a * in the editable nodes ( which is not ideal, but hey, when you have a lot of nodes modified by the code, it quickly becomes a pain to track them all - one by one - to flag them )

To the credit of TOPs and PDG, when it works, it's just amazing the whole extent of what we can do. Which makes it all the more frustrating when it doesn't, which, sadly, happens a lot, and sometimes at total random. I'm still trying to track an issue with incorrect translation of __PDG_DIR__ which sometimes results in the system being unable to resolve file paths, so that I can RFE it but it seems to happen quite at random for now

chrisgreb

May 6, 2019 10:51:49

The RopFetch PermissionError should be resolved in the next build.

Wolrajh

May 7, 2019 07:16:36

Just caught that in the daily changelogs, thanks a lot

kenxu

May 7, 2019 10:53:06

Hi Wolrajh,

So hopefully we are one step closer to getting it deployed properly for you guys. A solution for the DHCP issue is on the way as mentioned before - any other blockers, please let us know.

Wolrajh

May 7, 2019 12:54:53

Very glad to hear it really! Hopefully this will be the last of the blockers - rest will be the stuff of RFE, mainly options for quality of life improvements, things that are actually already available now when adding custom Job info ( for anyone reading, things like “LimitGroups” for proper handling of licences ).

I would have one last question while waiting for the DHCP solution implementation ; when/where exactly is “command” created ? I mean, it's added upon item generation, but I was thinking of “intercepting” the item.SetCommand(command) function that actually creates that attribute in order to replace the hostname with its IP ( call me lazy, but replacing the name by my IP in 16, 24, 32+ jobs manually in each Deadline job really gets tedious :p ). I don't know if we, as frontend users, can access that part of the process.
Or I could just tweak tbdeadline.py to find and replace hostname by IP once the cmdplugin* files are created, it just feels a bit less… Elegant !

edit - ah well, just a few lines later in tbdeadline.py, hostname <-> IP is working. Probably the best bandaid I can come up with, ha !