PDG product configurator

Forums PDG/TOPs PDG product configurator

5103 19 2


Jonathan de Blok: Member; 251 posts; Joined: July 2013; Offline

May 6, 2019 2:27 p.m.

For a client that sells aluminum display frames I need to render a lot of different configurations.. sounds like a nice project to start learning PDG

So I did the following, I have a folder with a lot of FBX files for various extruded aluminum profiles, all available in blank anodized aluminum and those with “_BLACK” in the filename are also available in black and those with “_MULTI” in the filename are in 8 different colors.

So I did the following: 3 file readers that use wildcards to read them all, another for _BLACK ones and one for the _MULTI files.

They are partitioned by distinct filename, and with some wedges I assign the material, the alu and black are just a wedge of 1 and the multi color wedge creates 3 colors per item. The wedges also create some extra attributes I use in the output filename.

I merge all those 3 streams into one big bucket of task and using a RopFetch I can render it all out locally just fine.

So far so good.. but now I'm trying to render it on the farm, not really necessary for this project but I want to get that workflow down as well.

Ideally I want to submit each workitem as a normal deadline Houdini job, but I don't think that's really possible?

So using the deadline scheduler i'm trying to run the tasks through deadline..

But this doesn't seem to work:

Error executing pre job script: script file "C:\Users\\AppData\Local\Thinkbox\Deadline10\slave\GT\jobsData\5cd07523b658f89b0421c090\$HFS/houdini/pdg/types/schedulers/deadline_jobpreload.py" does not exist.

And that file does indeed not exist.

Is this part still under construction or am I missing something?

And second can we get custom attributes into the workitem spreadsheet? I'd like to put the @filename in there for example so I can click on that row and don't have to guess which file belongs to it:

Attachments:
Capture.PNG (51.8 KB)
Capture2.PNG (604.0 KB)

More code, less clicks.


seelan: Member; 571 posts; Joined: May 2017; Offline

May 7, 2019 12:55 p.m.

You'll need to set $HFS in Deadline's Configure Repository Options. It should point to your Houdini installation on the farm machines. The error was due to Deadline not finding $HFS and therefore concatenating with the default Deadline job path.

See screenshot.

Attachments:
deadlin_map2.png (46.5 KB)


kenxu: Member; 544 posts; Joined: Sept. 2012; Offline

May 7, 2019 1:20 p.m.

WRT the custom columns in the task graph table, it's a well known RFE. We are working on it.

- Ken Xu


Wolrajh: Member; 123 posts; Joined: May 2015; Offline

May 7, 2019 9:11 p.m.

seelan
You'll need to set $HFS in Deadline's Configure Repository Options. It should point to your Houdini installation on the farm machines. The error was due to Deadline not finding $HFS and therefore concatenating with the default Deadline job path.

See screenshot.

Alternatively, you can also just add a HFS variable in your scene. It should also be pointing to the Houdini path the farm machines have access to. Our custom launcher set that variable for us so it's a bit easier when you potentially have different production / daily builds installed.
Remember to transform any \ to /, not sure it's necessary anymore but it's often the root of some issues at some point


Jonathan de Blok: Member; 251 posts; Joined: July 2013; Offline

May 8, 2019 8:04 a.m.

Ahh yes, that makes sense. Thanks guys!

More code, less clicks.


seelan: Member; 571 posts; Joined: May 2017; Offline

May 9, 2019 3:44 p.m.

In tomorrow's build, the TOP Deadline scheduler node will have a new HFS parm field that you can just set directly, or to a variable that your farm is setup to map (in Deadline's Configure Repository -> Mapped Paths).


Jonathan de Blok: Member; 251 posts; Joined: July 2013; Offline

May 12, 2019 3:14 p.m.

seelan
In tomorrow's build, the TOP Deadline scheduler node will have a new HFS parm field that you can just set directly, or to a variable that your farm is setup to map (in Deadline's Configure Repository -> Mapped Paths).

Awesome!

More code, less clicks.


Jonathan de Blok: Member; 251 posts; Joined: July 2013; Offline

May 22, 2019 8:47 a.m.

Back to this…
Using the $HSF remapping I get that to work part to work.. deadline executes the setup script..

But the actual rendering fails because it's lookin in the wrong place and the schedulars seem to be ignoring the path settings.

For now I got everything on the same machine, houdini, deadline repo, slave etc are all on the same machine and using the same fielsystem

Deadline is trying to render this file that doesn't exist:
Cannot find file E

Projects/MatrixFrames_Stills/Houdini/_PDG/topTest/pdgtemp/topTest.hiplc

Mainly because houdini's PID is injected into the filepath which I turned off..

Any clues?

Attachments:
ls.png (36.1 KB)
dls.png (50.2 KB)
pid.png (22.7 KB)

More code, less clicks.


seelan: Member; 571 posts; Joined: May 2017; Offline

May 22, 2019 9:25 a.m.

Are you using both local scheduler and deadline scheduler for cooking the TOP network? Try to use just one to schedule everything because each scheduler type will have slight differences in how the working directory is set up.

You might also have to delete all the generated results by going into Network Editor's Tasks > Delete All Results From Disk. Make sure the topTest.hiplc file has been deleted when you do that. Then set your primary scheduler to Deadline, and recook.

Check the TOP node that created the topTest.hiplc file to make sure it has its output path correct as well.


Jonathan de Blok: Member; 251 posts; Joined: July 2013; Offline

May 22, 2019 9:41 a.m.

seelan
Are you using both local scheduler and deadline scheduler for cooking the TOP network? Try to use just one to schedule everything because each scheduler type will have slight differences in how the working directory is set up..

Ahh I'll give that a go. I figured I needed local to generate tasks and I only put the final fetch_rop render task into the deadline scheduler..

More code, less clicks.


Jonathan de Blok: Member; 251 posts; Joined: July 2013; Offline

May 22, 2019 10:27 a.m.

This is now with a single deadline scheduler.. but still no go:

Are you sure this is working at all? It's adding pdgtmp/{pid} to the filename location but it's not telling that to what ever is loading it later again:

ShellExecute=False
Executable=C:/Program Files/Side Effects Software/Houdini 17.5.258/bin/hython.exe
Arguments="E:/Projects/MatrixFrames_Stills/Houdini/_PDG/TopTest/pdgtemp/39028/scripts/rop.py" "-p" "E:/Projects/MatrixFrames_Stills/Houdini/_PDG/TopTest/topTest.hiplc" "-n" "/out/Redshift_ROP1" "-i" "ropfetch2_1_73" "-s" "GT:10841" "-fs" "1" "-fe" "1" "-fi" "1"

resulting in a:

2019-05-22 16:25:07:  0: STDOUT: Loading .hip file E:/Projects/MatrixFrames_Stills/Houdini/_PDG/TopTest/topTest.hiplc.
2019-05-22 16:25:07:  0: STDOUT: Cannot find file E:/Projects/MatrixFrames_Stills/Houdini/_PDG/TopTest/topTest.hiplc

Edited by Jonathan de Blok - May 22, 2019 10:28:58

Attachments:
t3.png (89.9 KB)

More code, less clicks.


Jonathan de Blok: Member; 251 posts; Joined: July 2013; Offline

May 22, 2019 10:37 a.m.

if I look at the task info.. the __PDG_DIR__ seems to be the problem here:

Attachments:
task.png (96.2 KB)

More code, less clicks.


seelan: Member; 571 posts; Joined: May 2017; Offline

May 22, 2019 11:52 a.m.

The __PDG_DIR__ is your working directory, as set in Deadline TOP node. So looking at the screenshot above, it should be “E

Projects/MatrixFrames_Stills/Houdini/_PDG/TopTest/pdgtemp” because it is the concatenation of your Local Root Path (E

Projects/MatrixFrames_Stills/Houdini/_PDG) and Relative Job Directory (TopTest/pdgtemp). So there shouldn't be a PID in the file path. Can you reload your hip file in Houdini and try again?

You can also try it out in a simple scene with a single ROPFetch.


Jonathan de Blok: Member; 251 posts; Joined: July 2013; Offline

May 22, 2019 3:02 p.m.

Tried with a fresh scene.. I think it worked once but haven't been able to repeat that. It always stores stuff in a /pdgtmp/pid structure even when there is only a deadline scheduler present.

I've checked the pyhton code for the deadline scheduler, there is not even a os.getpid() in there so my guess is something is calling an invisible local scheduler that messes things up. Anyways, I've spend most of the day trying to make this work and I'd say it's broken.

I've attached a barebones test file with a wedge that sets a varying radius on a torus and a deadline scheduler to submit it.. curious to see if you can make it work.

Attachments:
broken_pdg.hiplc (196.8 KB)

More code, less clicks.


seelan: Member; 571 posts; Joined: May 2017; Offline

May 22, 2019 4:29 p.m.

I tried your hip file, and it works fine, but had to make changes to the deadline scheduler paths to my local setup. I noticed though that you still have the local scheduler set as your TOP network's Default TOP Scheduler. Make sure you change that to the Deadline node, then reload the hip file.

Note that the PID is used internally to store intermediate PDG files such as scripts and logs, but that shouldn't affect your ROPFetch. But please double check that the ROPFetch is NOT specifying an External HIP file path.

Are you overriding the job parameters under the Scheduler's tab on the ROPFetch node? Post screenshot of it if you are.

More importantly, can you post screenshots of your Deadline node's parm values for both Scheduler and Job Parms?

Do you have any Mapped Paths in the Deadline Repository Configuration settings? I don't think these should be causing issues, but check that as well.


Jonathan de Blok: Member; 251 posts; Joined: July 2013; Offline

May 23, 2019 2:10 a.m.

seelan
I tried your hip file, and it works fine, but had to make changes to the deadline scheduler paths to my local setup. I noticed though that you still have the local scheduler set as your TOP network's Default TOP Scheduler. Make sure you change that to the Deadline node, then reload the hip file.

Note that the PID is used internally to store intermediate PDG files such as scripts and logs, but that shouldn't affect your ROPFetch. But please double check that the ROPFetch is NOT specifying an External HIP file path.

Are you overriding the job parameters under the Scheduler's tab on the ROPFetch node? Post screenshot of it if you are.

More importantly, can you post screenshots of your Deadline node's parm values for both Scheduler and Job Parms?

Do you have any Mapped Paths in the Deadline Repository Configuration settings? I don't think these should be causing issues, but check that as well.

Thanks for checking but still no go here… I've restarted various times, wiped everything.. double checked everything. No overriding job parms, no mapped paths in deadline. No local schuduler in sight and it really keeps putting things in /pdgtmp/pid so when I find some time today I'll dive into the relevant python bits and see if I can find the problem.

screenshots:

Attachments:
check1.png (29.1 KB)
check2.PNG (106.7 KB)
check3.PNG (51.3 KB)

More code, less clicks.


seelan: Member; 571 posts; Joined: May 2017; Offline

May 23, 2019 8:56 a.m.

Ah, I see the bug with the transferFile method in scheduler.py which is the base class for tbdeadline.py. In there, we check if the file extension is .hip and move it into the working directory, otherwise its placed in the temporary directory (which has the PID). You can do a quick test by changing $HFS/houdini/python2.7libs/pdg/scheduler.py transferFile() (line 108) to the following:

elif file_name.endswith('.hip') or file_name.endswith('.hiplc'):

Then restart Houdini and try again. That should fix it. Otherwise, you can throw some print statements in there to see what is going on.


seelan: Member; 571 posts; Joined: May 2017; Offline

May 23, 2019 1:35 p.m.

The fix will be in tomorrow's build. Apologies for not catching it sooner.


Jonathan de Blok: Member; 251 posts; Joined: July 2013; Offline

May 23, 2019 2:27 p.m.

seelan
The fix will be in tomorrow's build. Apologies for not catching it sooner.

Yes, that was it.. finally!

No worries, I can see how this bug was a sneaky one

More code, less clicks.


Jonathan de Blok: Member; 251 posts; Joined: July 2013; Offline

May 23, 2019 2:31 p.m.

In the mean time I rendered it locally which did work fine.. rendered out EXRs with transparent background and used more PDG to run it through some COP network to add a white background and do some minor color tweaks.. works like a charm.

The beauty is now I can go from a single folder with a bunch of fbx files to a bunch product images with a single click, including correct files names, color options, positioning, etc.. awesome stuff!

Edited by Jonathan de Blok - May 23, 2019 14:32:35

Attachments:
result.png (294.2 KB)

More code, less clicks.

Quick Links

                    
                        Search links
                        Show recent posts
                        Show unanswered posts