when You run it directly after opening the file, it fails, top.getPDGNode() results in None. after cooking the top nodes with the ui and deleting the results, the script works. I wonder, what is going on here. Tt seems that a initial cool by hand, does initialize something so the getPDGNode does work. How can I do this?
yes we are using the muster scheduler. the vvertex developer is very supporting, very cool and we are in contact with him.
Hi Chris,
I get Your point (more or less), with a typical static job description You loose a lot of functionality. Still trying to get the broader concepts when submitting stuff to farms. I think I have to install at some point hqueue to be able to compare. The downside I see right now with a full pdg submission to farm is the lack of overview. Jobs pop into existence depending on previous jobs, what makes it hard to keep track if You have multiple jobs with dependencies. it is hard to keep track. Right now if a job in the chain fails, I'm not to sure if I can restart this specific job/junk and everything is just fine.
I specifically have have tasks in mind, that are low maintenance, proven and of high count. Generic stuff, creature fx, stuff like that. they are less dynamic in job creation. asset imports, collision preparation, presim, sim and postsim passes, flip book and ffmpeg. maybe a bit wedging. they are not difficult, but most of the time You have to deal with huge amounts and just want to throw it on to the farm, and see result clips before You continue to push it into the next department.
It would be easier with the top interface embedded into the scheduler. What position does Pilot have here?
Hi Manuel, thank You. I'm a bit confused as we now connect to the farm renders with a musterscheduler, not sure where a additional scheduler fits here. But I see, there are “submit graph as job buttons” in hqueue and deadline as well. Does that mean that the complete graph is processed as a job on farm, just as if You open the scene and do a Cook Output Node?
I was looking for a even more static way. More like all static tasks are evaluated with all dependencies, and then all nodes that create files are submitted as “indipendent” job to the dispatcher, but with correct dependencies set on job or chunk level. I tried to resolve the dependencies with the api, but I could not easily get over nodes like wait for all, as they seem to have no staticWorkItems jsut staticWrapers, that made me wonder is there maybe already a defined way in the api, to resolve all dependencies.
About the expanding jobs stuff, I think this is possible. right now when I start a job with several top nodes, the first batches of tasks get launched on the assigned computers, and no other job exists for muster at this moment. as soon as one batch is finished a dependent batch pops into existence on muster. the only info it has is the, top node, task name and framerange. looks like thisL: –batch -p “filename” -n “\obj\geo1\fileCache1\render” -i “ropfetch40” -fs 1 -fe 1 -fi 1 muster seems to just connect to the farm. all the dispatching happens in pdg. hope that helps
Hello, we successfully connected our dispatcher ( Muster )to PDG and the task are executed like expected on the assigned computers. That's very cool, but I must admit I do not fully understand the way this should work. In this setup I can launch PDG graphs on the dispatcher, but as soon as I quit houdini or it dies(happens from time to time), the graph is not executed anymore. It seems all the actual dispatching work is done in PDG and as soon as it goes down, the current task are finished, but task that should be started are not started anymore. Like expected.
But often we would need a more disconnected way to work with PDG graphs. For example it would be quite unrealistic to have a 3 day render on the farm and keep houdini open for three days, so it can proceed with some ffmpeg and comp tasks. Let's say I have a top with sim, a ifd generate, a ifd render and a ffmpeg and just want to send it to the farm, and then close houdini, or do something different. Are there any methods already to do this? Where does Pilot fit into this, by the way?
Still learning, but my impression of TOPS is a bit different, but I guess my expectations are also quite different. I see it more like a scheduler on steroids, that greatly increases You throughput on farm. To have all this fine controlled dependencies is just amazing.
Just some examples:
- Clustered and sliced simulations, in addition with wedging in a custom scheduler environment was doable, but You had to use environment variables, that communicate the wedge/slice/cluster information to the batch process. It is working, but would break much easier and was kind of tricky in developing state. PDG on the other side was just working and with a huge artistic usability.
- All kind of repetitive workflows are now much easier to set up, without the need to code a lot. You can plug Your workflow together and modify it on the fly. You can create Digital Assets from this workflows and create higher levels of abstraction for this, so that one node does in fact do several things. Coding and maintaining this is of course doable, but can get quite difficult as soon as You have multiple deep dependencies, that should be parallelized on farm, to use all the resources.
- But even without farm computers, we see now cpus with up to 64 cores in a single cpu setup, You have to use that resources as well.
For example, You process all the collision geometry preparation in the background, while already working on some other parts of Your scene, or even let PDG completely build all Your hips to that point, that all necessary assets are loaded, and the mandatory steps are also done in background, as well. You open a hip where everything is done to that point where You and Your artistic skills are really needed. After this PDG takes care of all the other stuff that is necessary in a production. It can free up a lot of Your time for the stuff that no machine can do. This means You are faster and more productive. A lot of work is still very repetitive and if You can shift it to PDG, great.
I might miss a lot here, but that is my initial impression of PDG. Probably it is much more, but step by step
we are switching our tool set completely to TOPS right now, still keeping our old submission tools available. The PDG system is quite new and I guess it will evolve quite fast. that keeps one busy but I think it is totally worth it. So far I just love it, but having a fall back is kind of relaxing.
We are not using tractor or deadline, but muster as a scheduler. and the muster developer is picking up stuff quickly, but I do have some rather basic questions, as the tool right now is heavily work in progress on the scheduler side.
When I cook a top it seems to be bound to the houdini running it, is there a way to submit a top tree to be processed in a more old school way.
To be more specific and that is just my impression, right now the top cooking is bound to the current houdini session, that's very cool for fluid working and developing, but I miss a way to just submit a dependency tree to the farm, and then I can close houdini or open a new scene. while the top processes are calculated on the farm. Is that possible in hqueue already? Am I missing here something or is that right?
always appreciate Your input! maybe a custom cook tool could do the job, that thats a unique environment variable and copies a the file with the unique id as well.
Hi Fuat , I just want to have a snapshot of the processed file, not wedge dependent. But as You constantly change stuff when working with TOPS and sometimes there are a lot of dependencies, things might change and simulations do not behave like in the cached version any more. Consider a collision object reused in several sims. and You change something for a later sim, that would also affect a sim that is already approved. And now You have to extend this approved simulation. Right now this would be at least not cool. But when every sim that is processed gets a scnap shot version of the hip and the name of the snap is added as a attribute to the cache, You can easily go back and see what changed and fix it, or just sim the file. But You are not stuck.
Hi when starting a top cook the current file gets saved and gets processed with something like this:
our current tools for farm submission work like this:
the hip gets saved
the hip file gets copied to a /submit directory with a unique hash added to the file
now the file is used for processing on the farm, all file out nodes add this file path as a attribute in the geo
the reason why we do this is that whatever happens You for sure can restore a file with simulation settings, as long as You have a cache or a playblast You can hunt down the file that generated that cache. I would like to use that workflow with pdg as well, but is there a way? now You work on Your file and might change stuff so it is impossible to recreate the simulation that might have been approved. As there can be a lot of dependencies in a file this easily happens.
right now the cooking process start a batch like this:
