Hello,
we successfully connected our dispatcher ( Muster )to PDG and the task are executed like expected on the assigned computers. That's very cool, but I must admit I do not fully understand the way this should work.
In this setup I can launch PDG graphs on the dispatcher, but as soon as I quit houdini or it dies(happens from time to time), the graph is not executed anymore. It seems all the actual dispatching work is done in PDG and as soon as it goes down, the current task are finished, but task that should be started are not started anymore. Like expected.
But often we would need a more disconnected way to work with PDG graphs. For example it would be quite unrealistic to have a 3 day render on the farm and keep houdini open for three days, so it can proceed with some ffmpeg and comp tasks.
Let's say I have a top with sim, a ifd generate, a ifd render and a ffmpeg and just want to send it to the farm, and then close houdini, or do something different.
Are there any methods already to do this?
Where does Pilot fit into this, by the way?
thanks
Martin
Found 352 posts.
Search results Show results as topic list.
PDG/TOPs » PDG Graphs on Farm
- sanostol
- 575 posts
- Offline
Technical Discussion » Blackbox otls by python?
- sanostol
- 575 posts
- Offline
Can we create Blackboxed HDAS with python?
Could not finde anything in the documentation.
thanks Martin
Could not finde anything in the documentation.
thanks Martin
PDG/TOPs » PDG/TOPs grounded - Open letter for SideFX
- sanostol
- 575 posts
- Offline
Still learning, but my impression of TOPS is a bit different, but I guess my expectations are also quite different.
I see it more like a scheduler on steroids, that greatly increases You throughput on farm. To have all this fine controlled dependencies is just amazing.
Just some examples:
- Clustered and sliced simulations, in addition with wedging in a custom scheduler environment was doable, but You had to use environment variables, that communicate the wedge/slice/cluster information to the batch process. It is working, but would break much easier and was kind of tricky in developing state. PDG on the other side was just working and with a huge artistic usability.
- All kind of repetitive workflows are now much easier to set up, without the need to code a lot. You can plug Your workflow together and modify it on the fly. You can create Digital Assets from this workflows and create higher levels of abstraction for this, so that one node does in fact do several things. Coding and maintaining this is of course doable, but can get quite difficult as soon as You have multiple deep dependencies, that should be parallelized on farm, to use all the resources.
- But even without farm computers, we see now cpus with up to 64 cores in a single cpu setup, You have to use that resources as well.
For example, You process all the collision geometry preparation in the background, while already working on some other parts of Your scene, or even let PDG completely build all Your hips to that point, that all necessary assets are loaded, and the mandatory steps are also done in background, as well. You open a hip where everything is done to that point where You and Your artistic skills are really needed.
After this PDG takes care of all the other stuff that is necessary in a production. It can free up a lot of Your time for the stuff that no machine can do. This means You are faster and more productive. A lot of work is still very repetitive and if You can shift it to PDG, great.
I might miss a lot here, but that is my initial impression of PDG. Probably it is much more, but step by step
I see it more like a scheduler on steroids, that greatly increases You throughput on farm. To have all this fine controlled dependencies is just amazing.
Just some examples:
- Clustered and sliced simulations, in addition with wedging in a custom scheduler environment was doable, but You had to use environment variables, that communicate the wedge/slice/cluster information to the batch process. It is working, but would break much easier and was kind of tricky in developing state. PDG on the other side was just working and with a huge artistic usability.
- All kind of repetitive workflows are now much easier to set up, without the need to code a lot. You can plug Your workflow together and modify it on the fly. You can create Digital Assets from this workflows and create higher levels of abstraction for this, so that one node does in fact do several things. Coding and maintaining this is of course doable, but can get quite difficult as soon as You have multiple deep dependencies, that should be parallelized on farm, to use all the resources.
- But even without farm computers, we see now cpus with up to 64 cores in a single cpu setup, You have to use that resources as well.
For example, You process all the collision geometry preparation in the background, while already working on some other parts of Your scene, or even let PDG completely build all Your hips to that point, that all necessary assets are loaded, and the mandatory steps are also done in background, as well. You open a hip where everything is done to that point where You and Your artistic skills are really needed.
After this PDG takes care of all the other stuff that is necessary in a production. It can free up a lot of Your time for the stuff that no machine can do. This means You are faster and more productive. A lot of work is still very repetitive and if You can shift it to PDG, great.
I might miss a lot here, but that is my initial impression of PDG. Probably it is much more, but step by step
PDG/TOPs » Making TOPs less confusing
- sanostol
- 575 posts
- Offline
we are switching our tool set completely to TOPS right now, still keeping our old submission tools available. The PDG system is quite new and I guess it will evolve quite fast. that keeps one busy but I think it is totally worth it. So far I just love it, but having a fall back is kind of relaxing.
We are not using tractor or deadline, but muster as a scheduler. and the muster developer is picking up stuff quickly, but I do have some rather basic questions, as the tool right now is heavily work in progress on the scheduler side.
When I cook a top it seems to be bound to the houdini running it, is there a way to submit a top tree to be processed in a more old school way.
To be more specific and that is just my impression, right now the top cooking is bound to the current houdini session, that's very cool for fluid working and developing, but I miss a way to just submit a dependency tree to the farm, and then I can close houdini or open a new scene. while the top processes are calculated on the farm.
Is that possible in hqueue already?
Am I missing here something or is that right?
thank You
We are not using tractor or deadline, but muster as a scheduler. and the muster developer is picking up stuff quickly, but I do have some rather basic questions, as the tool right now is heavily work in progress on the scheduler side.
When I cook a top it seems to be bound to the houdini running it, is there a way to submit a top tree to be processed in a more old school way.
To be more specific and that is just my impression, right now the top cooking is bound to the current houdini session, that's very cool for fluid working and developing, but I miss a way to just submit a dependency tree to the farm, and then I can close houdini or open a new scene. while the top processes are calculated on the farm.
Is that possible in hqueue already?
Am I missing here something or is that right?
thank You
PDG/TOPs » SendmailTOP with attachment?
- sanostol
- 575 posts
- Offline
ah, it works, I provided the Input. but that did not work, a string attribute works fine. Thank You again
PDG/TOPs » How to lock a TOP Node?
- sanostol
- 575 posts
- Offline
PDG/TOPs » SendmailTOP with attachment?
- sanostol
- 575 posts
- Offline
PDG/TOPs » SendmailTOP with attachment?
- sanostol
- 575 posts
- Offline
hello,
i do not get the attachment to work, what do I have to use in the attachment field?
thanks a lot
Martin
i do not get the attachment to work, what do I have to use in the attachment field?
thanks a lot
Martin
PDG/TOPs » How to lock a TOP Node?
- sanostol
- 575 posts
- Offline
Hi,
how can we lock a TOP node, so that it does never get executed anymore and the cache does not deleted at when delete result on disk gets executed?
Sometime Tops does evalutes a node, even if it has no changes.
When locked, it should just deliver all wedge informations, though.
thanks
Martin
how can we lock a TOP node, so that it does never get executed anymore and the cache does not deleted at when delete result on disk gets executed?
Sometime Tops does evalutes a node, even if it has no changes.
When locked, it should just deliver all wedge informations, though.
thanks
Martin
PDG/TOPs » can we modify the filename and path that gets used for cooking a top network
- sanostol
- 575 posts
- Offline
always appreciate Your input! maybe a custom cook tool could do the job, that thats a unique environment variable and copies a the file with the unique id as well.
PDG/TOPs » can we modify the filename and path that gets used for cooking a top network
- sanostol
- 575 posts
- Offline
Hi Fuat ,
I just want to have a snapshot of the processed file, not wedge dependent.
But as You constantly change stuff when working with TOPS and sometimes there are a lot of dependencies, things might change and simulations do not behave like in the cached version any more. Consider a collision object reused in several sims. and You change something for a later sim, that would also affect a sim that is already approved. And now You have to extend this approved simulation.
Right now this would be at least not cool. But when every sim that is processed gets a scnap shot version of the hip and the name of the snap is added as a attribute to the cache, You can easily go back and see what changed and fix it, or just sim the file. But You are not stuck.
I just want to have a snapshot of the processed file, not wedge dependent.
But as You constantly change stuff when working with TOPS and sometimes there are a lot of dependencies, things might change and simulations do not behave like in the cached version any more. Consider a collision object reused in several sims. and You change something for a later sim, that would also affect a sim that is already approved. And now You have to extend this approved simulation.
Right now this would be at least not cool. But when every sim that is processed gets a scnap shot version of the hip and the name of the snap is added as a attribute to the cache, You can easily go back and see what changed and fix it, or just sim the file. But You are not stuck.
Edited by sanostol - June 23, 2019 07:02:40
PDG/TOPs » Render flipbook
- sanostol
- 575 posts
- Offline
PDG/TOPs » can we modify the filename and path that gets used for cooking a top network
- sanostol
- 575 posts
- Offline
thank You, that is quite amazing.
Is there a way to generate a unique hip file name, so it does not get overwritten?
Is there a way to generate a unique hip file name, so it does not get overwritten?
PDG/TOPs » can we modify the filename and path that gets used for cooking a top network
- sanostol
- 575 posts
- Offline
Hi
when starting a top cook the current file gets saved and gets processed with something like this:
our current tools for farm submission work like this:
now You work on Your file and might change stuff so it is impossible to recreate the simulation that might have been approved. As there can be a lot of dependencies in a file this easily happens.
right now the cooking process start a batch like this:
is there a way to modify this so we can inject the copied file, instead of the original hip?
something like this:
thanks
Martin
when starting a top cook the current file gets saved and gets processed with something like this:
our current tools for farm submission work like this:
- the hip gets saved
- the hip file gets copied to a /submit directory with a unique hash added to the file
- now the file is used for processing on the farm, all file out nodes add this file path as a attribute in the geo
now You work on Your file and might change stuff so it is impossible to recreate the simulation that might have been approved. As there can be a lot of dependencies in a file this easily happens.
right now the cooking process start a batch like this:
"__PDG_HYTHON__" "__PDG_SCRIPTDIR__/rop.py" --batch -p "__PDG_DIR__/IO_Dev_Tool.v002_001.mma.h17.5.hip" -n "/obj/caching/fileCache1/render" -i "fileCache1_fileCacheRop0" -fs 1001 -fe 1030 -fi 1
something like this:
"__PDG_HYTHON__" "__PDG_SCRIPTDIR__/rop.py" --batch -p "__PDG_DIR__/submit/IO_Dev_Tool.v002_001.mma.h17.5.39anr02a83.hip" -n "/obj/caching/fileCache1/render" -i "fileCache1_fileCacheRop0" -fs 1001 -fe 1030 -fi 1
thanks
Martin
Technical Discussion » ANIMATION NOT UPDATING WHEN FLIPBOOKING
- sanostol
- 575 posts
- Offline
From yesterdays changelog:
Fixed flipbooking so it avoids the progressive object update that is new in 17.5.
I hope this addresses this
Fixed flipbooking so it avoids the progressive object update that is new in 17.5.
I hope this addresses this
Houdini Lounge » SideFX plans for Windows-Linux
- sanostol
- 575 posts
- Offline
BabaJno, I did not intend to use the license on another box. same computer just dual boot.
But if your already using the second license on another box…sanostol would still have the same issue.
i think the second license exist only for indie.
Edited by sanostol - June 10, 2019 18:14:23
Technical Discussion » ANIMATION NOT UPDATING WHEN FLIPBOOKING
- sanostol
- 575 posts
- Offline
Same here, had to flipbook it in 16.5. I was using the openGl rop and the viewport flipbook.
Houdini Indie and Apprentice » Edge Details in Voronoi Fracture.
- sanostol
- 575 posts
- Offline
You have to divide You voronoi fracture pieces with for example divide(bricker) to get some geometry to work with, than You add noise to the inside points
to the border You limit the noise to be just tangential by eliminating all parts noise that is parallel to the normal.
it is tricky with hard edges,though,You can drive the amount of noise additionally by mask maps,the fix problematic zones.
EDIT: Found an old quicktime, reminded me,that You also have to take care of texture coordinates
to the border You limit the noise to be just tangential by eliminating all parts noise that is parallel to the normal.
it is tricky with hard edges,though,You can drive the amount of noise additionally by mask maps,the fix problematic zones.
EDIT: Found an old quicktime, reminded me,that You also have to take care of texture coordinates
Edited by sanostol - May 25, 2019 03:27:51
Houdini Lounge » SideFX plans for Windows-Linux
- sanostol
- 575 posts
- Offline
The main issue I have with dualboot is licensing. Unfortunaley linux and windows on the same machine generate different server codes.
not a problem with floating licenses, but with nodelock this is at least unfortunate
not a problem with floating licenses, but with nodelock this is at least unfortunate
Houdini Lounge » AMD GPUs? Experiences?
- sanostol
- 575 posts
- Offline
aha, that's cool to know,thanks for the tip.
how does it work on animation? what happens on pure cpu base rendernodes
how does it work on animation? what happens on pure cpu base rendernodes
Edited by sanostol - Jan. 23, 2019 06:35:01
-
- Quick Links