Hi Andr,
So there are couple of issues here. The first problem is caused by the default setting on the Python Script node. The automatic setting here is putting the node in “dynamic” mode by default, where it should have been more consistent and put it to “static”. We'll make the change on the default, but for now you could either fix that by explicitly setting “static” on the python script node, or else turn on the “dynamic partitioning” setting on the partitioner node if you elect to keep the python script node “dynamic”.
The second issue is caused by the fact that when we have multiple workitems with the same attribute name, then the partitioner will elect to keep only 1 of the attributes as its own. You can get to choose which one through the sorting options in the advanced tab of partitioners.
In the longer run, we plan to have a specific node for this case. you'd wire it after the partitioner, and it exposes a bunch of merging settings sort of like the attribute promote sop. So you'd partition, then put down the “attribute promote” node which handles the merging of data from the items in each partition.
For now, your solution works because it orders the workitems in a way so that the one with the right attributes arrives first. Another way to solve this problem, for now, is to put a python processor after this partitioner and iterate through the upstream_items, which are partitions, and access their partitionItems and do whatever is needed with the data.
Found 132 posts.
Search results Show results as topic list.
PDG/TOPs » How-to combine a variable amount of geometryimport?
- kenxu
- 544 posts
- Offline
PDG/TOPs » TOP's hda's and presubmit scripts
- kenxu
- 544 posts
- Offline
This is another artifact from the fact that TOPs is just a wrapper for PDG. When you first open the file, the PDG network is not created yet. That's why that first call fails. If you run
executeGraph(filter_static, block, generate_only, tops_only)
on the TOP node with the “generate_only” flag, it will cause the PDG graph to be generated and thus the rest of the script to succeed.
executeGraph(filter_static, block, generate_only, tops_only)
on the TOP node with the “generate_only” flag, it will cause the PDG graph to be generated and thus the rest of the script to succeed.
PDG/TOPs » PDG/TOPs grounded - Open letter for SideFX
- kenxu
- 544 posts
- Offline
Hi Eric,
We are sorry to hear that your experience with PDG has not been smooth. Since launch, working with our clients, we have had promising successes but also cases where it's not quite where we want it to be. We have made an effort to be as transparent as possible with its current state, and freely acknowledge where we are still short, with a commitment to continuously strive to close the gaps where they exist.
With that said, while we are aware of some problems with PDG, instability and crashing all over the place is not one of them. We have a quite a number of clients actively testing and using the tech as this point, some pushing it quite hard, so we'd be quite surprised if there was some systematic instability with it that none of them informed us of. The main problems with PDG currently, as far as we are aware, are:
- UI / UX
https://www.sidefx.com/forum/topic/67311/ [www.sidefx.com]
- Peripheral stock nodes not having enough functionality (we have improved some, especially around CSV, but still probably not enough)
- Deployment issues around default farm bindings to Tractor and Deadline, with many issues already fixed such as the submitting too many jobs issue and firewall complications.
- Training. Still a relative lack of training material, especially around key core concepts to work well with the system. Master class(es) coming up.
I wonder whether some of what you're experiencing has to do with issues like this raised, where PDG was used in a way that's not really supported:
https://www.sidefx.com/forum/topic/67118/?page=1#post-285971 [www.sidefx.com]
That's still our fault, by the way, because we still haven't gotten the training material out on these “not supported” gotchas. All of this is not to say that your problem isn't real, nor is it an attempt to be defensive about this - it's entirely possible that you have found cases where PDG is not behaving well. If you could please provide us a test scene (perhaps in a different thread, so we can focus on just the technicals of that), it would help us track it down. We can also take up the HDA Processor slowness there as well, there are some features to mitigate this.
We are sorry to hear that your experience with PDG has not been smooth. Since launch, working with our clients, we have had promising successes but also cases where it's not quite where we want it to be. We have made an effort to be as transparent as possible with its current state, and freely acknowledge where we are still short, with a commitment to continuously strive to close the gaps where they exist.
With that said, while we are aware of some problems with PDG, instability and crashing all over the place is not one of them. We have a quite a number of clients actively testing and using the tech as this point, some pushing it quite hard, so we'd be quite surprised if there was some systematic instability with it that none of them informed us of. The main problems with PDG currently, as far as we are aware, are:
- UI / UX
https://www.sidefx.com/forum/topic/67311/ [www.sidefx.com]
- Peripheral stock nodes not having enough functionality (we have improved some, especially around CSV, but still probably not enough)
- Deployment issues around default farm bindings to Tractor and Deadline, with many issues already fixed such as the submitting too many jobs issue and firewall complications.
- Training. Still a relative lack of training material, especially around key core concepts to work well with the system. Master class(es) coming up.
I wonder whether some of what you're experiencing has to do with issues like this raised, where PDG was used in a way that's not really supported:
https://www.sidefx.com/forum/topic/67118/?page=1#post-285971 [www.sidefx.com]
That's still our fault, by the way, because we still haven't gotten the training material out on these “not supported” gotchas. All of this is not to say that your problem isn't real, nor is it an attempt to be defensive about this - it's entirely possible that you have found cases where PDG is not behaving well. If you could please provide us a test scene (perhaps in a different thread, so we can focus on just the technicals of that), it would help us track it down. We can also take up the HDA Processor slowness there as well, there are some features to mitigate this.
Edited by kenxu - July 3, 2019 17:14:25
PDG/TOPs » Modify attributes via Python Script node
- kenxu
- 544 posts
- Offline
Hi there,
You can just write something like:
work_item.data.setInt(“blueCorrect”, 1, 0)
You can get the PDG API if you search for “pdg” in the help. Note also the little drop down on the side, these are quite helpful - they contain some sample code for common operations.
You can just write something like:
work_item.data.setInt(“blueCorrect”, 1, 0)
You can get the PDG API if you search for “pdg” in the help. Note also the little drop down on the side, these are quite helpful - they contain some sample code for common operations.
PDG/TOPs » PythonProcesser get relative sopNode path
- kenxu
- 544 posts
- Offline
This is due to the fact that PDG exists as something quite separate from Houdini, and that TOPs is only a Houdini wrapper that goes over PDG. The Python code you write is executed in the PDG framework, and not Houdini. A consequence of this is that when the Python code runs, it doesn't know where “here” is relative to the Houdini node hierarchy. However, if the PDG node is created from a TOP node, it does know the TOP node's session id. With that session id you can look up the TOP node, and then go from there to reference other nodes.
Details are noted in this post here:
https://www.sidefx.com/forum/topic/67324/ [www.sidefx.com]
The reason things are this way is that ultimately we (and indeed the community) would like PDG to be able to run completely independent of Houdini, and so it is architected as such. The downside of this however is annoying artifacts like this, where things don't work quite the way you'd expect.
Details are noted in this post here:
https://www.sidefx.com/forum/topic/67324/ [www.sidefx.com]
The reason things are this way is that ultimately we (and indeed the community) would like PDG to be able to run completely independent of Houdini, and so it is architected as such. The downside of this however is annoying artifacts like this, where things don't work quite the way you'd expect.
PDG/TOPs » Making TOPs less confusing
- kenxu
- 544 posts
- Offline
My, a lively discussion around this thread
Ok, some notes from our earlier meeting on what to do about the usability issues. This is not the whole list, but just what's relevant to the discussion on this thread.
1. All agreed that the current cooking mechanism in TOPs is a bit hidden, and should be improved. We thinking about maybe a tool bar or control panel HUD … the exact mechanism is TBD, but something highly visible with some high level status and controls, like “cook the graph” or “cancel the cook” etc.
2. Yes, we should have a TOP context, and we will do it. We just didn't have time at the end of the last release.
3. Creating a TOP network inside a ROP network doesn't have the local scheduler by default - this is just a bug, and will be fixed.
4. Terminology & UI layout - we will continue to study those and incrementally improve as needed.
Roadmap: the bigger feature currently being developed is the ability to separate the compute of PDG from the UI. Once we are done with this, you will be able to submit a graph even during it's cook to the farm, and then attach to that headless session to visualize it's state.
Ok, some notes from our earlier meeting on what to do about the usability issues. This is not the whole list, but just what's relevant to the discussion on this thread.
1. All agreed that the current cooking mechanism in TOPs is a bit hidden, and should be improved. We thinking about maybe a tool bar or control panel HUD … the exact mechanism is TBD, but something highly visible with some high level status and controls, like “cook the graph” or “cancel the cook” etc.
2. Yes, we should have a TOP context, and we will do it. We just didn't have time at the end of the last release.
3. Creating a TOP network inside a ROP network doesn't have the local scheduler by default - this is just a bug, and will be fixed.
4. Terminology & UI layout - we will continue to study those and incrementally improve as needed.
Roadmap: the bigger feature currently being developed is the ability to separate the compute of PDG from the UI. Once we are done with this, you will be able to submit a graph even during it's cook to the farm, and then attach to that headless session to visualize it's state.
Edited by kenxu - June 27, 2019 15:58:13
PDG/TOPs » Making TOPs less confusing
- kenxu
- 544 posts
- Offline
Thank you guys for this very useful feedback. A lot of it is very much on point. A master class (in fact, one of several) is on its way. Our usability folks are also taking this input and discussing it. We'll update here once we've reached a conclusion.
Edited by kenxu - June 27, 2019 16:31:48
PDG/TOPs » Ability to set a PDG node as cooked via python.
- kenxu
- 544 posts
- Offline
You can trigger a cook with the cook method on pdg.GraphContext or the cook method on pdg.Node. However, I don't think you can just change the state through the API without actually cooking it…why do you need this?
PDG/TOPs » How to lock a TOP Node?
- kenxu
- 544 posts
- Offline
Currently locking is not supported on TOPs / PDG. You can bypass, but not lock. I've logged RFE 97695 to capture it.
PDG/TOPs » TOP's hda's and presubmit scripts
- kenxu
- 544 posts
- Offline
Hi Andrew,
So I can answer at least the first part of your question, which is more straightforward. Yes, we can create assets in TOPs. Select a few nodes, make a subnet, and go ahead and create digital asset - it is no different than any other process. When you insert such a an HDA in a TOP graph, it behaves like syntatic sugar - that is, the graph will evaluate as if the contents of the HDA is inlined into the graph. You can also create HDAs with a python processor, to better encapsulate and present the functionality of your custom processor. The difference between “save to digital asset” and “save to python script” is that one will be callable by “pure pdg” outside of TOPs (save to python script), and one will only be usable within TOPs (save to digital asset), but is friendlier to use.
The second part of your question is a bit trickier. There are no hooks within TOPs to allow some sort of script to be run just prior to file save, or prior to file submission. For a hook prior to file save, perhaps there is some mechanism in the standard hou python interface? We are thinking about some sort of pre-flight mechanism, so maybe there is some hook there we can expose…?
So I can answer at least the first part of your question, which is more straightforward. Yes, we can create assets in TOPs. Select a few nodes, make a subnet, and go ahead and create digital asset - it is no different than any other process. When you insert such a an HDA in a TOP graph, it behaves like syntatic sugar - that is, the graph will evaluate as if the contents of the HDA is inlined into the graph. You can also create HDAs with a python processor, to better encapsulate and present the functionality of your custom processor. The difference between “save to digital asset” and “save to python script” is that one will be callable by “pure pdg” outside of TOPs (save to python script), and one will only be usable within TOPs (save to digital asset), but is friendlier to use.
The second part of your question is a bit trickier. There are no hooks within TOPs to allow some sort of script to be run just prior to file save, or prior to file submission. For a hook prior to file save, perhaps there is some mechanism in the standard hou python interface? We are thinking about some sort of pre-flight mechanism, so maybe there is some hook there we can expose…?
PDG/TOPs » Tracking maxmimum memory usage per work item
- kenxu
- 544 posts
- Offline
PDG/TOPs » How-to combine a variable amount of geometryimport?
- kenxu
- 544 posts
- Offline
Ah, ok. Here is one possible solution:
Step 1. Create a custom python processor to scan your SOP network and create 1 workitem per matching node. Attach the path of that node as an attribute, say for example “node_path”.
Step 2. Use a geometry import node to pull in the points using the above attribute. So `@node_path` . This pulls all 14 points in this case: 2 from OUT_0, 4 from OUT_1, and 8 from OUT_2. Each point is a separate workitem.
Step 3. Create a custom python partitioner to create partitions based on unique combinations of attributes A,B,C. We don't have an exact pre-created partitioner that does this exact thing (partition by combination comes close, but not quite), so has to be custom partitioner.
I've attached the solution file here. Hopefully that helps.
Step 1. Create a custom python processor to scan your SOP network and create 1 workitem per matching node. Attach the path of that node as an attribute, say for example “node_path”.
Step 2. Use a geometry import node to pull in the points using the above attribute. So `@node_path` . This pulls all 14 points in this case: 2 from OUT_0, 4 from OUT_1, and 8 from OUT_2. Each point is a separate workitem.
Step 3. Create a custom python partitioner to create partitions based on unique combinations of attributes A,B,C. We don't have an exact pre-created partitioner that does this exact thing (partition by combination comes close, but not quite), so has to be custom partitioner.
I've attached the solution file here. Hopefully that helps.
Edited by kenxu - June 21, 2019 11:09:46
PDG/TOPs » How-to combine a variable amount of geometryimport?
- kenxu
- 544 posts
- Offline
Sorry, it's still not quite clear to us what you're trying to achieve. Are you saying that you want to run a particular HDA a variable number of times (3 in the above case), then import the geometry of each run as points, then somehow combine the point attributes from the different runs?
PDG/TOPs » Split Logic
- kenxu
- 544 posts
- Offline
I think it should happen before the partition by frame node. Basically the flow is to:
1) partition by attribute to figure out which movie an image should be in
2) partition by frame so all the images within the same movie and that has the same frame number are grouped together
3) run image magick to make the per frame contact sheet
4) wait for step 3 to fully complete (wait for all)
5) assemble the movie with the full list of results from step 3
I don't think you need to get into for loops for this workflow.
1) partition by attribute to figure out which movie an image should be in
2) partition by frame so all the images within the same movie and that has the same frame number are grouped together
3) run image magick to make the per frame contact sheet
4) wait for step 3 to fully complete (wait for all)
5) assemble the movie with the full list of results from step 3
I don't think you need to get into for loops for this workflow.
Edited by kenxu - June 18, 2019 11:13:34
PDG/TOPs » issue when writing geometry
- kenxu
- 544 posts
- Offline
This looks to be an issue within Houdini itself. Your HDA appears to be triggering some functionality within Houdini that is crashing. This is probably not to do with PDG itself, but rather some other bits of Houdini functionality. If you could please attach the HDA and the data involved, we could track it a bit further to see exactly what is crashing.
PDG/TOPs » Embedded hda can't be processed by HDA processor?
- kenxu
- 544 posts
- Offline
Yup, we probably have to extract the HDA definition in this case before running the HDA Processor executable. Logged Bug 97478 to track it.
PDG/TOPs » Split Logic
- kenxu
- 544 posts
- Offline
Maybe you could have an attribute that tags which variation should go in what contact sheet, and then partition by attribute?
PDG/TOPs » PDG Debugging
- kenxu
- 544 posts
- Offline
Yes, that could make sense. Could go along with features to better visualize / organize the various outputs.
PDG/TOPs » PDG Debugging
- kenxu
- 544 posts
- Offline
Yeah unfortunately there is not much we can do about the fact that the info goes to a log file - when it's running out of process, it's being run in a different program than Houdini entirely, so capturing the information in a log file is pretty much the only thing we can do in that case.
PDG/TOPs » General PDG/TOPs Issues and a Sprinkling of Redshift
- kenxu
- 544 posts
- Offline
Glad to hear of the progress
Unfortunately, there is no correlation between CPU and GPU. We have no information on our end about the process to be spawned and what its performance characteristics are… the best thing to do is to look up / ask about the performance characteristics of the application to be spawned, then work backwards from that information to determine the right CPU per task setting for the most optimal results.
Unfortunately, there is no correlation between CPU and GPU. We have no information on our end about the process to be spawned and what its performance characteristics are… the best thing to do is to look up / ask about the performance characteristics of the application to be spawned, then work backwards from that information to determine the right CPU per task setting for the most optimal results.
-
- Quick Links