*edit: deleted 2 questions from the list which I've found a solution for and added a couple more of new questions

Hello,
PDG tech is so intriguing yet so exotic for me, apologies in advance if you find the following questions trivial:

1. Why import geometry node is soooo much faster (like 400 times faster) than ropgeometry node in writing files to disk? In my system it takes geoimport node 1/10 of a second to write down to disk 10 files, while it's 40" for the ropgeo to accomplish the same task. Geoimport is set to evaluate the workitems attributes.
   
2. In a ‘Python SOP’ node I was trying the following code:
   

   topNode = node.inputs()[0].node("test")
   print topNode.getPDGNode()
   

   The returned result was quite exotic, something like “@G:”, varying on each cooking.
   Later I started to get this error instead:

   TypeError: Unable to convert function return value to a Python type! The signature was (self: _pdg.Node) -> unicode
   

   Is getPDGNode() function only usable in TOPs context?
   
3. Functions like getSelectedWorkItem(), setSelectedWorkItem(idx),etc don't use or return the @pdg_index. I guess they require the unique ID of the workitem, right? Is there any way to know the @pdg_index from the unique ID and vice-versa, so that I can use the aforementioned functions?
   
4. Are you considering to have the NULL node working as holder of the upstream workitems? It would be very handy to be able to select the workitems on a NULL, especially when its input is a switch node. Also being able to set the render flag to it would help. Otherwise, is there any topnode that I can safely append to work as output to display the geometry in SOPs? I'm using a blank python script for now.
   
5. If you execute a ropgeometry node without specifying the sop path ('use external sop' toggle is checked), it would cook normally without writing any file to disk, with no errors. Error would show up only later, when you try to ‘View Task Output’ as houdini is unable to find any file to load. This is potentially time wasting in the scenario where you want to actually use the file node inside the ropgeo subnet, but you forget to uncheck the ‘use external sop’ toggle. I assumed that when the sop path is left blank, the node would automatically use the file node inside as source.
   

thanks!

6) How to access in SOPs the ‘Raw Geometry’ stored by a geoimport node? My understanding is that Raw Geometry is just a cache of the file which is not being saved on the disk. But it doesn't show up in the topnet geometry spreedsheet. Sometimes I just need to read the geo on the fly. Writing it to disk would increase the latency of the operation.

7) Is there any way to avoid to write all the grouping information? This can be an overwhelming amount of unnecessary attributes, especially in the scenario where a single workitem corresponds to a single packed geometry (you dont really need groups as you can use point index)

8) Where is addItemToPartition(work_item, attrib) function documented? I can't happen to find it in the PDG API.

9)Don't know what I'm doing wrong, but I'm experiencing the following issue:
if I try to add a workitem attribute to the actual geometry in a geoimport node, the value would be the same for each file when the source geometry class is set to point.
Instead, the attribute value will be correctly written when the source geo class is set to detail.
The problem with the latter is that, being it a detail attribute, the geometry spreadsheet of the topnode would merge all the values into one single detail attribute.



I produced an example file

Hi Andr,

Sorry for the late reply on this topic. There is quite a number of issues to address and it took us a little bit to sync up on all the issues. Anyway, answers / further discussion below:

1. This likely due to the overhead of spinning up a hython session. The ROP geometry is running out of process, whereas the geometry import node can depending on its setting, be running in process.

2. This is either a bug or due to the fact that PDG node does not exist until TOPs graph gets cooked in some way to generate it. If you do Tasks -> Generate Static Workitems, and try again - does it still happen? If so, it's a bug and please upload a scene. If not, then it's due the the previously mentioned artifact - PDG is kept separate from TOPs so it can be detached in the future from Houdini and deployed completely independently.

3. Yes, those functions require the unique ID of the workitem. Index is not enough because it is not unique.

4. Consider using a merge node, along with a file SOP pointing to `@pdg_output` to visualize the output of a workitem or else use `@pdg_input` to visualize the input to a workitem, when you select the workitem.

5. I've logged Bug 98079

6. Only the invoke and geometry import can access the raw geometry at the moment.

7. Not at the moment. I've logged RFE 98081 to track it.

8. You can find it documented here: https://www.sidefx.com/docs/houdini/tops/pdg/PartitionHolder.html?origin_team=T045BRUUR [www.sidefx.com]

9. Likely a bug, we'll look into it further. Let us know if this is urgent, we'll try to look at it sooner in that case.

Hi, Ken:
Can we create our own In Process TOP nodes? I can see the Python Script TOP node is somehow running in process, but what if I want to manipulate the geometry a little bit, I need to use an HDA Processor or the ROP Geometry Output node which is not “in process” and slow to cook.

Thanks!

Hi Eric,

You can use the invoke node to run any SOP compile block in process. That's probably the best bet in terms of running something in process in order to manipulate geometry.

The other way to run something in process, as you know, is the Python script node. At some point in the future we'll get a C++ API to PDG, and that'll allow you to write more in process code - however, even then, it's unlikely you can just call into any geometry code - not all our geometry code is thread safe. Those that have been made thread safe are already available as SOP verbs and you can put those in compile blocks and already call into them via the invoke node as mentioned above.

Hi Ken,
Thank you for pointing out the Invoke node, and it looks interesting. But after playing it for several hours, I still can't figure it out, a simple question which has stuck me for a long time, how to create a HieghtField Import/Export TOP node runs as fast as the Geometry Import TOP node and does the job just like the HF_Import/Export SOP node?

I've attached an example of the invoke node. The example fetches a box from SOPs using the geometry import and feeds it into an invoke which subdivides the geometry and applies some noise in a compiled block.

The invoke is currently the only way to do in-process geometry processing in TOPs, however unfortunately I don't think the HF file node can be used in a compiled block at the moment.

In general TOP nodes can't access SOPs when cooking because TOPs cooks in parallel/off the main thread. The Geometry Import is able to load geometry from SOPs during its static work item generation step, but it currently can't do that dynamically or at work item cook time for the aforementioned reasons. We're working on improving our geometry handling and one of the top priorities is to remove that restriction. As you mentioned, for small tasks the HDA Processor and ROP Geometry nodes are often too heavy.

You could try using a Python Script TOP node and the HOM API to achieve what you want, e.g by getting a reference to a HF file node, accessing it's geometry as a hou.Geometry and saving it out with hou.Geometry.saveToFile(file_name). Note that there may be issues with that approach if you make changes to your SOP network while the TOP network is evaluating those work items, for example.

Hello Ken,
no worries and thanks a lot for caring to answer!

1. This likely due to the overhead of spinning up a hython session. The ROP geometry is running out of process, whereas the geometry import node can depending on its setting, be running in process.

Understood. It would nice to have a toggle to set the Rop Geometry to work in-process, for those situations when you don't really need multi-threading, but only a fast output to disk of few workitems.

2. This is either a bug or due to the fact that PDG node does not exist until TOPs graph gets cooked in some way to generate it. If you do Tasks -> Generate Static Workitems, and try again - does it still happen? If so, it's a bug and please upload a scene. If not, then it's due the the previously mentioned artifact - PDG is kept separate from TOPs so it can be detached in the future from Houdini and deployed completely independently.

Yes, even when only generated, It still shows the behavior: a series of weird chars 0bZ 0bZ @dZ and then the aforementioned error code. I attached the scene

3. Yes, those functions require the unique ID of the workitem. Index is not enough because it is not unique.

Now I realize that @pdg_index is different than work_item.index! And It makes perfectly sense. So we have 3 types of indexing in TOPs.
Now, given a topnode name and a @pdg_index it should be possible to retrieve the unique ID. But how?
As I'm building a hda Sop wrapper around a topnet, my goal is to be able to use an integer parameter to display a specific pdg_index of a specific topnode (using the function setSelectedWorkItem(idx)).
It's like a remote control, because the user won't be able to dive inside the topnet to manually select the workitem dot on the topnode.

Even better, it would be handy to have a new type of parameter with the ‘dotted UI’ similar to the topnet nodes. I guess for now one could simply use a chain of unlabelled parameter toggles, that would spawn in accordance to the amount of workitems of the topnode of choice.

6. Only the invoke and geometry import can access the raw geometry at the moment.

What I meant is: when I select the topnet at sop level, I expected to see the raw geometry information displayed in the geometry spreadsheet. So I could append some sop nodes to the output connector of the the topnet network. Now it's only possible when the topnet save files to disk.
I tried to use the invoke top node, but with the ‘write geometry’ unchecked it won't output any geometry at sop level.

Also, what is the purpose of the output node inside TOPs?
It doesn't work like in a normal subnetwork: If I add a second output node, no second outputs connectors would be spawned on the topnet node.
It would be nice to have multiple outputs from a topnet. In case of topnet with separate branches that I want to output at Sop level, I can only use the force-recook button and manually set the render flag to the branch I want to see at sop level. If I need to see both branches, I should split them in two different topnets.

8. You can find it documented here: https://www.sidefx.com/docs/houdini/tops/pdg/PartitionHolder.html?origin_team=T045BRUUR [www.sidefx.com]

Oh thanks, I couldn't find it when I searched through the main page. I think it's not indexed in the menu

9. Likely a bug, we'll look into it further. Let us know if this is urgent, we'll try to look at it sooner in that case.

Not super urgent, but would be great if you could just check the .hip file [www.sidefx.com] and confirm that it's a bug and nothing due to my inexperience with PDG that could easily be fixed by applying the correct workflow.

Anyway, this issue, along with issue 1) and issue 6) are somehow correlated in my scenario.
My goal is: output at Sop level some points with per-point pdg attributes created inside the topnet. It's very light task, no geo manipulation, only pdg attributes handling and assigning to geo.
Super ideal for me would be being able to do 6): read at Sop level the Raw Geometry from the topnet output, without wasting time in writing it first to disk and then re-import it.
Since it's not possible, I tried to use the Rop Geometry Output, and here it comes issue 1), the slowness of out-process writing to disk.
So I tried to use the faster in-process GeometryImport saving capabilities, but issue 9) will make the geoimport node write pdg attributes with fixed values on points.

As of now I can only use Rop Geometry Output, which is a no go.
In my scenario the user of my Digital Asset needs to play with some parameters of the hda, every parameter change would trigger a fast in-process cooking in the topnet, whose output (points with specific attribute values) is to be visualized in the Viewport. This workflow needs some fast feedback from the topnet as the user changes parameters.

I guess it's wiser to try to port all the attributes creation task at SOP level, and I will use TOPs for the very last part when the actual render happens.

Thanks again!