I got a draw curve based workflow where each curve will result in a computation heavy task worthy of bringing over to TOPs. As the geometry comes in as one detail the only way for pdg to find out what task data is dirtied is to compare to the input geo to that of last cook. Is this workflow possible? To diff the content of geometry, file or houdini geo, and dirty depending of the result?

PDG supports regeneration of work items between cooks. Normally work regeneration occurs when you cook the graph, and something about the node have changed since the last cook. The node will recreate its work items in-place using its standard work item generation logic. The changes are compared against the existing work items, and anything that modifies attributes or work items configuration causes those items to be dirtied. This includes changes to attributes, intrinsics like frame/index, or output files. Work items that are no longer needed are removed.

The Geometry Import and File Pattern have special implementations because they can create work items from external resources. It's also possible to write a custom regeneration implementation when writing a node, or when using a Python Processor.

I've attached an example file with three cases in. The first one is a Geometry Import which creates work items from points in a SOP network. Cook the example1 node. Then, go to the SOP network and modify the Group field on the Attribute Create SOP to include some extra points. Then, cook example1 again without dirtying anything (RMB -> Cook Node or Shift+G). Some of the work items will recook because items in the geometry import will be dirty from the changes made to the SOP node. You can also try doing an RMB -> Generate Node on example1 after modifying the SOP, which will more clearly show the modified work items.

The second one uses an attribute create in TOPS – if you modify for example the “Create When” condition and regenerate the work items using RMB -> Generate node in that branch, a similar thing will happen.

Finally, the third example has a Python Processor with a custom regeneration implementation. That overrides the built in behavior and allows you to dirty work items on a cook, based on the work items that already exist and the incoming work items from the input nodes. In the example file the node randomly dirties existing work items, but you could run custom logic to determine when existing items should be dirtied or delete. That hook is run before the graph cooks on static work items. In the current version of Houdini it's not possible to do regeneration of dynamic work items because items can't be dirtied during the cook. This is something that's going to be addressed in a future release.


In your specific case you may be able to use the Geometry Import, or you could use a custom implementation that compares the geometry files directly.

That's really great, I didn't know it was so clever out of the box.

Maybe this can be helped by making it visible a bit better. I see that on changes the node icon goes dark green and thus hints that something needs to be updated but it's a bit unfortunate that the individual task dots are still light green. Another thing, if I revert a change and thus restoring the state of the previous cook the node icon doesn't go light green again.

Last example is a bit crazy but I understand the demo purpose of it- it's possible to do some dark magic dependency management in the background.

I'm failing of making use of this. Attached is a scene that doesn't react to the updates. I move the primitives but the importgeometry isn't recooking it's tasks.

I also don't understand how I should cook this in the middle of my sop chain? And how do I merge the files together. I've used the topgeometry node as it looks like that should contain both of these features but I'm having chrashes and it recooks all tasks even when I just do UI changes. (Submited as #105238)

kahuna031
Another thing, if I revert a change and thus restoring the state of the previous cook the node icon doesn't go light green again.

That dark green tint indicates that a change occurred, either to the node itself or to an input node, and that the node may dirty work items on the next cook. It doesn't remember the original parm values – the TOP node is stale until the next time its cooked and won't go back if you undo the change. This is using Houdini's regular mechanism for marking nodes as needing to update. For example, if you put Houdini in Manual update mode, change a parm on a SOP node and then change it back, the SOP will still recook once you put Houdini back into Auto Update mode. The same is true for TOPs – once the parms have been modified the node is dirty, and will do the work of regenerating work items the next time that node cooks.

kahuna031
I'm failing of making use of this. Attached is a scene that doesn't react to the updates. I move the primitives but the importgeometry isn't recooking it's tasks.

It appears to be working as expected for me. If I modify the Edit node, then do an RMB -> Generate Node on the Geometry Import, the corresponding work item for that prim is dirtied. I can't cook your HDA Processor as it's referencing an HDA on your machine, but I suspect maybe the issue is that your HDA Processor is cooking from cache? Trying setting the cache mode to Write files.

The TOP Geometry is the same as TOP network, but it cooks blocking like a SOP and any work items in the display node have their geometry passed to SOPs when the TOP Geometry finishes cooking. To merge geometry files, you can use the Geometry Import node. If the source geometry in SOPs, you should merge it there.

It was probably the caching that got me. Partial updates works now. Cooks was probably too fast for me to even notice when it just read the cache. So why is it using the cache this way?

As the topgeometry is not working, is there a way to get the geometry back in a similar fashion when using the top network? Should I for-loop a file node?

kahuna031
It was probably the caching that got me. Partial updates works now. Cooks was probably too fast for me to even notice when it just read the cache. So why is it using the cache this way?

PDG's caching system is based on files. The Automatic caching mode means that if the expected output files are found on disk, then the work item will cook from the cache file. Write Files means it always overwrites the cache files on disk, and Read Files means it fails if there's no cache file to read. We've been making some improvements though, including more consistent behavior when upstream work items are modified (see details here: https://www.sidefx.com/forum/topic/73032/)

Dirtying cache files when attributes are modified upstream is technically possible as well, but may not always be the desired behavior. To do that properly PDG would need to track each of the attributes actually used by a work item, not just the ones that it actually has. Some attributes are just copied through the work item from their parent and then used later downstream. For example, the first node in the graph might be an attribute create or wedge used for configuration purposes, or something that loads a configuration file from disk.

kahuna031
As the topgeometry is not working, is there a way to get the geometry back in a similar fashion when using the top network? Should I for-loop a file node?

You can use a SOP for loop to load in the files, or you could also try using the File Merge SOP. That's mainly used for loading in distribute sim slices, but if you're writing out a sequence of files with consecutive numbers in the name, it can be easier to use than a loop.

I'm looking into the TOP Geometry issue you logged, although I'm unable to reproduce the crash on my end at the moment.

Thanks! So now I got a working setup. File merge is working well and by sending off python commands between SOPs and TOPs I can get it to update interactively. The topgeometry node will be welcome addition as my setup looks quite useful but extremely clunky.

Next step is to control the relationships by an attribute, guess I need a python mapper to do that? I got a feeling that too can be a fairly generic desire, maybe worth to consider for the topgeometry workflow?

Hi guys,

I try to follow example what tpetrick create, but it doesnt work. When i try change group and regenerate TOP node "example1", it make automaticaly dirty all processes even if just one point change and others stay still.

This doesnt work :

I've attached an example file with three cases in. The first one is a Geometry Import which creates work items from points in a SOP network. Cook the example1 node. Then, go to the SOP network and modify the Group field on the Attribute Create SOP to include some extra points. Then, cook example1 again without dirtying anything (RMB -> Cook Node or Shift+G). Some of the work items will recook because items in the geometry import will be dirty from the changes made to the SOP node. You can also try doing an RMB -> Generate Node on example1 after modifying the SOP, which will more clearly show the modified work items.

Is there some new updates what change behavior of PDG ?