Is there a way to override the entire pdg network to batch run in-process?


The hython spin-up overhead is way too much for many simple things that cook instantly like writing a CSV file, especially for large scenes with lots of HDA libraries and disk dependencies this can take several minutes just to write a file that should cook instantly


Furthermore I have run into several scenarios where everything cooks completely fine when running in-process like manually hitting render on a rop node but errors out completely when trying to run as a batch, with almost no error messages or log other than ‘task failed’. In all cases it involves importing alembic files using @pdg_attributes in the alembic sop filepath generated from a file pattern node, running through sops & writing geo back to disk + writing out csv files (gamedev csv exporter via rop fetch)


I tried using a python processor duplicating the upstream work items with inProcess=True flag but that either A)fails outright with no log, just State:Failed in the task info, or B) immediately hard crashes houdini, 50/50. The script works fine without the in-process flag.

As a side note how can you debug python processors that fail this way? There are no error messages, exceptions or logs it just says state:Failed, no log generated



The other errors I am getting are from a rop fetch pointing at the gamedev csv exporter writing a csv file for each alembic file -
Error rendering child: /obj/topnet1/ropnet1/rop_csv_exporter1/shell1

It renders fine if I go through manually clicking each work item and hitting render on the csv export rop, but nothing Ive tried will make it render in batch.

Thanks in advance for any help

Yeah, but as far as I know, there's no way to run ROP Fetch/HDA Processor/Hython in-process, so I write a lot in-process node with Python Processor and saved as TOP node hda on my own. like import image as HeightField geometry/output HeightField as Image, split geometry to workitem by attribute, etc. With Python Geometry class and Python SOP Verb I guess you can write many PDG version SOP easily, although I think it's should've done by SESI but sadly not. And what's more, with Invoke TOP node you can use some compiled compatibility SOP to run in-process, you can check the Partition by Bounds TOP node to see how to work with the Invoke Node.
Still, a lot of things can't run in-process easily or I just can't figure it out, like export geometry as fbx, or work with non-compiled compatibility SOP.
I think a lot of these functions really should offer by SESI but overall it's still very early stage for PDG. I hope the upcoming Solaris won't like these.

Neon Junkyards
Is there a way to override the entire pdg network to batch run in-process?

The hython spin-up overhead is way too much for many simple things that cook instantly like writing a CSV file,

*

The python script node has a toggle for evaluating it in-process.
It would be great to have such toggle for the rop geometry output node and other outputs nodes.

Hey there,

Couple of things to address here. First, it's unfortunately not possible for many things (such as, but not limited to, RopGeometry) to run in process. This is because of the fact that Houdini is in general not thread safe. We allude to some of problems here:

https://www.sidefx.com/tutorials/pdg-core-concepts/ [www.sidefx.com]

time 39:20

If you must use python script node (as opposed to python processor, which is always out of process), there are some tips here on how to enforce thread safety, basically by running the workitems in serial:

https://www.sidefx.com/forum/topic/68479/ [www.sidefx.com]

Now that said, there are ways to run stuff in process safely - chiefly through the invoke node to run a chunk of compiled sops, which is thread safe. We are also doing some things in H18 to make things better in this regard. There will be a native geometry attribute, that can be inherited from workitem to workitem, that can be handed off to the invoke node. We are making HDA processor pooling work better, which will also eliminate the spin up cost of hython in many cases.

Lastly, WRT to the gamedev csv exporter not working with PDG - in general I'm afraid the gamedev toolset is not well tested against PDG. We can't recommend their use in combination at this point. If you could please upload a sample with that problem, it would help us track down some of these issues, and hopefully get to a point where we can recommend their use in combination.

kenxu
Hey there,
There will be a native geometry attribute, that can be inherited from workitem to workitem, that can be handed off to the invoke node. We are making HDA processor pooling work better, which will also eliminate the spin up cost of hython in many cases.

Hi, Ken:
Will there be a native FBX Output TOP Node that run in-process in Houdini? It is so commonly used in Game Dev.

We have an FBX ROP, and you could just ROP Fetch it … why not just use that one? Are there concerns beyond performance?

kenxu
We have an FBX ROP, and you could just ROP Fetch it … why not just use that one? Are there concerns beyond performance?

The ROP Fetch node is really slow to cook, as it's mentioned in this topic. There should be some node run as fast as the Geometry Import node for just saving out the geometry as FBX.