Simplest way to add a dependency in the Python TOP

   396   6   0
BitreelTD
Member
19 posts
Joined: Dec. 2019
Offline
I'm struggling with figuring out how to get this working. Ideally, each of the tasks that this Python node would generate would simply wait for the previous one that was added before it to finish.

I tried putting this in the dependency, and getting this error. Where did I go wrong?

Attachments:
internal_dependencies.jpg (1.1 MB)

User Avatar
Staff
548 posts
Joined: May 2014
Offline
Node callbacks are only permitted to access the local variables passed into them. In this case, the callback is only allowed to add dependencies between the work items from internal_items list.

This is especially important if your node is dynamic. A dynamic node generates work items each time an input work item cooks, and it might be doing that in parallel. Therefore, the list of work items on the node itself may be incomplete at any point in time.

It's important to note that onAddDependencies is called each time the node generates work item, not once per node. For example if your node is dynamic and has 10 input itemss, onGenerate will be called 10 times (once for each cooked input work item). onAddDependencies will also be called 10 times as well, once for each of the lists of work items produced by the corresponding onGenerate call.

You will need to change your Generate When parameter to "All Upstream Items are Generated" or "All Upstream Items are Cooked" if you need to access all work items at the same time.
BitreelTD
Member
19 posts
Joined: Dec. 2019
Offline
Thanks, that does make sense. But doesn't give me the solution I'm looking for.

For this dynamically generated node. I want to form an internal dependency so that each item is dependent on the last item that was added. Where would I add this code, in the "on generate" I take it? And how would this dependency holder be constructed?
User Avatar
Staff
548 posts
Joined: May 2014
Offline
In order to do that, you need the node generate all of it's work items in one go.

If you set "Generate When" to "All Upstream Items are Cooked", then your node will generate all its work items with a single onGenerate call, once all upstream work items are cooked. It will then invoke onAddInternalDependencies once as well, on the full list of generated work items, which will allow you to add the dependencies you need.

If your node only needs the inputs to be generated in order to create its work items, then you can use "All Upstream Items are Generated" instead. But you have to use one of those options to force the node to do it's generation/internal dependencies for the whole node at once.
Edited by tpetrick - Jan. 24, 2023 17:37:54
BitreelTD
Member
19 posts
Joined: Dec. 2019
Offline
I might be after a different workflow all together then. I've had this come up multiple times for various tasks. Creating a directory, cleaning a directory, creating a file.

In these instances, I might have 100 tasks, but I only want this operation done once. The reason for this is that other conditions might not be ready before these tasks will run.

Here would be a scenario.

I want this Python TOP to write files to a directory
This directory should be cleaned before the python TOP runs
I don't want to wait for all of it's parent tasks to complete before running the first task, I also don't want each task to be deleting the outputs of other tasks.

For put more simply:
I want all files in this folder to be deleted before this top is allowed to run, but, there is no need to wait for all of it's parent tasks to complete to do this function.

Is there some sort of a pre-run area that can be accessed for a top/pdg node? Similar to the pre-render script of a rop?
User Avatar
Staff
548 posts
Joined: May 2014
Offline
Processor nodes have a Pre Cook callback which might do what you need, but it isn't exposed on the Python Processor TOP: https://www.sidefx.com/docs/houdini/tops/processors.html#onprecook-pdg-result [www.sidefx.com] It's currently only available if you write a custom node, but we can easily promote it up onto the Python Processor if you think that would be useful. Please file an RFE for that.

I do have a follow up question to your use case though. What happens if you cancel the cook with half of the work items completed, and then cook the node again to trigger the remaining work items? It sounds like you'd probably NOT want to run the pre cook logic in that case, even though it's technically a new cook, otherwise it would delete the outputs produced by the work items that were already cooked. So what you'd actually need is a "pre cook, but only before any work items cook" operation.

It might be tempting to assume that's the same as running the pre cook as part of the first work item, but work items don't necesasrily cook in order. The first logical item work in the node may infact be the last one to start cooking, depending on when its input tasks actually finish. Or the first one might fail, and then be cooked a second time at a later point due to scheduler retry settings, etc.

Adding a dependency between the first item and the remaning items in the node would fix that, as you were trying to do, but it would mean that any change to the first work item's state would effectively dirty the whole node. And every other work item in the node would end up having to wait on the first work item, which could end up being the same thing as waiting for all inputs to cook depending on the cook order of the input work items.

Generally speaking, PDG is designed to encourage work items that operate on local state only, i.e. the attributes and files associated with the work item. That way things in the same node can be run in parallel and in any order, once their dependencies are satisfied.

In other words, I'm not sure if there's a way to do exactly you want, depending on what's expected in the case were a node is only partially cooked and resumed. I think adding in pre/post hooks or a pre/post node is valid RFE, but the exact behavior needs to be well defined.
Edited by tpetrick - Jan. 25, 2023 10:59:35
BitreelTD
Member
19 posts
Joined: Dec. 2019
Offline
This sounds like a great solution. I brutally hacked a solution, I'll post it once it's sane, as well as do an RFE But probably the best way is the way you've described.
Edited by BitreelTD - Jan. 25, 2023 12:51:44
  • Quick Links