While you're browsing the forum, could you perhaps have a quick glance at this post : https://www.sidefx.com/forum/topic/74851/ [www.sidefx.com]
Looking for a reliable way to run a houdini instance where I'm constantly listening to input data.
Thanks!
Found 72 posts.
Search results Show results as topic list.
PDG/TOPs » Input items order, is it an unordered list?
- Johan Boekhoven
- 72 posts
- Offline
PDG/TOPs » Input items order, is it an unordered list?
- Johan Boekhoven
- 72 posts
- Offline
This makes a lot of sense, thanks it works fine now.
PDG is quite a beast to learn the ins and outs, but I will get there!
Cheers,
-Johan
PDG is quite a beast to learn the ins and outs, but I will get there!
Cheers,
-Johan
PDG/TOPs » Input items order, is it an unordered list?
- Johan Boekhoven
- 72 posts
- Offline
Hi @tpetrick,
If you look at the screenshot above you can see I've manually reordered the output files in a python node, they appear in correct order in the correct order with MMB debug window, but when I use @pdg_input.0, it gets the file that is alphabetically the first instead of the order as shown.
Here's a test file. Looking at it now I think somehow the python node that sort output items causes the real issue.
But after that node issues occur.
See attachment for the file.
If you look at the screenshot above you can see I've manually reordered the output files in a python node, they appear in correct order in the correct order with MMB debug window, but when I use @pdg_input.0, it gets the file that is alphabetically the first instead of the order as shown.
Here's a test file. Looking at it now I think somehow the python node that sort output items causes the real issue.
But after that node issues occur.
See attachment for the file.
PDG/TOPs » Input items order, is it an unordered list?
- Johan Boekhoven
- 72 posts
- Offline
Sorry for the bump.
I'd really like to know if we can have dependable resultdata order or not.
Thanks!
I'd really like to know if we can have dependable resultdata order or not.
Thanks!
PDG/TOPs » "While Loop", can we make TOP listen and act
- Johan Boekhoven
- 72 posts
- Offline
Ok, so I have found a solution, which I think could be improved upon.
This setup keeps a heartbeat runnig for 30 seconds and each second you can generate new work_items. You can obviously change number of heartbeats and wait time. So this allows you to fetch live data form a database or folder for example and act accordingly. There's no logic for tracking what was processed, you need to build that yourself. But it's at least a way to keep houdini polling.
Now in the file, I have also 2 python generators which I think would be the proper solution, but can't get to work.
I'd love to know if there's anything better of more robust perhaps.
Hope it helps someone and if you have feedback please let me know!
This setup keeps a heartbeat runnig for 30 seconds and each second you can generate new work_items. You can obviously change number of heartbeats and wait time. So this allows you to fetch live data form a database or folder for example and act accordingly. There's no logic for tracking what was processed, you need to build that yourself. But it's at least a way to keep houdini polling.
Now in the file, I have also 2 python generators which I think would be the proper solution, but can't get to work.
I'd love to know if there's anything better of more robust perhaps.
Hope it helps someone and if you have feedback please let me know!
PDG/TOPs » Python Object for attribute failed to deserialize
- Johan Boekhoven
- 72 posts
- Offline
There's a video, I think when PDG was released that goes into Maya/PDG. If I recall correctly that as soon as the server is running all python code is running in the PyMel env of Maya. So any special data you cary into maya will not just work out of the box I think. I'd say you need to convert the PyObject to something more basic before pushing it into maya server, like a basic dict or list.
Again never had to do it, so I might be way off.
edit : https://www.youtube.com/watch?v=t0bhiCoDR8M [www.youtube.com]
that's the video I had in mind, lot's of info on maya/pdg
Again never had to do it, so I might be way off.
edit : https://www.youtube.com/watch?v=t0bhiCoDR8M [www.youtube.com]
that's the video I had in mind, lot's of info on maya/pdg
Edited by Johan Boekhoven - July 24, 2020 07:35:01
PDG/TOPs » ResultData vs Attributes
- Johan Boekhoven
- 72 posts
- Offline
This is just my opinion so take it for what it's worth.
I see Attributes as data I want to cary through many nodes and ResultData as just the output of the previous node.
So say I want to have a specific output file or data from a node in the beginning to have an effect later on. I use a python or attr create node to ‘promote’ the ResultData to an attribute.
So temporary vs permanent. Other then that it's just how you like to work best I think.
Hope it helps somewhat.
I see Attributes as data I want to cary through many nodes and ResultData as just the output of the previous node.
So say I want to have a specific output file or data from a node in the beginning to have an effect later on. I use a python or attr create node to ‘promote’ the ResultData to an attribute.
So temporary vs permanent. Other then that it's just how you like to work best I think.
Hope it helps somewhat.
PDG/TOPs » "While Loop", can we make TOP listen and act
- Johan Boekhoven
- 72 posts
- Offline
I'm looking for a way start a listening/interactive mode with tops.
Can I have a loop that polls a database mysql/sqlite or something for changes and then query something and be processing whatever path I get back from it?
Not sure where to start. I can do it in python outside of houdini, but I'd rather have it all running inside of a houdini session that is just listening to a database.
Not sure where to start, if at all possible. Any pointers, help file or tutorial would be great!
Thanks!
Can I have a loop that polls a database mysql/sqlite or something for changes and then query something and be processing whatever path I get back from it?
Not sure where to start. I can do it in python outside of houdini, but I'd rather have it all running inside of a houdini session that is just listening to a database.
Not sure where to start, if at all possible. Any pointers, help file or tutorial would be great!
Thanks!
PDG/TOPs » Input items order, is it an unordered list?
- Johan Boekhoven
- 72 posts
- Offline
So I have a carefully crafted ordered Output list via the python node.
Now when I later down the pipe want to pick up the 1st file in that list I should be able to use
also
results in the same outcome
But the result is not correct, I get the result as if the list was alphabetically ordered.
Am I doing something wrong, is this as expected and can't we depend on the order of the output files?
Using production build .499.
Thanks,
-Johan
Now when I later down the pipe want to pick up the 1st file in that list I should be able to use
`@pdg_input.0`
`pdginput(0,'',1)`
But the result is not correct, I get the result as if the list was alphabetically ordered.
Am I doing something wrong, is this as expected and can't we depend on the order of the output files?
Using production build .499.
Thanks,
-Johan
Edited by Johan Boekhoven - July 23, 2020 11:54:38
PDG/TOPs » ROP Composite Output, how to batch succefully?
- Johan Boekhoven
- 72 posts
- Offline
To anyone interested I solved it like this.
I wanted to have on ROP node to process all ranges separately and in proper range order, so I could have 1 COP network when I need to change the settings. But I settled on this as ROPcomposite can also reference a separate COP network. All ranges are working fine and it starts 6 Hythons to process 6 ranges. So it works out great now.
Thanks again @tpetrick!
I wanted to have on ROP node to process all ranges separately and in proper range order, so I could have 1 COP network when I need to change the settings. But I settled on this as ROPcomposite can also reference a separate COP network. All ranges are working fine and it starts 6 Hythons to process 6 ranges. So it works out great now.
Thanks again @tpetrick!
PDG/TOPs » Wait for all incoming
- Johan Boekhoven
- 72 posts
- Offline
After a wait for all you can add a “partition expand” TOP node and you will get your workitems back. Think of partitions a bit like packed geometry. SO the expand TOP will unpack and give you back your workitems+attributes.
PDG/TOPs » ROP Composite Output, how to batch succefully?
- Johan Boekhoven
- 72 posts
- Offline
Thanks for the quick response! Will look at it first thing when I wake up
Much appreciated. I had a semi working option as well by not using the ROP Composite, and using ROP Fetch and a python generator to generate the frame range per partition and a file name looking like filename$F.jpg, looking at your comment, I think File range provides similar results.
Again thanks, will study the files and links!
Much appreciated. I had a semi working option as well by not using the ROP Composite, and using ROP Fetch and a python generator to generate the frame range per partition and a file name looking like filename$F.jpg, looking at your comment, I think File range provides similar results.
Again thanks, will study the files and links!
PDG/TOPs » ROP Composite Output, how to batch succefully?
- Johan Boekhoven
- 72 posts
- Offline
I'm unable to make any performant setup when I have the following data to start with.
I tried to use just all workitems separate though the composite output, or partition by range, 6 partitions of 22 items each.
When I have all items separate it processes extremely slow, I tried upping frames per batch but that didn't help. I'm thinking that for each workitem a new hython instance is started. Frames per batch etc, don't do nothing. Processing this manual in a COP network is plenty fast for my needs, but this is unbearably slow.
So what would be the best approach to have ROP Composite Output efficiently process image sequences from disk.
Can I better build some attributes like frame start/end and filename_$F.jpg to feed the ROP Composite Output?
I'm not sure what works with what on this node, I find the whole setup a bit confusing.
Any help or sample files appreciated. (screenshot image is the superslow setup)
- 6 jpg Image sequences, all starting at a different frame. 101-122, 201-222 etc. All same length.
- file pattern node which grabs all
I tried to use just all workitems separate though the composite output, or partition by range, 6 partitions of 22 items each.
When I have all items separate it processes extremely slow, I tried upping frames per batch but that didn't help. I'm thinking that for each workitem a new hython instance is started. Frames per batch etc, don't do nothing. Processing this manual in a COP network is plenty fast for my needs, but this is unbearably slow.
So what would be the best approach to have ROP Composite Output efficiently process image sequences from disk.
Can I better build some attributes like frame start/end and filename_$F.jpg to feed the ROP Composite Output?
I'm not sure what works with what on this node, I find the whole setup a bit confusing.
Any help or sample files appreciated. (screenshot image is the superslow setup)
Edited by Johan Boekhoven - July 22, 2020 16:38:54
PDG/TOPs » HDA processor and restarting a workitem
- Johan Boekhoven
- 72 posts
- Offline
I will have a look, but all the attributes used are tied very deep into the pipeline. I'm better of trying to reproduce it on a simple case. Will try later on. Thanks for checking!
PDG/TOPs » HDA processor and restarting a workitem
- Johan Boekhoven
- 72 posts
- Offline
Thanks Brandon,
I'm also on 499. My HDA has a lot of custom stuff on it, like python module settings params etc. I'd take it that shouldn't matter, but maybe there are circumstances where a recook just doesn't fire and keeps items queued? Maybe something on my scheduler?
I have a custom temp location and On Task Failure > Retry Task (max retries : 3).
So it should just instantly recook when I dirty a task and then press Cook Task, right?
I'm also on 499. My HDA has a lot of custom stuff on it, like python module settings params etc. I'd take it that shouldn't matter, but maybe there are circumstances where a recook just doesn't fire and keeps items queued? Maybe something on my scheduler?
I have a custom temp location and On Task Failure > Retry Task (max retries : 3).
So it should just instantly recook when I dirty a task and then press Cook Task, right?
PDG/TOPs » HDA processor and restarting a workitem
- Johan Boekhoven
- 72 posts
- Offline
Hi,
I've never been able to restart a workitem of a HDA processor via the RMB menu, everytime I dirty an item and try to cook again it will forever say scheduled. Am I doing something wrong?
Thanks!
-Johan
I've never been able to restart a workitem of a HDA processor via the RMB menu, everytime I dirty an item and try to cook again it will forever say scheduled. Am I doing something wrong?
Thanks!
-Johan
PDG/TOPs » Error handling while using a python server_begin/end
- Johan Boekhoven
- 72 posts
- Offline
That makes sense, thanks for that!
And would that look something like this?
Or do I need to do something else?
Thanks!
-Johan
And would that look something like this?
Or do I need to do something else?
Thanks!
-Johan
Edited by Johan Boekhoven - June 26, 2020 02:38:24
PDG/TOPs » Error handling while using a python server_begin/end
- Johan Boekhoven
- 72 posts
- Offline
Thanks Chris,
The issue arises from and HDA not being able to process the geo, so I don't think try catching would work, or do I not understand correctly. Could I make a python node following the HDA processor that handles the error?
But I won't have the data from the HDA processing down the line for a work_item to process.
I'd like to simple disable, bypass, delete the work_item all together. But then I can't be static in that part of the flow. Would such a thing be possible?
The issue arises from and HDA not being able to process the geo, so I don't think try catching would work, or do I not understand correctly. Could I make a python node following the HDA processor that handles the error?
But I won't have the data from the HDA processing down the line for a work_item to process.
I'd like to simple disable, bypass, delete the work_item all together. But then I can't be static in that part of the flow. Would such a thing be possible?
PDG/TOPs » Error handling while using a python server_begin/end
- Johan Boekhoven
- 72 posts
- Offline
Say I have 8 work_items coming into a python server.
The python server is set to static and Session Count from Upstream Items.
Now say work_item 5 failes, now my python server stops working while it waits for 5 to finish, which it will never do. The other work_items where cooked properly so I'm having to go in and jump through some crazy loops to finish the processing on the good items.
So is there a good way to handle failed jobs, I do some baking on very irregular meshes and sometimes something just doesn't compute. I want the rest of network to finish properly so I only have to fix the one issue, instead of everything after work_item 5.
1. Any smart tips on handling failed work_items would be great!
2. Can a server work serially while not wait for the proper index of work_item to hit it?
Thanks!
-Johan
The python server is set to static and Session Count from Upstream Items.
Now say work_item 5 failes, now my python server stops working while it waits for 5 to finish, which it will never do. The other work_items where cooked properly so I'm having to go in and jump through some crazy loops to finish the processing on the good items.
So is there a good way to handle failed jobs, I do some baking on very irregular meshes and sometimes something just doesn't compute. I want the rest of network to finish properly so I only have to fix the one issue, instead of everything after work_item 5.
1. Any smart tips on handling failed work_items would be great!
2. Can a server work serially while not wait for the proper index of work_item to hit it?
Thanks!
-Johan
PDG/TOPs » PDG_TEMP or workaround
- Johan Boekhoven
- 72 posts
- Offline
Thanks Chris,
This works for my needs thanks!
I think a lot of us would be very greatful with a lot more example files and documentation.
I know it's such a broad scope, but some more use cases would be great!
Again thanks!
-Johan
This works for my needs thanks!
I think a lot of us would be very greatful with a lot more example files and documentation.
I know it's such a broad scope, but some more use cases would be great!
Again thanks!
-Johan
-
- Quick Links