Found 15 posts.
Search results Show results as topic list.
PDG/TOPs » Can I get the overall cook state easily?
- joabs
- 17 posts
- Offline
Unless you're seeing wrong information in the toolbar or in the TOP nodes, it is safe to assume the events are correct, as both relies on them. You might be running into data race issues, try using a lock around your code.
PDG/TOPs » Can I get the overall cook state easily?
- joabs
- 17 posts
- Offline
PDG/TOPs » hython TOPs progress reporting . logs etc
- joabs
- 17 posts
- Offline
Hi again,
There is one other thing that might solve your problem, TOPs is actually using pdgd, and we have node called “Remote Graph” that allows you to connect to a remote instance, all you need is to make sure you have a pdgd server running on that instance.
If before you cook your PDG graph, you run this code:
You then can use Houdini to attach to this instance and visualize the graph, in the `tasks` context, drop a remote graph node, set the host address and connect, the remote graph node will create a replica of the remote graph and display progress just like if it was a local graph.
There is one other thing that might solve your problem, TOPs is actually using pdgd, and we have node called “Remote Graph” that allows you to connect to a remote instance, all you need is to make sure you have a pdgd server running on that instance.
If before you cook your PDG graph, you run this code:
import pdgd
server_manager = pdgd.DataLayerServerManager.Instance()
# The first parameter is the server type, the second
# is the port the server will listen, zero means the
# system will choose an available port
server = server_manager.createServer('DataLayerWSServer', PORT_NUMBER)
server.serve()
You then can use Houdini to attach to this instance and visualize the graph, in the `tasks` context, drop a remote graph node, set the host address and connect, the remote graph node will create a replica of the remote graph and display progress just like if it was a local graph.
PDG/TOPs » hython TOPs progress reporting . logs etc
- joabs
- 17 posts
- Offline
Hi!
We do have an experimental feature that might help you. The PDG Data Layer (pdgd) allows you to access PDG data using a subscription system, not easy to explain here but it is basically you subscribe to a PDG graph, node or work item and will get updates every time data changed.
Using pdgd you can choose to access to local PDG data or to connect to a remote instance.
Today's build should have an updated version of the examples, a good starting point is $HFS/houdini/pdgd/examples/visualizer/, that will give you a brief intro on using pdgd showing very basic usage of the library. I hope to add another example this week showing how to start a server and how to connect to a remote instance, using the same code to inspect local and remote PDG data.
It's important to note that this is an experimental feature, we are still working on it, we're also working on documentation and examples.
Please let me know if you have any questions,
Joab
We do have an experimental feature that might help you. The PDG Data Layer (pdgd) allows you to access PDG data using a subscription system, not easy to explain here but it is basically you subscribe to a PDG graph, node or work item and will get updates every time data changed.
Using pdgd you can choose to access to local PDG data or to connect to a remote instance.
Today's build should have an updated version of the examples, a good starting point is $HFS/houdini/pdgd/examples/visualizer/, that will give you a brief intro on using pdgd showing very basic usage of the library. I hope to add another example this week showing how to start a server and how to connect to a remote instance, using the same code to inspect local and remote PDG data.
It's important to note that this is an experimental feature, we are still working on it, we're also working on documentation and examples.
Please let me know if you have any questions,
Joab
PDG/TOPs » Showing the TOP status icon at the container HDA node level
- joabs
- 17 posts
- Offline
This should be fixed in the next daily build now, please let me know if you still have issues with it.
PDG/TOPs » Showing the TOP status icon at the container HDA node level
- joabs
- 17 posts
- Offline
Hi,
This sounds like a bug, I believe this is happening because even though the node is drawing the TOP badge, its drawing is still triggered only by events relevant to that node. I'll have a look at this and keep you posted.
Thanks,
Joab
This sounds like a bug, I believe this is happening because even though the node is drawing the TOP badge, its drawing is still triggered only by events relevant to that node. I'll have a look at this and keep you posted.
Thanks,
Joab
PDG/TOPs » Connect to PDG Result Server
- joabs
- 17 posts
- Offline
Ostap
I'm wondering why you don't use get_local_data_layer() for “Task Graph Table”?
When inspecting local graphs, the task graph table uses the local data layer, it doesn't call that specific function, simply because it didn't exist, but it uses the same code as that function: “pdgd.GlobalDataLayerInstance.Instance()”, GlobalDataLayerInstance is a not great naming choice for the local data layer instance (Global here means a single, “global” instance in the current houdini session).
Ostap“graph is the actual owner”- Do you mean a node graph in “Network View”?
I mean a PDG graph, one TOP network always creates one PDG graph.
Ostap“If you want to use PDGD”- What the sense to use PDGD locally?
PDGD is really nice for visualizing PDG, you have access to data in a really dynamic way and anytime you subscribe to an object you get the full state. The most obvious advantage is that you can have the same code for local or remote PDG instances, both TOPs and the task graph table are able to display remote PDG instances using PDGD.
Ostap
Do you have a visual representation of dependencies in PDG (some explanation map of what is going on)?
I believe our documentation does a good job of explaining it: https://www.sidefx.com/docs/houdini/tops/intro.html, [www.sidefx.com] the only thing that I don't think it explicitly states is the difference between the TOP Network and the PDG Graph, for most users this isn't important and would only be confusing to bring it up. But for using PDGD it becomes more important, as you'll be accessing PDG only data. The PDG graph is usually really close to the TOP network you're seeing, almost a one to one match between nodes, except for some nodes like the null node that doesn't need a PDG node.
PDG/TOPs » Connect to PDG Result Server
- joabs
- 17 posts
- Offline
Hi!
I still couldn't investigate the crash properly, I'll try to get to it as soon as possible.
Are you using the task graph table to a different houdini instance or are you testing a connection to the current instance? If you're trying the latter, it's possible that it's triggering a series of events that updates the current node selection, telling the task graph table to change its contents. But this is just a guess, and if you're connecting to a remote instance and that's still happening, it's pretty bad, I'll ensure that when connected to a remote (even if the remote is actually a local) instance, the task graph table ignores some selection events.
About nodes and work items paths, PDG has its graphs, nodes and work items, Houdini creates and prepares PDG graphs using TOPs, they might present as being one thing, but they are actually two different things. So we can't guarantee names will always match, for instance, you can have a topnet1 node in /obj and another topnet1 in /tasks, but PDG doesn't know about that and requires all graphs to be named uniquely, so you'll probably end up with a topnet1 and a topnet10 in PDG world. There are many internal factors that lead to the current mapping scheme in PDGD, but the most important are: graphs are the actual owners of both nodes and work items, names are not reliable identifiers as they can change.
The monitor instance example doesn't make much sense for a local only situation where you already know which graphs are running locally. If you want to use PDGD locally you don't need to start a server and you can access the local data layer interface, the pdgd.util module provides the get_local_data_layer() function.
I still couldn't investigate the crash properly, I'll try to get to it as soon as possible.
Are you using the task graph table to a different houdini instance or are you testing a connection to the current instance? If you're trying the latter, it's possible that it's triggering a series of events that updates the current node selection, telling the task graph table to change its contents. But this is just a guess, and if you're connecting to a remote instance and that's still happening, it's pretty bad, I'll ensure that when connected to a remote (even if the remote is actually a local) instance, the task graph table ignores some selection events.
About nodes and work items paths, PDG has its graphs, nodes and work items, Houdini creates and prepares PDG graphs using TOPs, they might present as being one thing, but they are actually two different things. So we can't guarantee names will always match, for instance, you can have a topnet1 node in /obj and another topnet1 in /tasks, but PDG doesn't know about that and requires all graphs to be named uniquely, so you'll probably end up with a topnet1 and a topnet10 in PDG world. There are many internal factors that lead to the current mapping scheme in PDGD, but the most important are: graphs are the actual owners of both nodes and work items, names are not reliable identifiers as they can change.
The monitor instance example doesn't make much sense for a local only situation where you already know which graphs are running locally. If you want to use PDGD locally you don't need to start a server and you can access the local data layer interface, the pdgd.util module provides the get_local_data_layer() function.
PDG/TOPs » Connect to PDG Result Server
- joabs
- 17 posts
- Offline
Thanks for letting me know that, you should have the WebSockets option, turns out this is a bug with the Mac build, I got it fixed and should be working on the next daily build.
PDG/TOPs » Connect to PDG Result Server
- joabs
- 17 posts
- Offline
I put together a quick example using javascript, as I said, this is very early and experimental, the plan is to have better examples and a more defined API in the future, but this should be enough so you can play around and get something going.
The example is a basic react app, created using `npx create-react-app`. `npm install` and `npm start` should be enough to get it running.
The DataLayerInterface.js and PDGDObject.js are helper classes to interface with a remote data layer server, ListGraphs.jsx, GraphInfo.jsx, NodeInfo.jsx and WorkItemInfo.jsx are react components subscribing and getting data from PDG objects.
The example is a basic react app, created using `npx create-react-app`. `npm install` and `npm start` should be enough to get it running.
The DataLayerInterface.js and PDGDObject.js are helper classes to interface with a remote data layer server, ListGraphs.jsx, GraphInfo.jsx, NodeInfo.jsx and WorkItemInfo.jsx are react components subscribing and getting data from PDG objects.
Edited by joabs - 2020年4月13日 12:16:10
PDG/TOPs » Connect to PDG Result Server
- joabs
- 17 posts
- Offline
It is possible, the default protocol for the Data Layer server is websockets and it simply sends and receive json data. But this feature is mostly experimental and we don't have proper documentation for it yet.
I do have some javascript code that connects to it, I can create a small example so you can play with it a bit and see if it solves your problem. But it will be a while before we start providing actual support and documentation for it (it's actually good to have someone using it and testing it, I just can't promise answering questions quickly).
I do have some javascript code that connects to it, I can create a small example so you can play with it a bit and see if it solves your problem. But it will be a while before we start providing actual support and documentation for it (it's actually good to have someone using it and testing it, I just can't promise answering questions quickly).
PDG/TOPs » Connect to PDG Result Server
- joabs
- 17 posts
- Offline
As Chris mentioned, when doing a Submit-Graph-As-Job type cook, both HQueue and Deadline schedulers have the option to enable the Data Layer server, enabling it shows the “Auto Connect”, if auto connect is also checked, a remote graph node will be created automatically for you and it will create a replica of the remote graph and you'll be able to visualize nodes and work items just like a local graph.
This would be the most common use case and all it is doing is spawning a Data Layer server on the remote machine and creating a remote graph node pointing to the remote machine's address. You could even do it manually between two Houdini instances, on one instance you start a Data Layer server (either using python or the PDG State Server panel) and on the other you create a remote graph node, set the address and push connect.
This would be the most common use case and all it is doing is spawning a Data Layer server on the remote machine and creating a remote graph node pointing to the remote machine's address. You could even do it manually between two Houdini instances, on one instance you start a Data Layer server (either using python or the PDG State Server panel) and on the other you create a remote graph node, set the address and push connect.
PDG/TOPs » Question about the pdgd module and monitoring instance
- joabs
- 17 posts
- Offline
Hi!
I'll try to explain a bit about pdgd and the example, but I'll address your question right after the explanation (feel free to jump to it!).
It's important to note that pdgd is still in a very experimental state, we are working to provide better API, documentation and examples with the next Houdini release (and as long as there are no internal breaking changes, the functionality will also be added to Houdini 18). You can find some documentation here: https://www.sidefx.com/docs/houdini/tops/pdgd/index.html [www.sidefx.com]
You can think of pdgd as a generalized subscription based event system, you probably have seen the other examples, and under examples/visualizer there are a few examples showing the most basic usage of pdgd.
Pdgd is extensible, so you can use it to expose your own data, this is done by writing subscription handlers, we're working on some utility classes to make it really straightforward to write them.
This monitoring example is meant to show two features, the first is how pdgd can be extended, the custom_subscription_handler.py creates a subscription handler that will be registered to pdgd, the second feature shown is how to use the remote graph node to connect to a remote pdgd server.
The custom subscription handler holds a list of machine addresses, it provides two commands: “register” and “deregister”. The houdini instance that will be monitoring other instances starts a pdgd server so other instances can communicate with it.
Even though this pdgd server will expose pdg graphs, that's not what we're interested in this case, the important thing is the custom subscription handler we created. Remote instances can connect to the monitoring instance and use the register command to let it know they started to cook a pdg graph, they can also deregister themselves to signal their data won't be available anymore.
The “list_external_pdg_addresses” node in the monitor.hip file creates a pdgd visualizer and subscribes to the custom subscription handler (registered as “external_pdg”), it will receive a snapshot with all the data held by that subscription handler, in this case it is just a list of addresses. The next node (“create_remote_viewers”) will create a remote graph node for each registered address.
On the other side of all this, we have the instances doing pdg work, the monitoring instance wants to remotely watch their work. We have no mechanisms for knowing that a houdini instance in the network started, that's why we need the register/deregister system described above, so each instance must report itself to the monitor. This probably can be done by other means and each network will have its solutions and limitations on how to achieve this.
A remote instance must ensure that its pdg data is available, this is done by starting a pdgd server, once this server is running, other houdini instances can attach to it and visualize their TOP graphs using a remote graph node.
Before cooking its graph, every remote instance will need to first register to the monitor instance, in this example, this is done by connecting to the monitor instance's pdgd and sending a “register” command, the code for this is in the “register” method in register_pdg_instance.py, this script is executed in the first node of the pdg graph.
Summing up:
* Monitor instance
1. Creates a custom subscription handler, allowing external instances to register/deregister themselves
2. Starts a pdgd server, in order to make that subscription handler available to the network
3. Listens for registrations, for each registration it creates a remote graph pointing to it
* Remote instance
1. Starts a pdgd server, in order to expose its pdg data
2. Sends a register command to the monitor instance
3. Proceeds to cook its graph
About your question, the example sets the monitor address to ‘localhost:1314’, you just need to set it to the address of your monitoring machine. Please note that in a farm setup you may have network restrictions such as firewalls, so you need to make sure that the monitor instance and the remote instances can communicate to each other. The register/deregister strategy the example uses will not work in situation where the monitor instance can ping the remote instance but not the other way around, in this case you would need a different approach.
I hope this helped, please let me know if you have any further questions,
- Joab
I'll try to explain a bit about pdgd and the example, but I'll address your question right after the explanation (feel free to jump to it!).
It's important to note that pdgd is still in a very experimental state, we are working to provide better API, documentation and examples with the next Houdini release (and as long as there are no internal breaking changes, the functionality will also be added to Houdini 18). You can find some documentation here: https://www.sidefx.com/docs/houdini/tops/pdgd/index.html [www.sidefx.com]
You can think of pdgd as a generalized subscription based event system, you probably have seen the other examples, and under examples/visualizer there are a few examples showing the most basic usage of pdgd.
Pdgd is extensible, so you can use it to expose your own data, this is done by writing subscription handlers, we're working on some utility classes to make it really straightforward to write them.
This monitoring example is meant to show two features, the first is how pdgd can be extended, the custom_subscription_handler.py creates a subscription handler that will be registered to pdgd, the second feature shown is how to use the remote graph node to connect to a remote pdgd server.
The custom subscription handler holds a list of machine addresses, it provides two commands: “register” and “deregister”. The houdini instance that will be monitoring other instances starts a pdgd server so other instances can communicate with it.
Even though this pdgd server will expose pdg graphs, that's not what we're interested in this case, the important thing is the custom subscription handler we created. Remote instances can connect to the monitoring instance and use the register command to let it know they started to cook a pdg graph, they can also deregister themselves to signal their data won't be available anymore.
The “list_external_pdg_addresses” node in the monitor.hip file creates a pdgd visualizer and subscribes to the custom subscription handler (registered as “external_pdg”), it will receive a snapshot with all the data held by that subscription handler, in this case it is just a list of addresses. The next node (“create_remote_viewers”) will create a remote graph node for each registered address.
On the other side of all this, we have the instances doing pdg work, the monitoring instance wants to remotely watch their work. We have no mechanisms for knowing that a houdini instance in the network started, that's why we need the register/deregister system described above, so each instance must report itself to the monitor. This probably can be done by other means and each network will have its solutions and limitations on how to achieve this.
A remote instance must ensure that its pdg data is available, this is done by starting a pdgd server, once this server is running, other houdini instances can attach to it and visualize their TOP graphs using a remote graph node.
Before cooking its graph, every remote instance will need to first register to the monitor instance, in this example, this is done by connecting to the monitor instance's pdgd and sending a “register” command, the code for this is in the “register” method in register_pdg_instance.py, this script is executed in the first node of the pdg graph.
Summing up:
* Monitor instance
1. Creates a custom subscription handler, allowing external instances to register/deregister themselves
2. Starts a pdgd server, in order to make that subscription handler available to the network
3. Listens for registrations, for each registration it creates a remote graph pointing to it
* Remote instance
1. Starts a pdgd server, in order to expose its pdg data
2. Sends a register command to the monitor instance
3. Proceeds to cook its graph
About your question, the example sets the monitor address to ‘localhost:1314’, you just need to set it to the address of your monitoring machine. Please note that in a farm setup you may have network restrictions such as firewalls, so you need to make sure that the monitor instance and the remote instances can communicate to each other. The register/deregister strategy the example uses will not work in situation where the monitor instance can ping the remote instance but not the other way around, in this case you would need a different approach.
I hope this helped, please let me know if you have any further questions,
- Joab
Edited by joabs - 2020年2月25日 09:51:23
PDG/TOPs » Showing the TOP status icon at the container HDA node level
- joabs
- 17 posts
- Offline
Hi!
In the next H18 daily build, any node with the parameter “targettopnetwork” will use that parameter and display the progress and counts for the network referenced by it.
In addition to that, the “Node Properties” tab in the parameter interface editor will have a new group of parameters. This group has a operator path (for the “targettopnetwork” parm) and cook control buttons.
You'll be able to simply add this group of parameters, set the target top network node and you'll have cook controls and progress information.
- Joab
In the next H18 daily build, any node with the parameter “targettopnetwork” will use that parameter and display the progress and counts for the network referenced by it.
In addition to that, the “Node Properties” tab in the parameter interface editor will have a new group of parameters. This group has a operator path (for the “targettopnetwork” parm) and cook control buttons.
You'll be able to simply add this group of parameters, set the target top network node and you'll have cook controls and progress information.
- Joab
PDG/TOPs » PDG distinguish workitems in Task Graph Table
- joabs
- 17 posts
- Offline
Hi,
H18 Task Graph Table will allow you to hide/show all work item attributes.
Cheers,
Joab
H18 Task Graph Table will allow you to hide/show all work item attributes.
Cheers,
Joab
-
- Quick Links