Connect to PDG Result Server

Forums PDG/TOPs Connect to PDG Result Server

4045 19 3


Ostap: Member; 209 posts; Joined: Nov. 2010; Offline

April 7, 2020 11:38 p.m.

Hi,

First question: Is PDG State Server (with dedicated python panel name) is the same as PDG Result Server?
If the first question is true then second: is it possible to connect to PDG Result Server without starting anything else? Just with knowing the address of PDG Result Server?

Thanks
Ostap


Ostap: Member; 209 posts; Joined: Nov. 2010; Offline

April 8, 2020 12:01 a.m.

And another question: Is it possible to retrieve data from Result Server without Houdini? Or maybe it is possible with a minimum amount of dependencies? But looks like “_pdgd.so” is stopping us to do that.


chrisgreb: Member; 603 posts; Joined: Sept. 2016; Offline

April 8, 2020 9:12 a.m.

Is PDG State Server (with dedicated python panel name) is the same as PDG Result Server?

No, PDG Result Server is the message queue process that jobs connect to in order to communicate back to PDG. PDG State Server is a panel used for testing the ‘Data Layer’ (PDGD). It sounds like you want to connect to a running job to see some of the state. This is possible using the remotegraph TOP. More Info [www.sidefx.com]


Ostap: Member; 209 posts; Joined: Nov. 2010; Offline

April 8, 2020 4:50 p.m.

Thank you Joab for the detailed explanation.
But I still have questions about how to retrieve data from pdgd server without Houdini? Or maybe it is possible with a minimum amount of dependencies?

It sounds like you want to connect to a running job to see some of the state. This is possible using the remotegraph TOP.

Even “remotegraph” node has this “Data layer WebSockets Client” type connection - is that meant I still have to start state server to have working remotegraph node? Because otherwise, I can't make it work


chrisgreb: Member; 603 posts; Joined: Sept. 2016; Offline

April 8, 2020 4:58 p.m.

> is that meant I still have to start state server to have working remotegraph node

Yes, you need to have that server running in Houdini to use the remotegraph TOP node. This is turned on with parms on our Scheduler Nodes when doing a Submit-Graph-As-Job type cook.


joabs: Member; 17 posts; Joined: Feb. 2019; Offline

April 8, 2020 5:17 p.m.

As Chris mentioned, when doing a Submit-Graph-As-Job type cook, both HQueue and Deadline schedulers have the option to enable the Data Layer server, enabling it shows the “Auto Connect”, if auto connect is also checked, a remote graph node will be created automatically for you and it will create a replica of the remote graph and you'll be able to visualize nodes and work items just like a local graph.

This would be the most common use case and all it is doing is spawning a Data Layer server on the remote machine and creating a remote graph node pointing to the remote machine's address. You could even do it manually between two Houdini instances, on one instance you start a Data Layer server (either using python or the PDG State Server panel) and on the other you create a remote graph node, set the address and push connect.


Ostap: Member; 209 posts; Joined: Nov. 2010; Offline

April 8, 2020 5:38 p.m.

How about to connect to State Server without Houdini? For example, I have WEB client and I would like to retrieve PDG state using standard socket API - is it possible?
What the minimum requirement to get data from the State Server (just make it as light as possible)?


joabs: Member; 17 posts; Joined: Feb. 2019; Offline

April 9, 2020 1:49 p.m.

It is possible, the default protocol for the Data Layer server is websockets and it simply sends and receive json data. But this feature is mostly experimental and we don't have proper documentation for it yet.

I do have some javascript code that connects to it, I can create a small example so you can play with it a bit and see if it solves your problem. But it will be a while before we start providing actual support and documentation for it (it's actually good to have someone using it and testing it, I just can't promise answering questions quickly).


Ostap: Member; 209 posts; Joined: Nov. 2010; Offline

April 9, 2020 4:54 p.m.

Hi Joab,
That would be amazing if you can share this script!


joabs: Member; 17 posts; Joined: Feb. 2019; Offline

April 13, 2020 12:13 p.m.

I put together a quick example using javascript, as I said, this is very early and experimental, the plan is to have better examples and a more defined API in the future, but this should be enough so you can play around and get something going.

The example is a basic react app, created using `npx create-react-app`. `npm install` and `npm start` should be enough to get it running.

The DataLayerInterface.js and PDGDObject.js are helper classes to interface with a remote data layer server, ListGraphs.jsx, GraphInfo.jsx, NodeInfo.jsx and WorkItemInfo.jsx are react components subscribing and getting data from PDG objects.

Edited by joabs - April 13, 2020 12:16:10

Attachments:
webexample.zip (30.0 KB)


Ostap: Member; 209 posts; Joined: Nov. 2010; Offline

April 14, 2020 3:54 a.m.

Thank you for example! Web client is starting but maybe do you know why from pdgd registered type I have available only “Binary Server”? I'm assuming to work with Web client I need “WebSocket” server? And also, when I'm starting “Binary Server” and then creating any new PDG node - Houdini is freezing (18.0.435, MacOS 10.15.4).


joabs: Member; 17 posts; Joined: Feb. 2019; Offline

April 14, 2020 8:46 p.m.

Thanks for letting me know that, you should have the WebSockets option, turns out this is a bug with the Mac build, I got it fixed and should be working on the next daily build.


Ostap: Member; 209 posts; Joined: Nov. 2010; Offline

April 16, 2020 5:01 a.m.

Thanks for the fast fix! But I still have Houdini crashing/freezing when I'm starting “WebSocket” server and creating any new PDG node(18.0.437, MacOS 10.15.4).
Any idea how to fix it? Maybe it some permission error? But Houdini doesn't ask about any permission (except root while installation).

Attachments:
crash.untitled.yarynapochtarenko_12902_log.txt (4.4 KB)


Ostap: Member; 209 posts; Joined: Nov. 2010; Offline

April 17, 2020 5:15 a.m.

And can you please explain why TaskGraph losing connection so often.
Even with the simplest scene and selecting different workitem (Houdini 18.0.419)?

Edited by Ostap - April 17, 2020 05:16:35

Attachments:
task_graph_disconnect.hipnc (58.1 KB)
screen.gif (252.4 KB)


Ostap: Member; 209 posts; Joined: Nov. 2010; Offline

April 20, 2020 5:19 a.m.

Other questions are regarding path mapping and registration:
* Why node/workitem path is different comparing pdgd data and that what we see in Houdini structure? For example “pdg/topnet10/work_item_14” and “topnet1/genericgenerator1/genericgenerator10_8”. Why you are not following the same mapping?
* How monitor instance registration works (or data retrieving) if we using only local task graph panel (As I understand State Server is not running for local task graph panel)?

Edited by Ostap - April 20, 2020 05:19:20


joabs: Member; 17 posts; Joined: Feb. 2019; Offline

April 20, 2020 9:12 a.m.

Hi!

I still couldn't investigate the crash properly, I'll try to get to it as soon as possible.

Are you using the task graph table to a different houdini instance or are you testing a connection to the current instance? If you're trying the latter, it's possible that it's triggering a series of events that updates the current node selection, telling the task graph table to change its contents. But this is just a guess, and if you're connecting to a remote instance and that's still happening, it's pretty bad, I'll ensure that when connected to a remote (even if the remote is actually a local) instance, the task graph table ignores some selection events.

About nodes and work items paths, PDG has its graphs, nodes and work items, Houdini creates and prepares PDG graphs using TOPs, they might present as being one thing, but they are actually two different things. So we can't guarantee names will always match, for instance, you can have a topnet1 node in /obj and another topnet1 in /tasks, but PDG doesn't know about that and requires all graphs to be named uniquely, so you'll probably end up with a topnet1 and a topnet10 in PDG world. There are many internal factors that lead to the current mapping scheme in PDGD, but the most important are: graphs are the actual owners of both nodes and work items, names are not reliable identifiers as they can change.

The monitor instance example doesn't make much sense for a local only situation where you already know which graphs are running locally. If you want to use PDGD locally you don't need to start a server and you can access the local data layer interface, the pdgd.util module provides the get_local_data_layer() function.


Ostap: Member; 209 posts; Joined: Nov. 2010; Offline

April 21, 2020 5:10 a.m.

I'm wondering why you don't use get_local_data_layer() for “Task Graph Table”?

“graph is the actual owner”

- Do you mean a node graph in “Network View”?

“If you want to use PDGD”

- What the sense to use PDGD locally?

Do you have a visual representation of dependencies in PDG (some explanation map of what is going on)?

-

Are you using the task graph table to a different houdini instance or are you testing a connection to the current instance?

I'm testing a connection to the current instance. Just creating any TOP node is killing Houdini (If the server is running).
While running Houdini I have next message (MacOS):

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/Applications/Houdini/Houdini18.0.442/Frameworks/Python.framework/Versions/Current/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/Applications/Houdini/Houdini18.0.442/Frameworks/Houdini.framework/Versions/Current/Resources/houdini/python2.7libs/houdinihelp/api.py", line 553, in run
    with indexer.writer() as w:
  File "/Applications/Houdini/Houdini18.0.442/Frameworks/Houdini.framework/Versions/Current/Resources/houdini/python2.7libs/bookish/search.py", line 408, in writer
    return self.index.writer(**kwargs)
  File "/Applications/Houdini/Houdini18.0.442/Frameworks/Houdini.framework/Versions/Current/Resources/houdini/python2.7libs/whoosh/index.py", line 464, in writer
    return SegmentWriter(self, **kwargs)
  File "/Applications/Houdini/Houdini18.0.442/Frameworks/Houdini.framework/Versions/Current/Resources/houdini/python2.7libs/whoosh/writing.py", line 514, in __init__
    delay=delay):
  File "/Applications/Houdini/Houdini18.0.442/Frameworks/Houdini.framework/Versions/Current/Resources/houdini/python2.7libs/whoosh/util/filelock.py", line 54, in try_for
    v = fn()
  File "/Applications/Houdini/Houdini18.0.442/Frameworks/Houdini.framework/Versions/Current/Resources/houdini/python2.7libs/whoosh/util/filelock.py", line 98, in acquire
    self.fd = os.open(self.filename, flags)
OSError: [Errno 13] Permission denied: '/Users/yarynapochtarenko/Library/Preferences/houdini/18.0/config/Help/index2.7/MAIN_WRITELOCK'

I use “Houdini Launcher” to install Houdini.

Linux machine works fine with WEB connection/visualization of PDG workitems except warning:

Can't call setState on a component that is not yet mounted. This is a no-op, but it might indicate a bug in your application. Instead, assign to `this.state` directly or define a `state = {};` class property with the desired state in the ListGraphs component.


joabs: Member; 17 posts; Joined: Feb. 2019; Offline

April 21, 2020 10:04 a.m.

Ostap
I'm wondering why you don't use get_local_data_layer() for “Task Graph Table”?

When inspecting local graphs, the task graph table uses the local data layer, it doesn't call that specific function, simply because it didn't exist, but it uses the same code as that function: “pdgd.GlobalDataLayerInstance.Instance()”, GlobalDataLayerInstance is a not great naming choice for the local data layer instance (Global here means a single, “global” instance in the current houdini session).

Ostap
“graph is the actual owner”
- Do you mean a node graph in “Network View”?

I mean a PDG graph, one TOP network always creates one PDG graph.

Ostap
“If you want to use PDGD”
- What the sense to use PDGD locally?

PDGD is really nice for visualizing PDG, you have access to data in a really dynamic way and anytime you subscribe to an object you get the full state. The most obvious advantage is that you can have the same code for local or remote PDG instances, both TOPs and the task graph table are able to display remote PDG instances using PDGD.

Ostap
Do you have a visual representation of dependencies in PDG (some explanation map of what is going on)?

I believe our documentation does a good job of explaining it: https://www.sidefx.com/docs/houdini/tops/intro.html, [www.sidefx.com] the only thing that I don't think it explicitly states is the difference between the TOP Network and the PDG Graph, for most users this isn't important and would only be confusing to bring it up. But for using PDGD it becomes more important, as you'll be accessing PDG only data. The PDG graph is usually really close to the TOP network you're seeing, almost a one to one match between nodes, except for some nodes like the null node that doesn't need a PDG node.


Ostap: Member; 209 posts; Joined: Nov. 2010; Offline

April 22, 2020 4:59 a.m.

Thanks for your explanation!
I sated my information thirst (for now) : )


Ostap: Member; 209 posts; Joined: Nov. 2010; Offline

April 28, 2020 11:40 p.m.

Hi,

How to retrieve all workitem attributes? For example “__result_tag”, “__pdg_outputfiles”, etc.
Using:

PDGObject('pdg/topnet2/work_item_1',data_layer).object

I can retrieve only:

'__node','__node_name','__command','__dependencies','__is
_control','__id','__in_process','__attr__test','__type','__no_generate','__cook_time','__index','__frame','__has_frame','__name','__state','__priority'

I see that TaskGraph gathering more attributes but I can't understand how to get the same data (object) as TaskGraph pannel.

Can you help me, please?

Quick Links

                    
                        Search links
                        Show recent posts
                        Show unanswered posts