What's new PDG/TOPs in Production Build 18.0.391

Forums PDG/TOPs What's new PDG/TOPs in Production Build 18.0.391

5874 22 4


chrisgreb: Member; 603 posts; Joined: Sept. 2016; Offline

Feb. 25, 2020 4:43 p.m.

Here's a list of some of the more significant changes we've made since the last production build.

New Nodes

Range Generate [www.sidefx.com]
Generate a range of work items
File Range [www.sidefx.com]
Generate a range of frames, based on a file pattern.
In-Process Scheduler [www.sidefx.com]
Added a new In Process Scheduler TOP which can be used to configure the maximum number of concurrent in-process work items. Previously PDG would always run HOUDINI_MAX_THREADS in-process work items, without any controls exposed for changing that behavior.
Foundry's Nuke Server [www.sidefx.com]
Joining Houdini and Maya Command Chains [www.sidefx.com]

Overhauled Nodes

Python-based PDG Nodes dynamic reloading
Added support for reloading PDG node and scheduler Python definitions in a live Houdini session.
Currently this can only be done using the PDG Python API by looking up the type object from a node or scheduler instance, and then calling reloading on it. For example:

import pdg
types = pdg.TypeRegistry.types()
rt = types.registeredType(pdg.registeredType.Scheduler, "localscheduler")
rt.reload()
# Or
hou.node('/tasks/topnet1/localscheduler').getPDGNode().type.reload()

When a Python definition is reloaded, all existing instance of the node are dirtied and their underlying PDG node object is reset to use the new class definition created by reloading the Python module.

USD Import TOP [www.sidefx.com] can point at a LOP node and copy USD attributes on to work items.

Send Email [www.sidefx.com] TOP now supports TLS and Authentication.

ROP Fetch can now report result files from ROPs which support a convention using the pdg_logoutput [www.sidefx.com] attribute

HQueue Scheduler [www.sidefx.com]

MQ Job can be optionally disabled
Customize which Python executable is used
Option for jobs to inherit the local environment

Performance Improvement with Services [www.sidefx.com] for HDA Processor

Licence for all PDG python scripts changed from Proprietary to MIT.

Edited by chrism - March 20, 2020 13:45:05


Ostap: Member; 209 posts; Joined: Nov. 2010; Offline

Feb. 29, 2020 3:04 a.m.

Hi,

Can you please provide an example of how to get scheduler instance (to properly type reload)?
Because probably pdg.types.schedulers.local is not what do you mean.

Thanks
Ostap


chrisgreb: Member; 603 posts; Joined: Sept. 2016; Offline

March 1, 2020 1:26 p.m.

You can also reload a scheduler type like this:

import pdg
types = pdg.TypeRegistry.types()
rt = types.registeredType(pdg.registeredType.Scheduler, "localscheduler")
rt.reload()


EricSheng: Member; 159 posts; Joined: Feb. 2018; Offline

March 1, 2020 8:49 p.m.

In-Process Scheduler
Finally, great updates make it more robust for PDG.
Thank you, can't wait to update.


pbowmar: Member; 7024 posts; Joined: July 2005; Offline

March 1, 2020 10:20 p.m.

Thanks for this Chris, please continue these “what's new” summaries, there is so much going on in HOudini these are super helpful!

Cheers,
Peter B


Ostap: Member; 209 posts; Joined: Nov. 2010; Offline

March 2, 2020 2:39 p.m.

Thank you Chris for an example! But why it doesn't work for custom schedulers? For example (returns None, build 18.0.391):

types.registeredType(pdg.registeredType.Scheduler, "deadlinescheduler")

Should scheduler be in the ‘schedulers’ subdirectory to register properly? Or “$HH/pdg/types” is enough?


chrisgreb: Member; 603 posts; Joined: Sept. 2016; Offline

March 2, 2020 2:42 p.m.

You should put your custom scheduler python file into for example: $HOUDINI_PATH/pdg/types/myscheduler.py (no subdirectory). And that file should have the registerTypes function in it:

def registerTypes(type_registry):
    type_registry.registerScheduler(MyScheduler)


Ostap: Member; 209 posts; Joined: Nov. 2010; Offline

March 2, 2020 3:02 p.m.

Ok, clear! That what I did

When I'm trying to reload myscheduler it says:

RuntimeError: Unable to reload module 'pdg.types.myscheduler'

How about "deadlinescheduler'? Does it work for you Chris?

types.registeredType(pdg.registeredType.Scheduler, "deadlinescheduler")


tpetrick: Staff; 585 posts; Joined: May 2014; Offline

March 2, 2020 4:46 p.m.

If you don't have the Deadline API installed on your machine's Python Path, there won't be a type definition for it. You can see the list of available scheduler definitions using:

import pdg
types = pdg.TypeRegistry.types()
types.typeNames(pdg.registeredType.Scheduler)

I have Tractor installed but not Deadline, so for me the output is:

[u'inprocessscheduler', u'tractorscheduler', u'localscheduler', u'hqueuescheduler', u'pythonscheduler']


tpetrick: Staff; 585 posts; Joined: May 2014; Offline

March 2, 2020 4:48 p.m.

With regard to your issue with reloading your custom MyScheduler type, it looks like there's a bug in the reload functionality when the type is defined in a top-level Python file. $HOUDINI_PATH/pdg/types/myscheduler.py won't work, but if you put your custom types into a directory it'll work as expected, for example $HOUDINI_PATH/pdg/types/custom/myscheduler.py, with a $HOUDINI_PATH/pdg/types/custom/__init__.py. You can see an example of how we use that internally by looking at $HH/pdg/types/schedulers. We're investigating the cause of the issue and will have it fixed in a daily build this week.

Edited by tpetrick - March 2, 2020 16:49:09


Ostap: Member; 209 posts; Joined: Nov. 2010; Offline

March 2, 2020 6 p.m.

Thanks for “deadlinescheduler” explanation but in case of MyScheduler type is still erroring and you did a custom dir for MyScheduler but its interference with the previous message from Chris:

You should put your custom scheduler python file into for example: $HOUDINI_PATH/pdg/types/myscheduler.py (no subdirectory).


tpetrick: Staff; 585 posts; Joined: May 2014; Offline

March 2, 2020 6:38 p.m.

Chris and I just discovered the problem I mentioned moments before I posted my comment, while trying to reproduce the issue you reported. The subdirectory should not be necessary, but it currently is because of an issue with the way Python's module reloading works. We'll have the underlying issue fixed shortly (most likely Wednesday's daily build of 18.0).

What does work from our testing is the following:

$HOUDINI_PATH/pdg/types/custom/myscheduler.py contains your custom scheduler defintion.
$HOUDINI_PATH/pdg/types/custom/__init__.py contains something like the following:

from . import myscheduler

def registerTypes(type_registry):
    type_registry.registerScheduler(myscheduler.MyScheduler)

Then, to reload:

>>> import pdg
>>> types = pdg.TypeRegistry.types()
>>> t = types.registeredType(pdg.registeredType.Scheduler, "myscheduler")
>>> t.reload()

If you're still experience errors can you attach your files? Either here or as an issue with support?

Edited by tpetrick - March 2, 2020 18:40:10


tpetrick: Staff; 585 posts; Joined: May 2014; Offline

March 3, 2020 2:36 p.m.

The type reloading issue with for Python files placed directly in the ${HOUDINI_PATH}/pdg/types folder is fixed in tomorrow's 18.0 daily build (18.0.399).

Edited by tpetrick - March 3, 2020 14:36:31


Ostap: Member; 209 posts; Joined: Nov. 2010; Offline

March 3, 2020 3:50 p.m.

Thank you!


atnreg: Member; 57 posts; Joined: April 2018; Offline

March 5, 2020 5:01 a.m.

Hi!

I don't know if this is related to same but since 398 (also in 399) every time when I start Houdini (Indie) I get these errors in console:
PDG Type Registry: Failed to import duplicate module ‘houdini’ which was previously imported from ‘C:/PROGRA~1/SIDEEF~1/HOUDIN~1.399/houdini/pdg/types\houdini’
PDG Type Registry: Failed to import duplicate module ‘partitioners’ which was previously imported from ‘C:/PROGRA~1/SIDEEF~1/HOUDIN~1.399/houdini/pdg/types\partitioners’
PDG Type Registry: Failed to import duplicate module ‘schedulers’ which was previously imported from ‘C:/PROGRA~1/SIDEEF~1/HOUDIN~1.399/houdini/pdg/types\schedulers’
PDG Type Registry: Failed to import duplicate module ‘utils’ which was previously imported from ‘C:/PROGRA~1/SIDEEF~1/HOUDIN~1.399/houdini/pdg/types\utils’

Everything still works but that shows there is something wrong, how to fix it?

Thanks

Antti

Edited by atnreg - March 5, 2020 05:02:04

My system:
AMD Ryzen Threadripper 2990 (32-core),64GB RAM,NVIDIA RTX 3090,Win10
UE5.1,Houdini 19.5 Indie(always latest daily),Blender,Clarisse,Onyx,ZBrush...
Clarisse 2016-09, Houdini 2018-05 (seriously 2019-10)
I do all 3D for fun, no business


chrisgreb: Member; 603 posts; Joined: Sept. 2016; Offline

March 5, 2020 9:09 a.m.

This error is likely due to having duplicate Houdini paths. Please check your houdini.env file to see if you have any path entries ending with &. If so, comment them out or remove them and try it. This could be caused by having the Labs Toolset path entry in this file.


atnreg: Member; 57 posts; Joined: April 2018; Offline

March 5, 2020 9:43 a.m.

chrisgreb
This error is likely due to having duplicate Houdini paths. Please check your houdini.env file to see if you have any path entries ending with &. If so, comment them out or remove them and try it. This could be caused by having the Labs Toolset path entry in this file.

I have this in my houdini.env:
—–
HOUDINI_NVIDIA_OPTIX_DSO_PATH = C:/Users/anuoreva/Documents/houdini18.0\optix

PATH = E:/_LIB/HOUDINI/octaneplugin/bin;$PATH
HOUDINI_PATH = $HOUDINI_PATH;E:/_LIB/HOUDINI/octaneplugin;&
—–

The only part that has & is the latter path to octaneplugin path and if I comment it out, the error goes away.

I have not used Octane for a while (since beginning of Jan) but before 398 I got no errors. So how the Octane path should be changed so that it works with 398+?

Thanks

Antti


tpetrick: Staff; 585 posts; Joined: May 2014; Offline

March 5, 2020 12:13 p.m.

The & token means to append the regular Houdini path onto the end. So you're effectively doing that twice when you have HOUDINI_PATH = $HOUDINI_PATH;E:/_LIB/HOUDINI/octaneplugin;& That's like setting the variable to HOUDINI_PATH = $HOUDINI_PATH;E:/_LIB/HOUDINI/octaneplugin;$HOUDINI_PATH. You probably want to remove the leading $HOUDINI_PATH, something like HOUDINI_PATH = E:/_LIB/HOUDINI/octaneplugin;&.

Edited by tpetrick - March 5, 2020 12:31:44


atnreg: Member; 57 posts; Joined: April 2018; Offline

March 5, 2020 12:53 p.m.

Oh, great! Yes now it works without errors

Then I actually wonder, why it did work before 398 at all

Anyway, now it's fine, thank you very much!

Antti


tpetrick: Staff; 585 posts; Joined: May 2014; Offline

March 5, 2020 1:31 p.m.

Those warning messages themselves are new, to help debug problems with adding custom PDG node types onto your path that conflict with our built-in ones. They don't change any functionality though - they're just reporting information that wasn't reported before.

We can add a note to that message that suggests checking the $HOUDINI_PATH variable.

Quick Links

                    
                        Search links
                        Show recent posts
                        Show unanswered posts