I'm trying to integrate a variety of ML capabilities into a Houdini project. Since each one of these capabilities requires a sort of jenga tower of dependencies, I'm managing all of this using virtual environments. This is currently working entirely in the OBJ context. I just make sure everything in the venv is created using the same 3.11.7 version as hou python, and it seems to work well enough if I just use a script node to call a .py that activates an environment at the beginning before it runs each time.

It works well enough, but obviously it isn't as great as TOPS can be (I'll be running these inference operations many times on many different objects, so efficiency is key) so I was pretty excited when I saw that there is a TOPS virtual environment node. But when I point it to a virtual environment and point it to another python interpreter and list out some packages for it to import, I get nothing. Every script I run that tries to use any of the libraries listed to import error out because it can't find them. I've also let the tops node create a brand new virtual environment and still nothing.

So, how is this supposed to work?

Am I correct in thinking that once I have this working, basically any python process that is activated by tops nodes after the virtual environment has been activated via a python virtual environment node, they will be running using that virtual environment?

Specifically, let's say that I have a python script in the OBJ context that is running in either a python node or a script node, and that python script needs to import scipy or whatever to run. If I cook that SOP from the tops context after I've initialized a python venv node that says "use this venv, use this python interpreter, import these libraries", will all of those python scripts run using the virtual environment that was declared by the tops venv node?

Or does it only make the venv accessible in the tops context? (I can't get tops python nodes to see the libraries that are imported by the venv python node either)

Is there anything else I need to do for the settings on this thing? I pointed it to a virtual environment, I pointed it to a python interpreter, and I gave it a list of valid packages to install with pip, but I can't get any python node anywhere in houdini to see those package imports.

I'm currently using Houdini 20.5 OSX apple silicon. I haven't messed with it much on my windows workstation yet. Is this something that doesn't work that great on Mac and might work better on windows? (I'll try myself tomorrow, just putting this in there as another thought)

Anyways, thanks.

Yes, the Virtual Environment node works. It's used internally by the ML Regression Train [www.sidefx.com] node that ships with H20.5, as well as the ML terrain [www.sidefx.com] example in the content library that works in both H20 and H20.5.

The virtual environment node creates an environment using Python's venv module -- it does not work with other venv solutions like Conda. It creates the appropriate directory on disk and installs a list of packages to the environment with pip.

It does not, however activate the environment or run any code in the environment. It's still up to you to actually use the venv for something, for example by configuring a Python Ssript TOP to execute in that environment. You can do that by setting the Python Script's Cook Type to Out of Process, and changing the Python Bin parameter to virtual environment. Tasks in the Python Script node will then run using the interpreter in the virtual environment.

The virtual environment cannot be used with in-process Python Script TOP tasks, or other nodes like the Python SOP or Python Snippet COP because all of those nodes run their code inside of Houdini. They can therefore only use the Python interepreter/packages that are emebedded inside of Houdini.

I've attached an example scene file. If you cook the Python Script node and inspect the log for the task in that node, you'll see a print-out showing that the imported module was from the venv's package path.

Ok, fantastic explanation. Thanks for that. Yea, I missed that the venv just sets it up and doesn't do anything to change the environment.

So then if that's all it does, can I just create the virtual environment the old fashioned way from the command prompt or similar? Can I just point an out of process python tops node to any pre-existing venv and any python interpreter, regardless of whether they were created using the venv tops node or through some other means?

tpetrick
Yes, the Virtual Environment node works. It's used internally by the ML Regression Train [www.sidefx.com] node that ships with H20.5, as well as the ML terrain [www.sidefx.com] example in the content library that works in both H20 and H20.5.

The virtual environment node creates an environment using Python's venv module -- it does not work with other venv solutions like Conda. It creates the appropriate directory on disk and installs a list of packages to the environment with pip.

It does not, however activate the environment or run any code in the environment. It's still up to you to actually use the venv for something, for example by configuring a Python Ssript TOP to execute in that environment. You can do that by setting the Python Script's Cook Type to Out of Process, and changing the Python Bin parameter to virtual environment. Tasks in the Python Script node will then run using the interpreter in the virtual environment.

The virtual environment cannot be used with in-process Python Script TOP tasks, or other nodes like the Python SOP or Python Snippet COP because all of those nodes run their code inside of Houdini. They can therefore only use the Python interepreter/packages that are emebedded inside of Houdini.

I've attached an example scene file. If you cook the Python Script node and inspect the log for the task in that node, you'll see a print-out showing that the imported module was from the venv's package path.

can the virtual env be used with python processors?

Does anyone tried it on Windows?
I get permission error, like it can't create .lock file:

Running 'C:\PROGRA~1\SIDEEF~1\HOUDIN~1.440\python311\python.exe' with PID 22008
Waiting on environment lock file...
Traceback (most recent call last):
File "C:\Users\bogdana\AppData\Local\Temp\houdini_temp\NeuralTerrainPaint\pdgtemp\8844\scripts\pythonvenv.py", line 241, in <module>
cook(pdgjson.WorkItem.fromJobEnvironment())
File "C:\Users\bogdana\AppData\Local\Temp\houdini_temp\NeuralTerrainPaint\pdgtemp\8844\scripts\pythonvenv.py", line 191, in cook
with pdgcmd.FileLock(lock_file, is_windows) as lock:
File "C:\Users\bogdana\AppData\Local\Temp\houdini_temp\NeuralTerrainPaint\pdgtemp\8844\scripts\pdgcmd.py", line 242, in __enter__
self._lock_windows()
File "C:\Users\bogdana\AppData\Local\Temp\houdini_temp\NeuralTerrainPaint\pdgtemp\8844\scripts\pdgcmd.py", line 269, in _lock_windows
self.fd = os.open(self.file_name, self.flags)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
PermissionError: Permission denied: 'd:/PRODUCTIONS/ContentLibrary_NeuralTerrainPaint/venv.localscheduler/.lock'
*** Failed with Exit Code = 4294967295

Update. Got it working:
Moved my project from network drive to local drive.
Disabled use symlink.