How to use scipy with Houdini 17 on windows?

   14816   26   4
User Avatar
Member
18 posts
Joined: Jan. 2017
Offline
Hello!

before Houdini 17 this link showed how to use scipy with Houdini even for windows http://wordpress.discretization.de/houdini/home/advanced-2/installing-and-using-scipy-in-houdini/ [wordpress.discretization.de]

However, this trick does not work anymore. I really need it for my work.

Overwriting too many files using the anaconda distribution gets Houdini stuck at startup.

Copying only the scipy and/or numpy packages into C:\Program Files\Side Effects Software\Houdini 17.0.352\python27\lib\site-packages results in errors at the node.
Error
Python error: Traceback (most recent call last):
File “”, line 3, in
File “C:\PROGRA~1\SIDEEF~1\HOUDIN~1.352\python27\lib\site-packages\scipy\__init__.py”, line 119, in
from scipy._lib._ccallback import LowLevelCallable
File “C:\PROGRA~1\SIDEEF~1\HOUDIN~1.352\python27\lib\site-packages\scipy\_lib\_ccallback.py”, line 1, in
from . import _ccallback_c
ImportError: DLL load failed: The specified module could not be found.

When overwriting the C:\Program Files\Side Effects Software\Houdini 17.0.352\python27\dlls with the anaconda distribution, _ctypes.pyd and _socket.pyd seem to cause errors at startup.

I hope that someone here has more knowledge on setting up python libraries and can help out.

Has anyone managed to use scipy in Houdini and can please share how to do that here?

I will update the above website with that information.

thanks!
User Avatar
Member
129 posts
Joined: Jan. 2013
Offline
also ask
User Avatar
Member
385 posts
Joined: Nov. 2008
Offline
That error looks like you will probably have to build scipy with Visual Studio 17 (C++14)
Houdini 17.0 has already numpy 1.12.1
Edited by pezetko - Oct. 18, 2018 14:18:18
User Avatar
Member
18 posts
Joined: Jan. 2017
Offline
Yes you might be right and I wish I could try it out.

According to this site https://docs.scipy.org/doc/scipy/reference/building/windows.html [docs.scipy.org] I need to go through several steps in order to try it. I have not worked with Visual Studio yet. Has anyone tried this successfully?
User Avatar
Member
897 posts
Joined: July 2018
Offline
I've spent a few weeks on this and asked more clever people for guidance. After setting up mingwpy and going through various compilation guides I gave up and told Sidefx to drop the rfe I've posted for including it to Houdini as I understood the scale of issue.
B.Henriksson, DICE
User Avatar
Member
18 posts
Joined: Jan. 2017
Offline
kahuna031
I gave up

Me too. Scipy on Houdini 17 also did not work on my opensuse but I did not experiment enough why.

I just made the rfe too.
User Avatar
Member
897 posts
Joined: July 2018
Offline
What are you after from scipy? I'm thinking a.more realistic expectation would be for sidefx to support specific features rather than the whole package.
B.Henriksson, DICE
User Avatar
Member
18 posts
Joined: Jan. 2017
Offline
kahuna031
What are you after from scipy? I'm thinking a.more realistic expectation would be for sidefx to support specific features rather than the whole package.

We do mathematics and computer graphics research and need the linear algebra packages because of its sparse matrix solvers. Python is the goto language for many algorithms we need and it would be great to stick to it.(examples [page.math.tu-berlin.de])
Edited by MarcelPP - Nov. 12, 2018 12:33:33
User Avatar
Member
897 posts
Joined: July 2018
Offline
I was also after the sparse solvers but ended up modifying a git project that was wrapping libigl that in turn is a Eigen wrapper where you got sparse solvers:
https://github.com/bhenriksson031/EdgyEggs [github.com]
Edited by kahuna031 - Nov. 12, 2018 12:40:00
B.Henriksson, DICE
User Avatar
Member
96 posts
Joined: Jan. 2008
Offline
Any news on this?
It's actually of major importance for us as well.
User Avatar
Member
18 posts
Joined: Jan. 2017
Offline
No sorry still no news. We just keep using 16.5 and hope for change.
User Avatar
Member
18 posts
Joined: Jan. 2017
Offline
I am trying this again with Houdini 17.5

When installing a version of scipy, I copy over the folders from anaconda2

scipy
scipy-1.1.0-py2.7.egg-info

into C:\Program Files\Side Effects Software\Houdini 17.5.173\python27\lib

then get this error when trying to import scipy in a python node e.g. from scipy.sparse import csr_matrix

Python error: Traceback (most recent call last):
File “”, line 1, in
File “C:\PROGRA~1\SIDEEF~1\HOUDIN~1.173\python27\lib\site-packages\scipy\__init__.py”, line 77, in
from . import _distributor_init
ImportError: cannot import name _distributor_init

according to this source: https://stackoverflow.com/questions/48423782/importerror-cannot-import-name-distributor-init [stackoverflow.com]
I can comment out the import _distributor_init and in the __init__.py so after that I get this message:

Error
Python error: Traceback (most recent call last):
File “”, line 1, in
File “C:\PROGRA~1\SIDEEF~1\HOUDIN~1.173\python27\lib\site-packages\scipy\__init__.py”, line 119, in
from scipy._lib._ccallback import LowLevelCallable
File “C:\PROGRA~1\SIDEEF~1\HOUDIN~1.173\python27\lib\site-packages\scipy\_lib\_ccallback.py”, line 1, in
from . import _ccallback_c
ImportError: DLL load failed: The specified module could not be found.

I think this error goes back to the _ctypes.pyd in C:\Program Files\Side Effects Software\Houdini 17.5.173\python27\dlls
placing the_ctypes.pyd into that folder makes Houdini not start up anymore.

What now? Can anyone help out? Especially if you have windows, please write us if anything worked for you.

thanks in advance.
User Avatar
Member
96 posts
Joined: Jan. 2008
Offline
As far as I know the python distribution that comes with Houdini on Windows was compiled with Visual Studio 2017. You can read this of from the first line in the python shell

Python 2.7.15 (default, Jan 13 2019, 12:40:45) on win32


So whatever you want to add to Houdini on Windows needs to be compiled with the same version ( MSC v.1916 ). Unfortunately, I wasn't able to find any scipy binaries compiled with this particular compiler for python 2.7. So I guess if you don't want to compile it yourself, and sesi cannot do that for you, it might challenging to get it to work.


Depending on your usecase there might be certain workarounds possible. But all of them are not ideal.
Please let me know if you find any solution working for you.
User Avatar
Member
897 posts
Joined: July 2018
Offline
The same compilers is simply not available on windows and it would be a massive job for sidefx supporting a workaround. Reading the history around the project behind the old win-scipy has convinced me of this.

As I mentioned before my suggestion would be to find the specific feature you want from scipy and implement this. Think that would also be a more reasonable rfe for sidefx.
B.Henriksson, DICE
User Avatar
Member
18 posts
Joined: Jan. 2017
Offline
siavash82ir
What are you after from scipy?
I'm thinking a.more realistic expectation would be for sidefx to support specific features rather than the whole package.

We need the linear algebra packages together with the sparse matrix handlers and solvers. That would be a ton of work to re-implement considering that this usage of scipy works well in pre Houdini 17.0 versions.

python is full of packages anyway and many of them are supported by Houdini. Before we managed to insert scipy through the back door but now this is not working anymore. I was thinking that the entire issue could be solved if SideFx includes this one more package in its installation.
Edited by MarcelPP - April 22, 2019 10:42:23
User Avatar
Member
82 posts
Joined: March 2017
Offline
Fingers crossed sesi will sort it out soon!
User Avatar
Member
32 posts
Joined: Nov. 2017
Offline
I'm starting a project like this myself and asking the same questions.

Realistically (and very sadly), I think you'll have more luck doing your scipy work in a modern python like python3.7, and starting hython as a subprocess and communicating with it via sockets: either make it a web service, or use execnet (https://pypi.org/project/execnet/) or something similar. IPC always makes everything harder and slower, but trying to write and support anything significant in python2 at this point is asking for trouble.
User Avatar
Member
18 posts
Joined: Jan. 2017
Offline
Hi everyone,

I have spent some time today trying to see if the new Houdini 18 has changed any of this. I was not successful in trying to manually place the scipy/numpy packages.

Does anyone have any new ideas?

Does anyone maybe know how to locally, inside a python script, load numpy/scipy from an entirely different location?

cheers

Marcel
Edited by MarcelPP - Dec. 3, 2019 10:00:09
User Avatar
Member
5 posts
Joined: Jan. 2019
Offline
How do we request scipy being shipped with Houdini 18 and future? I am still using 16.5 if I need scipy.
User Avatar
Member
18 posts
Joined: Jan. 2017
Offline
You could do a request for enhancement. I did that when this thread opened.
https://www.sidefx.com/bugs/submit/ [www.sidefx.com]

On another email in January we know this:
SEV 2 (High Priority), This was on 1/27/20

Given that python 2.7 won't have any support anymore I hope that they work over their python in general.
Link [dev.to]
  • Quick Links