Can't find PY_Py_Finalize in PY/PY_CPythonAPI.h
3447 14 1- calving
- Member
- 27 posts
- Joined: Sept. 2016
- Offline
- rvinluan
- Staff
- 1255 posts
- Joined: July 2005
- Offline
Hello,
I'm not sure what you are trying to do exactly but you shouldn't need to call Py_Finalize in your HDK plugin. If you just want to clean up Pythonic things on shutdown then you can register an exit callback with UT_Exit::addExitCallback:
http://www.sidefx.com/docs/hdk16.0/class_u_t___exit.html [sidefx.com]
Cheers,
Rob
I'm not sure what you are trying to do exactly but you shouldn't need to call Py_Finalize in your HDK plugin. If you just want to clean up Pythonic things on shutdown then you can register an exit callback with UT_Exit::addExitCallback:
http://www.sidefx.com/docs/hdk16.0/class_u_t___exit.html [sidefx.com]
Cheers,
Rob
- calving
- Member
- 27 posts
- Joined: Sept. 2016
- Offline
rvinluan
Hello,
I'm not sure what you are trying to do exactly but you shouldn't need to call Py_Finalize in your HDK plugin. If you just want to clean up Pythonic things on shutdown then you can register an exit callback with UT_Exit::addExitCallback:
http://www.sidefx.com/docs/hdk16.0/class_u_t___exit.html [sidefx.com]
Cheers,
Rob
Thanks for your prompt! So does it mean I don't need to call PY_Py_InitializeEx in HDK as well ?
And I have ran into another problem about PY_IntrepreterAutoLock now. Can I get any help about this function ? I have browsed the messages in that header file but still feel a little bit confuse. And the wrong way to use it cause my Houdini crash with a segmentation fault.
Edited by calving - April 18, 2017 07:09:13
- rvinluan
- Staff
- 1255 posts
- Joined: July 2005
- Offline
That's correct. You do not need to call Py_Initialize since Houdini does the Python initialization for you on startup.
Check out
Cheers,
Rob
Check out
$HFS/toolkit/samples/HOM/ObjNode_setSelectable.C
. There's an example of using PY_InterpreterAutoLock in the HOMextendLibrary() function. The idea is to create a PY_InterpreterAutoLock object right before you intend to make Python calls. Typically you create the object on the stack so that it is created when execution enters the code block's scope and is destroyed when execution exits the scope.Cheers,
Rob
- calving
- Member
- 27 posts
- Joined: Sept. 2016
- Offline
- calving
- Member
- 27 posts
- Joined: Sept. 2016
- Offline
rvinluan
That's correct. You do not need to call Py_Initialize since Houdini does the Python initialization for you on startup.
Check out$HFS/toolkit/samples/HOM/ObjNode_setSelectable.C
. There's an example of using PY_InterpreterAutoLock in the HOMextendLibrary() function. The idea is to create a PY_InterpreterAutoLock object right before you intend to make Python calls. Typically you create the object on the stack so that it is created when execution enters the code block's scope and is destroyed when execution exits the scope.
Cheers,
Rob
Hi, Rob. I have followed the way to use PY_InterpreterAutoLock from ObjNode_setSelectable, but it looks can't work well in my context.
I try to call external python module in my custom FS_ReaderHelper, the pseudo-codes are like this:
pyHelper(UT_String source, UT_String &target)
{
PY_InterpreterAutoLock interpreter_auto_lock;
…make Python API calls…
}
MY_Helper::createStream(const char *source, const UT_Options *)
{
…call pyHelper to convert source…
}
void installFSHelpers()
{
new MY_Helper();
}
Then Houdini will crash during starting with a segmentation fault from PY_PYModule_GetDict in my pyHelper function, the traceback information figure out things happen in libpthread. Obviously about GIL.
This is confusing me. Any suggestions would be helpful!
Edited by calving - April 19, 2017 05:54:15
- rvinluan
- Staff
- 1255 posts
- Joined: July 2005
- Offline
- calving
- Member
- 27 posts
- Joined: Sept. 2016
- Offline
rvinluan
The pseudo-code looks ok to me.
Would you be able to send me your source code? I can give it a try here and see if I can reproduce the crash.
Thanks so much. The attachment is my source code.
The whole version is in our company's computer, I simply rewrite this to reproduce the crash. But interestingly, I found it will crash Houdini/Hython/Hbatch in Linux but only crash Houdini in Windows.
Edited by calving - April 20, 2017 12:34:50
- rvinluan
- Staff
- 1255 posts
- Joined: July 2005
- Offline
I was able to reproduce the crash on startup. It's a tricky situation that took me a bit of time to figure out.
The problem is that
The best solution that I could come up with was to force loading the Python library and then check if Python is initialized. If it is initialized then run Python code as usual, otherwise, do nothing since Python is not ready yet. Don't even try to initialize Python. Just let Houdini initialize Python later on during the regular startup sequence since Houdini needs to do other important things for the initialization.
Anyway, in
I hope this helps.
Cheers,
Rob
The problem is that
FS_HelperConverter::pyExecution
is called extremely early on Houdini startup, well before the Python library has loaded and initialized.The best solution that I could come up with was to force loading the Python library and then check if Python is initialized. If it is initialized then run Python code as usual, otherwise, do nothing since Python is not ready yet. Don't even try to initialize Python. Just let Houdini initialize Python later on during the regular startup sequence since Houdini needs to do other important things for the initialization.
Anyway, in
pyExecution
I added these lines of code at the very top of the function body:static bool has_loaded_python_library = false; if (!has_loaded_python_library) { // Use the HDK to load the Python library. PYloadPythonLibrary(); has_loaded_python_library = true; } if (!PY_Py_IsInitialized()) return false;
I hope this helps.
Cheers,
Rob
- graham
- Member
- 1906 posts
- Joined: Nov. 2006
- Offline
Years ago I ran into the same issue trying to have a FS helper that would do various Python related calls. I think we got around the issue (at the suggestion from someone at SESI) by checking whether OPgetDirector() returned a non-null value. Basically our helper would only work if Houdini was in a far enough along state that we'd know Python was initialized properly. If OPgetDirector() didn't return valid we'd just say we couldn't handle things, etc.
Graham Thompson, Technical Artist @ Rockstar Games
- rvinluan
- Staff
- 1255 posts
- Joined: July 2005
- Offline
Oh, one other thing I forgot to mention is that `sys.path` may not be configured yet with Houdini paths when
Without setting PYTHONPATH I found that myHelper.py could not be found and imported so
pyExecution
runs. When I was testing, I had to set the PYTHONPATH environment variable to point to the folder containing myHelper.py
before launching Houdini.Without setting PYTHONPATH I found that myHelper.py could not be found and imported so
pyExecution
would crash still.
- rvinluan
- Staff
- 1255 posts
- Joined: July 2005
- Offline
graham
Years ago I ran into the same issue trying to have a FS helper that would do various Python related calls. I think we got around the issue (at the suggestion from someone at SESI) by checking whether OPgetDirector() returned a non-null value. Basically our helper would only work if Houdini was in a far enough along state that we'd know Python was initialized properly. If OPgetDirector() didn't return valid we'd just say we couldn't handle things, etc.
Ah, that's good way to check too. It is a more indirect check for Python initialization but a good one especially if the helper needs to access Houdini things.
Also, if you check for
OPgetDirector()
instead then I believe that you don't need to set the PYTHONPATH environment variable beforehand like I mentioned above.
- calving
- Member
- 27 posts
- Joined: Sept. 2016
- Offline
rvinluan
Oh, one other thing I forgot to mention is that `sys.path` may not be configured yet with Houdini paths whenpyExecution
runs. When I was testing, I had to set the PYTHONPATH environment variable to point to the folder containingmyHelper.py
before launching Houdini.
Without setting PYTHONPATH I found that myHelper.py could not be found and imported sopyExecution
would crash still.
It works! There are only two questions which aren't really serious.
First, it looks like HDK doesn't need PY_Py_DECREF to release the memory of PY_PyObject, this function will crash Houdini.
And second, when I try to let createStream & splitIndexFileSectionPath & combineIndexFileSectionPath call Python at same time, Houdini will crash when it starts (but Hython & Hbatch are well). I'm not sure whether my code leads to this or not.
Thanks a lot, I'll keep work on this and share new findings here.
- rvinluan
- Staff
- 1255 posts
- Joined: July 2005
- Offline
YKZDYI guess that's possible if something else is implicitly calling PY_Py_DECREF for you. I don't think it's Houdini/HDK though.
First, it looks like HDK doesn't need PY_Py_DECREF to release the memory of PY_PyObject, this function will crash Houdini.
YKZDY
And second, when I try to let createStream & splitIndexFileSectionPath & combineIndexFileSectionPath call Python at same time, Houdini will crash when it starts (but Hython & Hbatch are well). I'm not sure whether my code leads to this or not.
Hmmm. Maybe you need to create the interpreter auto lock in each of those functions?
Cheers,
Rob
- calving
- Member
- 27 posts
- Joined: Sept. 2016
- Offline
rvinluan
Hmmm. Maybe you need to create the interpreter auto lock in each of those functions?
Cheers,
Rob
No, there's only one function contains CPythonAPI, but there are lots of methods (over ten) from FS_InfoHelper & FS_ReadHelper need to call this function at same time. So finally I make a static local variable in that function then use it to avoid repeatedly python call.
So everything work well now, thanks for your help !
-
- Quick Links