Is it possible to create custom window inside Houdini for my own 3D rendering with D3D?
I'd love to hear anything that may help me to achieve this. I've checked HDK documentation and found nothing useful.

Nothing supported. I suppose you might be able to create your own child window from a viewport hook [sidefx.com] and do it that way. I don't think it will be easy though. Are you sure that you can't just use OpenGL?