Hello,
The Houdini UI probably hasn't opened yet which explains why hou.qt.mainWindow() is None. You could try deferring the code until Houdini's UI is ready.
Using the hdefereval module like so might work:
import hdefereval
from hutil.Qt import QtCore
theDialog = None
def openDialog():
global theDialog
theDialog = MyDialog()
theDialog.setParent(hou.qt.mainWindow(), QtCore.Qt.Window)
theDialog.show()
hdefereval.executeDeferred(openDialog)
Note that the script variable, theDialog, is needed to keep the dialog alive past the scope of the openDialog() function. If you use just a regular function variable then the dialog will most likely close and disappear when the function finishes execution.
Also, 456.py is called whenever a .hip file is opened. This means openDialog() can be called multiple times. If you just want the function to be called once at Houdini startup then try placing the code in pythonrc.py instead of 456.py.
Lastly, hutil.Qt was intended to be used by only Houdini internals. Feel free to use the module but be aware that the behaviour of this module can change in future Houdini releases.
Cheers,
Rob