As you can see, the Invoke node costs really a lot of RAM and it won't free it after cooking. I've let Houdini waiting for a long time and doing nothing, the memory costs still not going down.
Is it an issue or I should free the RAM manually in some way?

This sounds like a bug. Is this 17.5 or 18.0, and is the issue resolved if you dirty the invoke node? Can you file a issue with an example .hip file?