That's say I have a custom 3D format called ".custom" and I'd like to export geometry from Houdini in this format.

I've checked the document: https://www.sidefx.com/docs/houdini/io/formats/create_external_format.html [www.sidefx.com]

But it's so confusing. It says:

$HFS/houdini/support/GEOio

But I don't even have supportfolder in my $HFS/houdini. Plus, what's the next step? It doesn't automatically create an ROP node, right? Then how do we incorporate this new format with normal Houdini workflow, e.g. mass export with a TOP?

Try placing your custom format python export code inside the Post-Write event of a ROP Output Driver or File Cache node. Then you can use the Save buttons to write out your geometry.

You can check out an example of this technique here:
https://forums.odforce.net/topic/25260-clarisse-work-flow/?do=findComment&comment=147996 [forums.odforce.net]