…I have already stated I don't think Houdini is capable of having the workflow I want…
Of course Houdini isn't capable of doing what you want. Neither is any other software out there.
The reason being is because you simply want an enormous amount of objects with most or not all of them having the capacity to easily manipulate quickly to your liking, individually and uniquely, which means you have to do it manually for many different objects and/or sub-components of those objects.
… if you consider a house is made of ~8000 objects…. but there are a lot of things that can't be merged… For example, a humanoid is made of ~50 objects that are animated separately. 100 characters on screen is ~5000 objects. Then consider trees that can sway where each branch might be a separate object and so on. It all adds up very quickly….
If it weren't for the fact that you want to retain the capacity to manipulate uniquely many different objects you could simply use the copy and stamp or instancing nodes and use scripts to drive all those objects that can change each and every aspect of those objects with an algorithm that follows a pattern.
But I suspect this still would not be good enough for your intentions. What if you want to change some of those objects out of those pattern of changes seperately - you have to get in there to make the changes manually.
This is the kind of thinking I keep seeing from you. And that's not criticising, I think your simply ambitious.
Even if you could hire some software developers in which you think they are going to give you what you want, they won't be able to do it; Not with the current state of technology.
Maybe at some point with more development and access to AI technology, you would be able to get what you want:
In essence basically something that reads your mind as you look at your screen, upgrading and changing what you want without the need to manually pick out an object/s and using mouse/keyboard make the changes.