Most objects in my ‘.hip’ will be one of, or made up from, ~500 ‘.bgeo’ files. When I say “frequently used”, I mean hundreds-of-thousands of times.

Importing them individually each time I want to use them is out of the question, because there is a lot of manual overhead associated with locating each file by name.

My next idea is to import all the files, then add each one to the gallery manually. Afterwards it would just be a case of dragging them from the tool palette to the node editor, which is not so bad. Not sure if that is the best setup though.

So what is the best way to manage them?

The problem with adding nodes to the gallery is that each time you add that node from the tool palette it creates a copy, so if you update the file path of a file node, for example, the nodes previously copied will not update.

There are a few ways to solve this, but they also have their problems.

Reference expressions can be used as the values of the nodes in the gallery, which would to point a master node somewhere. This means that in the worst case, only ~500 nodes will need to be updated if the file structure or naming convention in the file system changes. Though if the master node is moved at all, even just into a subnetwork, all the references would need to be changed. Using this method would improve the ability to reorganise the file system, but would just transfer the problem into Houdini.

Using global variables in the nodes instead of references solves this because any updates to the variable values are reflected in those already in use. This means however, that the variable names cannot be changed, or again, all the node expressions would need to be changed as well. Less severe, but reorganising the now very large global variable list is no longer practical.

Still not sure this is the best way to do it. In programming integrated development environments (IDEs), when variable names are updated, so can all the uses of that variable name throughout the code base. I don't know if Houdini has such a capability, it should for the sake of consistent naming conventions and the ability to reorganise projects.

What I would also like is maybe a standalone file manager. Within it files could be moved around and renamed, and the programme would update all the references to them in selected ‘.hip’ files.

I was thinking, you might find using Python Source Editor helpfull.

There you could create aliases that reference the directory/file locations on your system.

Then if your directory/files change, you only have to update the Python Source Editor.

You could also write Python Scripts to update the aliases too (what they're ‘pointing’ to), in Houdini, or even externally then copying and pasting back into the Python Source Editor.

BabaJ
I was thinking, you might find using Python Source Editor helpfull.

There you could create aliases that reference the directory/file locations on your system.

This would probably be easier to manage than global variables. It might still suffer from not being able to renamed the aliases without breaking the expression references. However, if it is possible to see the code of those expressions in files somewhere, I could just use ‘find and replace’.

BabaJ
You could also write Python Scripts to update the aliases too

Not sure what you mean by this. What the aliases point to would be file paths. If a file is moved or renamed in the file system, how would the script know in order to update the path?

It might still suffer from not being able to renamed the aliases without breaking the expression references.

Nope..you don't change the aliases…that's the whole purpose.

You use the alias in the expression. If the file position changes and you've updated the Python Source Editor where the aliases are, the expression doesn't break.

how would the script know in order to update the path?

It knows by the way you wrote the script.

Of course if you don't know how your files are stored on your computer or how you may want to change it;

Then yeah the script won't know because you don't.

If your not sure, that's probably what you need to work on.

Get a system down for how you want to structure your ‘database’ of files on your system.

Only then can you work with scripts to make changes. It has nothing to do with Houdini - all the tools are already there in Houdini.

BabaJ
Nope..you don't change the aliases…that's the whole purpose.

I do have a naming convention for the objects, but as new ones are created I might find I want to rename some that already exist. Using aliases means the filenames can be changed, but practically the alias themselves cannot be, and as the project grows the naming convention starts to break because the names of the aliases and the names of the files no longer match up.

This is why refactoring is useful, and I'm trying to get as close as possible to that in Houdini.

Maybe if I can access all the expressions from all the nodes I could create a find and replace script, ‘find_and_replace(x, y) # Find and replace x with y.’

BabaJ
Of course if you don't know how your files are stored on your computer or how you may want to change it;

Then yeah the script won't know because you don't.

If the file structure is set, then what would the script be updating?

Using aliases means the filenames can be changed, but practically the alias themselves cannot be,

Not sure why you would think this is an issue. The suggestion of using alias to reference a file is in case the file location changes. There is no need to change the alias. You've got your object.

And the node that is ‘importing’ your file can be named something completely different than the alias and changed when ever you want without affecting or needing to change the alias; You have the file/object imported.

And in changing that objects name, if other nodes are referencing that object in their parameter expressions, then those expressions get updated too.

This isn't always the case say in if your doing a channel reference in with your own code in a wrangler, and there may be other places in Houdini where it doesn't work out as in a typical parameter expression.

If the file structure is set, then what would the script be updating?

The script would be updating where the files are because you alluded to that possibility…

This means that in the worst case, only ~500 nodes will need to be updated if the file structure or naming convention in the file system changes.

Probably best at this point, if you want more help on this matter…to do a ‘mock up’ smaller version of what you intend to do in a hip file and upload it here.

Not sure of the setup but it sounds like you want an asset management system like ftrack.

The objective is to be able to practically refactor the project even if it has a great deal of objects. Aliases are almost there, but the aliases themselves cannot be renamed. This is a problem because if an object file is named ‘a_b_c.bgeo’ then the alias will be call ‘A_B_C’, but if the file name changes to ‘a_b_d.bgeo’ the alias will stay the same, and it will become confusing.

This isn't a short-term, closed project, so refactoring later for everything is necessary.

I'll probably have a look at what is possible with Python scripting.

This is a problem because if an object file is named ‘a_b_c.bgeo’ then the alias will be call ‘A_B_C’, but if the file name changes to ‘a_b_d.bgeo’ the alias will stay the same, and it will become confusing.

My original suggestion of using aliases was because you alluded to the possibility that the file structure would change.

But then you also suggested in another post that the file structure would not.

In that case aliases are not necessary.

But if your file structure might change, when you go to change the reference to the file anyways, you can also change the alias, because your right there. No issues.

You were asking before how a script can know how to make ‘changes’.

Well if your naming your aliases to represent the file names then you should be able to do something like this for both file name and alias:

From -

A_B_C = “C\documents\MyBgeos\a_b_c.bgeo”

To -

A_C_D = “C\documents\MyBgeos\a_c_d.bgeo”


But part of my suggestion to use aliases was also assuming the possiblity of using the same referenced file like “C\documents\MyBgeos\a_b_c.bgeo” being imported into more than one file node. Then you only change it once rather than 5 times for each node( if 5 different nodes where reference that same file ).

Another approach without aliases say in the case of the bgeo files is to just make a python “table”.

List all the file nodes that will be importing your bgeos and assign the string value to the nodes import parameter that represents the file.

And yes you still have to make changes to the table if the file name changes.

Maybe you can use some Python function like listdir()

Just tried this in Python Shell:


This then lists out all the primary directory folders under C: drive.

From a cursory glance it looks like Python has other directory related functions.

That being the case, if you develope some kind of ‘system’ to how you structure your directories and files then you can write corresponding python functions to work with them.

Say for example you have a directory name ‘Face Bgeos’ and in that directory you have 5 files named:

A_A.bgeo
A_B.bgeo
A_C.bgeo
A_D.bgeo
A_E.bgeo

And say your Python script gets the names of all those bgeo files, but because it gets them in sequence, you ‘fetch’ the file name by the index of the order it lies in that directory.

So say you have a node that uses A_C.bgeo. Well your script didn't get A_C.bgeo by name. It got it by it's position in the list (third one). So if A_C.bgeo name changes to B_C.bgeo, you node will still be using the newly name file.

If you put some thinking into how you would like to structure the directories and files for the long term ( how it may expand and change ) and look into additional python related functions like listdir(),

I'm pretty sure you can come up with some robust solution.

Good point, the ‘Geometry File’ parameter could just be a reference to a Python list, rather than an alias.

There is one thing I forgot to mention in my last few replies, which is the names of the nodes. I would like those to be based on the file name with a number on the end for copies, ‘filename_1’, ‘filename_2’ etc. Not really an issue if it is possible to change the names using Python.

What I might do is create a poor man's file manager as a Python script, which would update all the references in the ‘.hip’ file. Something like
and

BabaJ
And say your Python script gets the names of all those bgeo files, but because it gets them in sequence, you ‘fetch’ the file name by the index of the order it lies in that directory.

Oh, so the script you suggested wouldn't really update the references if the file name or location changed, it would just populate a list of file locations in the first place.

Oh, so the script you suggested wouldn't really update the references if the file name or location changed, it would just populate a list of file locations in the first place.

Actually, the script would update the references, but the way I describe was only in part of whatever the ‘larger’ script was doing.

In this case, in this particular directory and for the particular node that was importing the file. The script was always getting the third file in that directory regardless of what it was named.

Of course, that requires that you always make sure you don't change the order of the files in a directory, although you could always change their names. Because the script is taking file @index n, copying the ‘string’ value of that file and inserting it into node that needs that string information to find the file. Then just by looking at node paramter that has referenced file you always know what it is not matter what you changed the name too.

This was only a simple possible example.

Your directory/file structure might be more ‘complex’ with the intent to change down the road and in this case my example doesn't work at all. But hopefull might give you some ideas of the tools available and like you say might be able to develop a ‘poor’ mans file manager.

And on that note. I am wondering if Python can access directory/file information on ones system. Maybe it also has toold for renaming directories too.

If that's the case, why not do the Ptyhon ‘poor’ mans file manager that works across both sides. Not only does it have the capacity to rename files and directories, but also move files around.

Once you have that ‘system’ down, it would be ‘easy’ ( meaning algorithm wise )to ensure updates are reflected correctly in Houdini.

It's an interesting ‘problem’ you have, hope you devise a way that is workable.

FYI, search and replace can be done with the opchange hscript command:
https://www.sidefx.com/docs/houdini/commands/opchange [sidefx.com]

BabaJ
If that's the case, why not do the Ptyhon ‘poor’ mans file manager that works across both sides. Not only does it have the capacity to rename files and directories, but also move files around.

That is exactly what the ‘manager_move()’ script would do. Move files or directories, and update all the references to those files in Houdini. And that script could just be run as a command in the ‘Command Line Tools’ Houdini utility.

edward
FYI, search and replace can be done with the opchange hscript command:
https://www.sidefx.com/docs/houdini/commands/opchange [sidefx.com]

Will likely use this, thanks.

edward
FYI, search and replace can be done with the opchange hscript command:
https://www.sidefx.com/docs/houdini/commands/opchange [sidefx.com]

Do you know if there is a hou equivalent of this?

hou.hscript()

Thought so.

Hi,
Maybe I am not understanding the problem here (asset management is way above the scale of projects I have) but wouldn't a list of shared HDA will solve the problem? You could even add HDA libraries that refers to different type of objects and even create tools that quickly creates the thumbnail for the each of the objects.
Granted, like adding obj to the library, there will be a huge overhead in the beginning. But later you will have the ability to update automatically individual assets in your team and even versioning of each object.
Kind of like the system Jordi Bares propose in his houdini for Softimage users guide for asset management but addapted to your needs.

Bests,
Nicolas.

Not entirely sure what the advantages are for using Houdini Digital Assets, a custom node isn't necessarily what I need, which is just to instance these ~500 objects as easily as possible.

Right now I've discovered that using packed disk primitives makes no difference in performance if each of the objects are separate ‘Geometry’ nodes, which can only handle ~2000 objects, so everything I do must be within ‘Geometry’ nodes which can handle ~350,000. This is leading me to toy with using ‘Copy and Transform’ nodes instead of using individual ‘File’ nodes for each instance, because hundreds-of-thousands of ‘File’ nodes would be impractical. The problem with ‘Copy and Transform’ is you can't delete the original instance that is being copied from, otherwise all the copies disappear as well. I'm now trying to find a way to hide the original without hiding the copies.

Edit: Found that you can just use ‘Blast’ nodes to hide the original copies. Conversely I have realised that you can't delete copies, instead you must blast them, which makes for a very strange workflow, where if there are 100 copies and the first 50 need to be deleted and the last 50 need to stay, the first 50 must be blasted. So effectively every deletion takes up some memory and increases the cook-time.

Still having trouble. I'm trying to copy the ~500 ‘File’ nodes easily, which needs to be done on the geometry level so the packed disk primitives are utilised. The problem with creating a ‘Copy and Transform’ node for each individual ‘File’ node is it becomes quite difficult to find and increment one when there are ~500, especially considering these copies need to be created hundreds-of-thousands of times.

Is there a faster way to create instances?

Edit: Reworded.

Edit 2: Considering incrementing the ‘Copy and Transform’ nodes in batches and creating a sort of palette of objects to drag the instances from, but even that would take a relatively long time.

Edit 3: Started a new thread for this question [sidefx.com].