Hey all!
I am currently learning USD and ran into an issue I am unable to understand/resolve.
When you e.g. have an asset, “box.usd” which contains “box_model.usd” and “box_materials.usd”.
Once you start using box.usd as an asset in layout (if I understand it correctly), every change you make to the data inside that box.usd are live and automatically propagated forward through the pipeline to every scene that uses box.usd (nested references).
Daniel Flood, Technical Lead at UTS Animal Logic Academy, calls this a “push”-pipeline (https://youtu.be/o6VwS7VVx1I?t=383)
Minute 6:32: "Rather than requiring artists to pull updates into their scene we can simply determine that the next time an artist opens their scene there'll be new assets in the scene.“ He furthermore explains: ”We built-in controls to that as well because that can be a bad thing potentially".
And thats exactly the point I don't know how to handle/solve:
I get why you need to have controls for that. What if someone in lighting is currently rendering a version that uses box.usd, and the asset department simulatenously decides to update box.usd? That would update and screw the running renders, wouldn't it?
But how exactly can I prevent such updates from propagating forward prematurely and control when such updates take effect?
Or am I missing something here and that isn't really a problem?
Thanks in advance for sharing your insights!
You’re correct that it is a problem, and not one specific to USD. Most studios have some kind of asset publishing system where each new version of Box (using your example) would be written out to a new version. Other processes happening concurrently (layout, renders etc) would be reading older versions and therefore unaffected by the ‘latest’ version of Box. Once a version is published it is never modified.
One thing usd does do a little different is introduce a concept called an Asset Resolver which in broad strokes is a mechanism to figure out a real, possibly versioned path to an asset using some (likely unversioned) input path. This avoids the need to bake versioned path names into your usd files which can be problematic for many reasons.
Using an Asset Resolver and versioned assets you can pretty much choose whether you work using a push or pull model.
One thing usd does do a little different is introduce a concept called an Asset Resolver which in broad strokes is a mechanism to figure out a real, possibly versioned path to an asset using some (likely unversioned) input path. This avoids the need to bake versioned path names into your usd files which can be problematic for many reasons.
Using an Asset Resolver and versioned assets you can pretty much choose whether you work using a push or pull model.
Thanks for your reply Antc! 
Yes, that workflow (a “pull-pipeline” when I understand correctly) is what I am used to. Someone publishes a version, and the following departments/artists “pull” those updates in as required / told to (updating via shotbuild (and choosing the versions to load) or manual imports etc.).
That is interesting! I would like tho understand this Concept of an Asset Resolver (AR) better. So, when you load a shot in say, lighting, you would bring in the “layout.usd” layer, which is full of references to assets (=other usd-files), wich could be also full of references to other files (materials etc.). So all is heavily nested (correct me please if I am wrong).
How does this AR now determine which versions to load/use? All those files are as said nested in layout.usd. Does this AR change the reference-filepath of the references of nested usd file inside other usd files depending on some rules? If so, how? Can you maybe give me an example how that could work? I tried to read about it here: http://graphics.pixar.com/usd/docs/api/ar_page_front.html [graphics.pixar.com] but I have to admit that it still goes a bit over my head.
Thanks again to the community for the support! I hope this also helps others and not just me. The topic is for sure super interesting!
antc
Most studios have some kind of asset publishing system where each new version of Box (using your example) would be written out to a new version. Other processes happening concurrently (layout, renders etc) would be reading older versions and therefore unaffected by the ‘latest’ version of Box. Once a version is published it is never modified.
Yes, that workflow (a “pull-pipeline” when I understand correctly) is what I am used to. Someone publishes a version, and the following departments/artists “pull” those updates in as required / told to (updating via shotbuild (and choosing the versions to load) or manual imports etc.).
antc
Using an Asset Resolver and versioned assets you can pretty much choose whether you work using a push or pull model.
That is interesting! I would like tho understand this Concept of an Asset Resolver (AR) better. So, when you load a shot in say, lighting, you would bring in the “layout.usd” layer, which is full of references to assets (=other usd-files), wich could be also full of references to other files (materials etc.). So all is heavily nested (correct me please if I am wrong).
How does this AR now determine which versions to load/use? All those files are as said nested in layout.usd. Does this AR change the reference-filepath of the references of nested usd file inside other usd files depending on some rules? If so, how? Can you maybe give me an example how that could work? I tried to read about it here: http://graphics.pixar.com/usd/docs/api/ar_page_front.html [graphics.pixar.com] but I have to admit that it still goes a bit over my head.
Thanks again to the community for the support! I hope this also helps others and not just me. The topic is for sure super interesting!
Scratch
So, when you load a shot in say, lighting, you would bring in the “layout.usd” layer, which is full of references to assets (=other usd-files), wich could be also full of references to other files (materials etc.). So all is heavily nested (correct me please if I am wrong).
How does this AR now determine which versions to load/use? All those files are as said nested in layout.usd. Does this AR change the reference-filepath of the references of nested usd file inside other usd files depending on some rules?
Yes the asset resolver converts/resolves the asset path specified in the usd file (irrespective of how deeply nested it may be) to a real, on-disk path. The usd file containing the path is not modified however - it all happens under the covers at the time the asset path is encountered and needs resolving. An asset path is a special type built specifically for this purpose and therefore very different to a plain string. Via the api an asset path [graphics.pixar.com] can be queried for both its authored (unresolved) path and its resolved path. Another way of looking at it is the path in the usd file doesn't need to be a real operating system style path pointing to a directory or file (although it can be of course). It just needs to be meaningful as input to the resolver.
It's probably worth playing around with the ArDefaultResolver [graphics.pixar.com] to get a feel for how things work. The default resolver is structured loosely around how asset resolution works at pixar. Its a search-path based strategy where a list of directories are searched for an asset. In other words you would need to define your versioning in terms of directories and files rather than say a database or global configuration file.
So, for example in your usd files you might author the path to your Box asset to be something like “Box/Box.usd” and the resolver will search the directories specified in PXR_AR_DEFAULT_SEARCH_PATH looking for that file. Note that unlike an os path “Box/Box.usd” is considered to be a “search path”, not a relative path - relative asset paths must be prefixed with “./”
To get a pull based mechanism working with the default resolver should be possible. I guess each shot could have a directory that contains links to the “current” version for each asset. Updating to a newer version would require updating the link in the directory.
/asset_version_storage
/Box
/v1
/Box
/Box.usd
/v2
/Box
/Box.usd
/v3
...
/shot1.versions
/Box -> /asset_version_storage/Box/v2/Box (symlink)
By placing /shot1.versions at the front of the PXR_AR_DEFAULT_SEARCH_PATH list, the default resolver given the asset path “Box/Box.usd” should resolve to version 2. Hopefully that makes some sense and apologies if there's mistakes - I just typed this out and haven't actually verified this works like I'm saying. There may also be other functionality in the default resolver that would work better for certain cases than what I've suggested. But hopefully that's enough to give you an idea. And of course if you are versioning using some database or something you'll probably need a custom resolver.
Ben and I talk a little more about the same pipeline in this video:
https://youtu.be/ulkJEPflgvk [youtu.be]
But ultimately it's pretty simple; you can separate what you need in a shot from their locations on disk. So rather than saying
'c:/assets/box/v02/box_v02.usd'
you say
'asset=box&v=2'
Or in our case we default to
'asset=box&v=latest'
In terms of lighting shots being clobbered by upstream changes, yes, it's by design.
Dan and I have worked in studios that push always-latest, and in other studios that use strict versioned assets, also called ‘gated’ assets; ie, each department has to explicitly ‘open the gate’ to allow certain versions to flow downstream, and similarly downstream departments have to ‘open their gate’ to take assets into their realm.
Gating is safer, but you risk overly conservative asset updates. Departments hold onto assets for too long, suddenly there's a flood at the last minute, and lighting now have a nightmare trying to work out what changed between v02 and v90 that broke their setup.
Compared to a always-latest workflow, if you have tools that are designed to work procedurally (houdini, katana), they can be more robust with rapidly changing assets. But more importantly in a social engineering standpoint, if 30 shots suddenly stop rendering, it's noticed very quickly, the cause identified, a fix is rolled out.
We always have the option to lock versions if required, and we often do this in the last few weeks of production for lighting. But overall we've found its better to keep this ‘latest is greatest’ mentality, it keeps everyone honest, keeps the production moving.
-matt
https://youtu.be/ulkJEPflgvk [youtu.be]
But ultimately it's pretty simple; you can separate what you need in a shot from their locations on disk. So rather than saying
'c:/assets/box/v02/box_v02.usd'
you say
'asset=box&v=2'
Or in our case we default to
'asset=box&v=latest'
In terms of lighting shots being clobbered by upstream changes, yes, it's by design.
Dan and I have worked in studios that push always-latest, and in other studios that use strict versioned assets, also called ‘gated’ assets; ie, each department has to explicitly ‘open the gate’ to allow certain versions to flow downstream, and similarly downstream departments have to ‘open their gate’ to take assets into their realm.
Gating is safer, but you risk overly conservative asset updates. Departments hold onto assets for too long, suddenly there's a flood at the last minute, and lighting now have a nightmare trying to work out what changed between v02 and v90 that broke their setup.
Compared to a always-latest workflow, if you have tools that are designed to work procedurally (houdini, katana), they can be more robust with rapidly changing assets. But more importantly in a social engineering standpoint, if 30 shots suddenly stop rendering, it's noticed very quickly, the cause identified, a fix is rolled out.
We always have the option to lock versions if required, and we often do this in the last few weeks of production for lighting. But overall we've found its better to keep this ‘latest is greatest’ mentality, it keeps everyone honest, keeps the production moving.
-matt
Thanks to both of you for taking the time to write such detailed replies! Much appreciated!
So what I think I understand:
Asset Resolver:
Instead of using a real file-path as reference-file-path, you can use some sort of expression (like a search query) that is then read by the Asset Resolver (on load) and converted (resolved) into an actual filepath on disk.
Advantage:
The reference paths in the USD files are separated from the actual file location on disk.
USD can e.g. “talk” to shotgun (ftrack etc.), to get the v002, or the latest file's path on disk etc.
That enables you to either go for a pull or push workflow / approach / pipeline:
Pull: you use version numbers, e.g. ‘asset=box&v=2’ as ref-path, or even a hardcoded file-path (skipping the asset resolver). Assets, but also department layers like “layout”, “anim”, “fx” etc. get versioned, and you just load/update to whatever version you need. That “gated” approach might be a bit more conservative, but on the other hand also very save (causing less issues or specials situations that need handling).
Push: you would always default to ‘asset=box&v=latest’ as ref-path (or latest approved, or latest non-declined, etc.) and the asset resolver always returns (the filepaths to) the latest files when loading a shot/scene. (I assume that works kinda similar to sym links). This approach ensures that updates are propagated downstream quickly, but it is at the same time a bit more risky (prone to error), and thus needs thorough approval structures and - to still have control when needed - the ability to lock versions at certain points.
Locking versions -> instead of using “latest”, setting a specfic version to use. There won't be any more updates until the version is unlocked again.
Cases that most likely require locking versions:
Rendering -> Locking versions of renders that have been sent to the farm, so they don't get updated while waiting to be or being processed.
Creative choice -> When there is a decision to stick to / or roll back to an earlier version due to creative reasons / technical difficulties etc. (Although you maybe could just publish that older version again to make it the latest version).
What I don't understand (yet):
How do you actually lock versions?
Is that something the asset-resolver enables you to do? That expression/query ‘asset=box&v=2’ or ‘asset=box&v=latest’ is written into the usd files (instead of actual reference file paths) and I assume once that usd file is written, the asset resolver just reads the query, resolves it, and the DCC app takes that info and loads the files accordingly. But the Asset Resolver can not change the referece path inside of files, or can it?
Locking dependency trees?
If you lock e.g. layout.usd, does the entire dependency tree within layout.usd get locked aswell? Meaning all assets nested inside of layout.usd which consist e.g of model.usd and material.usd etc.?
Thanks again, I can't stress enough how helpfull this discussion is!
So what I think I understand:
Asset Resolver:
Instead of using a real file-path as reference-file-path, you can use some sort of expression (like a search query) that is then read by the Asset Resolver (on load) and converted (resolved) into an actual filepath on disk.
Advantage:
The reference paths in the USD files are separated from the actual file location on disk.
USD can e.g. “talk” to shotgun (ftrack etc.), to get the v002, or the latest file's path on disk etc.
That enables you to either go for a pull or push workflow / approach / pipeline:
Pull: you use version numbers, e.g. ‘asset=box&v=2’ as ref-path, or even a hardcoded file-path (skipping the asset resolver). Assets, but also department layers like “layout”, “anim”, “fx” etc. get versioned, and you just load/update to whatever version you need. That “gated” approach might be a bit more conservative, but on the other hand also very save (causing less issues or specials situations that need handling).
Push: you would always default to ‘asset=box&v=latest’ as ref-path (or latest approved, or latest non-declined, etc.) and the asset resolver always returns (the filepaths to) the latest files when loading a shot/scene. (I assume that works kinda similar to sym links). This approach ensures that updates are propagated downstream quickly, but it is at the same time a bit more risky (prone to error), and thus needs thorough approval structures and - to still have control when needed - the ability to lock versions at certain points.
Locking versions -> instead of using “latest”, setting a specfic version to use. There won't be any more updates until the version is unlocked again.
Cases that most likely require locking versions:
Rendering -> Locking versions of renders that have been sent to the farm, so they don't get updated while waiting to be or being processed.
Creative choice -> When there is a decision to stick to / or roll back to an earlier version due to creative reasons / technical difficulties etc. (Although you maybe could just publish that older version again to make it the latest version).
What I don't understand (yet):
How do you actually lock versions?
Is that something the asset-resolver enables you to do? That expression/query ‘asset=box&v=2’ or ‘asset=box&v=latest’ is written into the usd files (instead of actual reference file paths) and I assume once that usd file is written, the asset resolver just reads the query, resolves it, and the DCC app takes that info and loads the files accordingly. But the Asset Resolver can not change the referece path inside of files, or can it?
Locking dependency trees?
If you lock e.g. layout.usd, does the entire dependency tree within layout.usd get locked aswell? Meaning all assets nested inside of layout.usd which consist e.g of model.usd and material.usd etc.?
Thanks again, I can't stress enough how helpfull this discussion is!
Scratch
How do you actually lock versions?
Is that something the asset-resolver enables you to do? That expression/query ‘asset=box&v=2’ or ‘asset=box&v=latest’ is written into the usd files (instead of actual reference file paths) and I assume once that usd file is written, the asset resolver just reads the query, resolves it, and the DCC app takes that info and loads the files accordingly. But the Asset Resolver can not change the referece path inside of files, or can it?
The asset resolver just allows the asset path in the usd file to resolve to a real on-disk file. An asset can be considered locked I guess when the resolver resolves to some version that isn’t the latest. How it goes about doing that depends on the particular resolver implementation.
I can't comment much on how things work if you place the version number in the usd file asset path with something like ‘asset=box&v=latest’ as it seems to me you would need to republish the usd file to change the version. But maybe that's ok if it fits your workflow.
However the example I showed a couple of posts ago using the default resolver had Box “locked” to version 2 (without the version appearing in the asset path). To achieve the same via a database I presume you would just query your database for the version directly right? I don't have experience with database versioning though because I'm only familiar with the searchpath + symlink approach.
Scratch
Locking dependency trees?
If you lock e.g. layout.usd, does the entire dependency tree within layout.usd get locked aswell? Meaning all assets nested inside of layout.usd which consist e.g of model.usd and material.usd etc.?
No there's nothing automatic like that. Path resolvers don't inherently know anything about hierarchies of references. I mean in theory your own custom path resolver could get as crazy as you want but resolvers typically need to very fast and so try to perform as little work as possible.
If the desire was to lock all the assets used in a layout you'd ordinarily run a script that figures out what's utilized by the layout. That's fairly straight forward using the usd api. If the versioning in your pipeline is super granular it will create more hoops to jump though so it's worth thinking about the pros/cons of having lots of versioned files compared to a smaller set of high-level asset entities.
Hi antc!
Thanks again for your explanations, this (all) makes a lot more sense now!
So, you could aways reference the sym-link inside the USD-file, and then - manually or by using pipeline scripts / tools - define to which file that sym-link actually points to. You could define that those links are pointing to the “latest-”, “latest approved-”, or “specific-version”, which can also be used to lock versions or even whole dependency trees. Another advantage using sym-links: you can change the version without having to republish the usd-file. The path to the sym-link inside the USD-file can be either an actual path or something meaninfull that an Asset Resolver then “translates” to an actual path on disk.
Or, if I get your first post correctly, have a sym-link for each asset within the shot-directory which you can point to every asset version within the asset-library.
Balancing granularity (many assest vs. fewer highlevel assets) is also important, as you have to change/update/manage all the sym-links.
Please correct me should any of this be wrong. I'm also aware that my sumary is super-simplified and in reality requires a lot of brain/pipe-magic to be robust and smooth, especially when combined / hooked up to shotgun/ftrack etc.
But, the concept is way clearer now, so thanks again to the two of you (antc, Matt), for taking time out of your busy schedules to help me figure this out! I'm sure there is still lots more to learn about USD/Solaris, but this is very cool progress!
Thanks again for your explanations, this (all) makes a lot more sense now!
So, you could aways reference the sym-link inside the USD-file, and then - manually or by using pipeline scripts / tools - define to which file that sym-link actually points to. You could define that those links are pointing to the “latest-”, “latest approved-”, or “specific-version”, which can also be used to lock versions or even whole dependency trees. Another advantage using sym-links: you can change the version without having to republish the usd-file. The path to the sym-link inside the USD-file can be either an actual path or something meaninfull that an Asset Resolver then “translates” to an actual path on disk.
Or, if I get your first post correctly, have a sym-link for each asset within the shot-directory which you can point to every asset version within the asset-library.
Balancing granularity (many assest vs. fewer highlevel assets) is also important, as you have to change/update/manage all the sym-links.
Please correct me should any of this be wrong. I'm also aware that my sumary is super-simplified and in reality requires a lot of brain/pipe-magic to be robust and smooth, especially when combined / hooked up to shotgun/ftrack etc.
But, the concept is way clearer now, so thanks again to the two of you (antc, Matt), for taking time out of your busy schedules to help me figure this out! I'm sure there is still lots more to learn about USD/Solaris, but this is very cool progress!
it always makes me nervous when pipeline allows pushing changes onto unsuspecting artists or onto live renders on the farm
changing symlinks is source of major headache, even though I see the appeal of not looking back and just power through and hoping nothing breaks and if it does then fix later
but in such workflow if someone asks to go back to old version of the scene, which used to assemble hundreads of versions of different assets and you open your file and everything is different than it was just because the assets resolved in different versions you will never piece it back together unless you have very good pipe in place which will allow you to know the state of that file at the time of render or so
I very much prefer resolving and updating versions consciously, you still can have tools in place that warn user or ask about old versions in the files , but that let it up to user to decide whether to update or not, and automatize that as much as you want
in any case I prefer artist submitting a render, caching a sim or simply doing their work to know that something has updated to be able to double check that it still works if they are ready to ingest that version, sometimes you simply need to get stuff out and new version would require change in your setup so you just output you new work using older version and work on update for new one next
I know I derailed from purely .usd asset versioning to the actual production assets, but I think it's a very similar situation and requires very robust and transparent tools
changing symlinks is source of major headache, even though I see the appeal of not looking back and just power through and hoping nothing breaks and if it does then fix later
but in such workflow if someone asks to go back to old version of the scene, which used to assemble hundreads of versions of different assets and you open your file and everything is different than it was just because the assets resolved in different versions you will never piece it back together unless you have very good pipe in place which will allow you to know the state of that file at the time of render or so
I very much prefer resolving and updating versions consciously, you still can have tools in place that warn user or ask about old versions in the files , but that let it up to user to decide whether to update or not, and automatize that as much as you want
in any case I prefer artist submitting a render, caching a sim or simply doing their work to know that something has updated to be able to double check that it still works if they are ready to ingest that version, sometimes you simply need to get stuff out and new version would require change in your setup so you just output you new work using older version and work on update for new one next
I know I derailed from purely .usd asset versioning to the actual production assets, but I think it's a very similar situation and requires very robust and transparent tools
As Matt detailed a few posts ago there's definitely pros and cons to both approaches.
I mostly work in an asset department (environments) and from my perspective we don't typically want tons of versions (if any) in play at the same time. 1000 assets with half a dozen versions in use in shots is really an inventory of 6000 assets. Also, 9 times out of 10 an asset update is to address a note and folks want to verify that the update hits the note as quickly as possible. Getting eyes on things sooner rather than later is usually the overarching goal. The more assets in a shot the harder and more time consuming it is to manage them all manually too.
That said I think it does depend on how much time you've got. If a bunch of shots have to leave the building tomorrow lunchtime then I agree the last thing you need is the rug pulled from under you with a bunch of updates. That's the most common scenario I see for requiring version locking.
Live renders on the farm shouldn't get updates I wouldn't have thought. Typically a single version of each asset should get resolved at the start of the job (AR can run independently of any usd file) so that all the various tasks see the same versions.
Just to clarify too - the push approach doesn't mean individual artists can't buy out of updates by creating local overrides to specific versions if that works better for them. It makes complete sense that an fx artist would want a stable set of asset versions while they're iterating heavily and firing off their own renders and what not.
I mostly work in an asset department (environments) and from my perspective we don't typically want tons of versions (if any) in play at the same time. 1000 assets with half a dozen versions in use in shots is really an inventory of 6000 assets. Also, 9 times out of 10 an asset update is to address a note and folks want to verify that the update hits the note as quickly as possible. Getting eyes on things sooner rather than later is usually the overarching goal. The more assets in a shot the harder and more time consuming it is to manage them all manually too.
That said I think it does depend on how much time you've got. If a bunch of shots have to leave the building tomorrow lunchtime then I agree the last thing you need is the rug pulled from under you with a bunch of updates. That's the most common scenario I see for requiring version locking.
Live renders on the farm shouldn't get updates I wouldn't have thought. Typically a single version of each asset should get resolved at the start of the job (AR can run independently of any usd file) so that all the various tasks see the same versions.
Just to clarify too - the push approach doesn't mean individual artists can't buy out of updates by creating local overrides to specific versions if that works better for them. It makes complete sense that an fx artist would want a stable set of asset versions while they're iterating heavily and firing off their own renders and what not.
Hey all,
sorry all for the late answer - busy days!
I can only agree with what antc/Matt said. Tamte has pointed out a valid challenge of the “push” approach, and building in some flexibility for custom control (opt out etc.) is certainly a good idea.
Thanks again to everyone for all the usefull contributions to this topic!
sorry all for the late answer - busy days!
I can only agree with what antc/Matt said. Tamte has pointed out a valid challenge of the “push” approach, and building in some flexibility for custom control (opt out etc.) is certainly a good idea.
Thanks again to everyone for all the usefull contributions to this topic!
Having almost no Solaris or USD experience yet, take the following with a grain of salt…
Personally I would prefer Tomas's approach as it's more deterministic, hence predictable. It's the same reason why one doesn't lightly change software versions mid-production, or only in a way that doesn't disturb existing renders.
Maybe one could store, via some method, the version that “latest” points to when rendering / saving a work file and a toggle when submitting renders could swap the “latest” string in a resolver path to that version? That would involve application specific scripting I suppose, unless the resolver itself could somehow do it.
-Jon
Personally I would prefer Tomas's approach as it's more deterministic, hence predictable. It's the same reason why one doesn't lightly change software versions mid-production, or only in a way that doesn't disturb existing renders.
Maybe one could store, via some method, the version that “latest” points to when rendering / saving a work file and a toggle when submitting renders could swap the “latest” string in a resolver path to that version? That would involve application specific scripting I suppose, unless the resolver itself could somehow do it.
-Jon