Hi,
Are file rule supported in Karma XPU? we are having a hard time moving production from H19.5 to H20 and getting XPU to match CPU.
out of the box H20 renders our hip files differently. This is a big problem for long forms that are in production for a year while we want to get new features of houdini. For tests we created a simple setup with 3 textures:

utility-sRGB-texture JPG
aces_cg EXR
utility-linear-sRGB EXR

our ocio file rules are very basic just to cover the test case. the textures have no colour space in the names (on purpose to limit the possible issues)



We have a few problems:

1. we do not know how to make H20 render the same way as H19 to start the transition. We use OCIOv1 in production and with the same config we see differences in how the textures are handled. One major difference is that H19 assumed EXRs to be ACES_CG while H20 assumes them to be linear-sRGB. We tried to play with render space in the ocio settings and created OCIO v2 config trying to create rules to load the textures the same way.
2. OCIO file rules seem to have no effect until we provide HOUDINI_OCIO_FILENAME_COLORSPACE environment variable
3. HOUDINI_OCIO_FILENAME_COLORSPACE affects XPU and CPU differently and also gives us different results almost for each mode.
4. with HOUDINI_OCIO_FILENAME_COLORSPACE=2 Karma CPU seems to be obeying the OCIO file rules fine but XPU seems to be using only the "default" which we set to aces_cc to very clearly see when it is being used. All of the other rules are ignored by XPU.




clearly I'm missing something and I do not understand how to use this system. I thought that by using OCIO v2 which ships with houdini with a few basic rules I should be able to get the test case going fine.

any help very welcome :-)

I feel your pain. Subscribing to this thread since I genuinely do not get Karma's OCIO handling at all - hoping somebody figures it out and posts here!

pawel-fin
Hi,
Are file rule supported in Karma XPU? we are having a hard time moving production from H19.5 to H20 and getting XPU to match CPU.
out of the box H20 renders our hip files differently. This is a big problem for long forms that are in production for a year while we want to get new features of houdini. For tests we created a simple setup with 3 textures:

File rules are supported in H20. OCIO support in H19.5 was not as complete as in H20. In H19.5, Karma had a hard-coded Linear Rec 709 working space. In H20, we use scene_linear. In H19.5 texture conversion may have used the "built-in" Houdini conversions rather than OCIO (in some cases). In H20, color handling is much more consistent and aligned with OCIO (throughout the whole package).

Trying to match H19.5 and H20 renders can be challenging because H19.5 was broken in many subtle ways.

If there's a particular use case that you need help matching between the two versions, please submit a bug/issue with an example and we can try to help sort things out.

Changing rules has no effect on texture appearance for me. Never has. Make a grid with a JPG texture attached, render in Karma CPU or XPU, looks fine. Change the rule to ACEScct (for example), nothing changes. No amount of resetting Karma or updating textures has any effect. I thought maybe Houdini needs a restart for the changes to take effect, but all the changes to rules are reverted after restarting. It's all very confusing.

BrianHanke
Changing rules has no effect on texture appearance for me. Never has. Make a grid with a JPG texture attached, render in Karma CPU or XPU, looks fine. Change the rule to ACEScct (for example), nothing changes. No amount of resetting Karma or updating textures has any effect. I thought maybe Houdini needs a restart for the changes to take effect, but all the changes to rules are reverted after restarting. It's all very confusing.

The metadata in the image is used before any rules so if your image has incorrect metadata that can throw it off. The metadata kind of has to take precedence since it's the only way multiple color spaces in the same file can be supported.

Hmm, interesting. Is there any documentation about that?

I just ran some quick tests to try to pin things down more and here's what's happening so far:

Houdini 20.0.610 using the official ASWF "CG" config. Tested both Karma XPU and CPU. Mtlx Standard Surface with Mtlx Image plugged in to base_color channel. When changing textures I always went through all the Update Textures/Reset Karma/etc. rigmarole. Testing with ACEScc since it looks so different and it's obvious when things are in that color space.

- image.ACEScc.jpg - not displayed in ACEScc.
- image_ACEScc.jpg - not displayed in ACEScc.
- Arnold render with ACEScg metadata and ACEScc in file name - not displayed in ACEScc.
- Linear EXR from HDRI Haven with ACEScc in file name - not displayed in ACEScc.
- Set up custom ACEScc rule in Houdini with * glob and jpg and exr extensions - not displayed in ACEScc.
- Clicked Apply and Accept after setting up rule, rule is showing in list. Closed OCIO Settings window. Open same window again a second later, rule is gone.

Long story short, nothing works! What's the intended workflow here? This is so different from other apps that I assume I must be missing something major.

Clicked Apply and Accept after setting up rule, rule is showing in list. Closed OCIO Settings window. Open same window again a second later, rule is gone

Do you have a system wide OCIO environment variable?

When you add a custom rule through the dialog, Houdini makes a copy of the config with the new rule and stores it in your Houdini pref folder, however the system OCIO varible will take precedence. So either point your env var to the new file or copy/paste the new rule to whatever config your env var is pointing to.

I can't help much past this suggestion, I don't seem to have any glaring issues with OCIO on my end.

I see, that makes sense. It's a poor user experience since there is no indication of that in the UI, but at least that answers one of my questions, thanks!

there is no indication of that in the UI

Admittedly easy to miss but the bottom of the OCIO dialog will display the path to the config it's sourcing. After adding a new rule you'll see the path change to the new file in your prefs folder. Close and launch the dialog again and you'll see it's back to using the system OCIO path.

Wow, so it does. That is sneaky!

BrianHanke
Wow, so it does. That is sneaky!

you have to relaunch houdini for any changes to OCIO to take effect.

I switched to the native Houdini config (no ASWF CG system-wide config) and it is picking up new rules now without restarting Houdini. Karma CPU is at least, XPU still nothing. Color space tokens in file names also still not working.

I'm also very curious about the metadata topic. Let us know if you learn anything!

I'll mess around with that env variable. I read the docs and it's not quite clear to me how it works. I'll just have to try the different values and see what happens.

mark

pawel-fin
Hi,
Are file rule supported in Karma XPU? we are having a hard time moving production from H19.5 to H20 and getting XPU to match CPU.
out of the box H20 renders our hip files differently. This is a big problem for long forms that are in production for a year while we want to get new features of houdini. For tests we created a simple setup with 3 textures:

File rules are supported in H20. OCIO support in H19.5 was not as complete as in H20. In H19.5, Karma had a hard-coded Linear Rec 709 working space. In H20, we use scene_linear. In H19.5 texture conversion may have used the "built-in" Houdini conversions rather than OCIO (in some cases). In H20, color handling is much more consistent and aligned with OCIO (throughout the whole package).

Trying to match H19.5 and H20 renders can be challenging because H19.5 was broken in many subtle ways.

If there's a particular use case that you need help matching between the two versions, please submit a bug/issue with an example and we can try to help sort things out.

thank you for the help.

HOUDINI_OCIO_FILENAME_COLORSPACE env var is meant to turn back the built-in houdini conversions and I also tried switching rendering space to rec709 on top but no combination of the two gave me the same results as h19.

from the docs:
>When set to 0, detection is not performed and Houdini’s native heuristics are used to determine the colorspace. When using OpenColorIO 2.0 and above, this uses the file_rules defined in the OpenColorIO config file.

this should make it like H19, shouldn't it? file rules seem to have no effect on the outcome in this mode at all btw. Below karma render and ocio rules that should make this render look very broken but they do not.





Matching to H19 is one issue. The larger problem is that the new system with the rules seems to be broken as the results are different between CPU and XPU Karma renders. With images having no color space in their names, changing the HOUDINI_OCIO_FILENAME_COLORSPACE has massive impact on the results. In some cases the ocio rules are totally ignored in some behave differently and often do not match between xpu and CPU. I spent hours looking for patterns and trying to understand the logic. I will post my test scene with examples.

attached the test scene

for a simple config:


matrix of correct results:

jsmack
The metadata in the image is used before any rules so if your image has incorrect metadata that can throw it off. The metadata kind of has to take precedence since it's the only way multiple color spaces in the same file can be supported.

Small correction. It turns out that if there are any file rules, they need to take precedence over the metadata.

This post: https://www.sidefx.com/forum/topic/93002/ [www.sidefx.com] pointed out that if we relied on metadata first, textures like ACEScg_ColorChecker.exrwould not be interpreted as being in ACEScg.

This is a little unfortunate since if you have any file rules which match a filename, the metadata will be ignored.

Ideally, there would be some way to say apply priorities to file rules, so you could have something like: "If the filename has ACEScg, force the colorspace to ACEScg, otherwise, use metadata, otherwise fall back to the file extension rule.". Or maybe have a file rule that says to check the file's metadata.

This gets more complicated because there's no "standard" way of specifying color space metadata in OpenEXR files (there are various competing methods in play).

I have a bit of progress to report: setting HOUDINI_OCIO_FILENAME_COLORSPACE = 3 enables the use of file name tokens in both CPU and XPU. All works as expected. However, this setting seems to ignore file rules in the config, so it's not perfect. I haven't found any way yet to get XPU to recognize those rules.

mark
Or maybe have

Personally

mark

jsmack
The metadata in the image is used before any rules so if your image has incorrect metadata that can throw it off. The metadata kind of has to take precedence since it's the only way multiple color spaces in the same file can be supported.

Small correction. It turns out that if there are any file rules, they need to take precedence over the metadata.

This post: https://www.sidefx.com/forum/topic/93002/ [www.sidefx.com] pointed out that if we relied on metadata first, textures like ACEScg_ColorChecker.exrwould not be interpreted as being in ACEScg.

This is a little unfortunate since if you have any file rules which match a filename, the metadata will be ignored.

Ideally, there would be some way to say apply priorities to file rules, so you could have something like: "If the filename has ACEScg, force the colorspace to ACEScg, otherwise, use metadata, otherwise fall back to the file extension rule.". Or maybe have a file rule that says to check the file's metadata.

This gets more complicated because there's no "standard" way of specifying color space metadata in OpenEXR files (there are various competing methods in play).

Personally, I think that the metadata should not be used when the file rules are in place. With both active it will be very hard to predict the outcome and debug problems. I expect the rule that says "all EXRs are ACEScg" to take over everything else.
Additionally, controlling metadata that is coming from a variety of DCC apps is not easy.
If the file rules take over metadata like you described then I think this is the correct behaviour.

BrianHanke
I have a bit of progress to report: setting HOUDINI_OCIO_FILENAME_COLORSPACE = 3 enables the use of file name tokens in both CPU and XPU. All works as expected. However, this setting seems to ignore file rules in the config, so it's not perfect. I haven't found any way yet to get XPU to recognize those rules.

HOUDINI_OCIO_FILENAME_COLORSPACE = 1

makes XPU obey the default rule at least. I think 1 and 2 affect XPU but are not right.
The easiest way to test it is by setting all rules to ACEScc. This should make the textures to have huge contrast and saturation.

BrianHanke
I have a bit of progress to report: setting HOUDINI_OCIO_FILENAME_COLORSPACE = 3 enables the use of file name tokens in both CPU and XPU. All works as expected. However, this setting seems to ignore file rules in the config, so it's not perfect. I haven't found any way yet to get XPU to recognize those rules.

leaving it at default also works as expected. File rules in the config are followed for both CPU and XPU. I have seen one instance of a the extension file rule being ignored by XPU when tagging exr as ACEScg where metadata is not present in the image. CPU had correct behavior in every case. I'm trying to narrow down when exactly XPU doesn't follow the rules.