What I am trying to do:
I would like to pre-compile a custom vop function (specifically a function, not a material with surface and displacement). This pre-compiled function's outputs would be used in a larger shader network to generate the base_color, roughness, metallic and disp inputs of a Principled Shader node. The reason I want to pre-compile this function (contains a lot of nodes) is to reduce the Mantra startup time (which, in my particular use case, would be > 6 min without pre-compiling and < 30s with pre-compiling).

Approach 1:
I first attempted to use the Material Builder's vex code caching feature to pre-compile my custom vop function. I used Parameter vop nodes to add custom outputs to the Material Builder node and wired these into the corresponding inputs on the Principled Shader. This worked for base_color, roughness, and metallic, but when plugging the Material Builder's custom output into disp it generated correct displacement but bad normals (see the attached screenshots which show the bad normals in one and what the normals should look like in the other).

Approach 2:
Since the above method was not working, I instead tried manually compiling the custom function to vex code with Save > Save VEX Code to File, and then #including this vex code inside a Snippet vop. This approach solved the bad normals issue seen in Approach 1, but it has a caveat. The issue with Approach 2 is what I will refer to as "tweak time", the time to see something update in the Mantra render view (with Mantra already running) when a parameter is changed/tweaked in the shader network. If I use the snippet version (Approach 2) of my custom function as an input to base_color for example, I get a way longer tweak time than when using the Material Builder version (Approach 1) of my custom function. In my actual shader setup this amounts to a 2-minute tweak time vs a 1 second tweak time (i.e., the 2min tweak time is approximately the same amount of time it would take to just restart mantra, so something isn't right...). Having to wait 2 minutes every time I tweak a parameter in the shader network is not ideal.

Questions:
1. Can the bad normal issue associated with Approach 1 be solved, if so, how?
2. Can the tweak time issue with Approach 2 be solved, if so, how?

I have attached a simplified hip file demonstrating the issues associated with the above two approaches.

Any help resolving this would be much appreciated, so thank you in advance.

No one has replied yet, so I am wondering if the question is unclear?
More generally, what I am asking above is how people go about creating an “uber” (lots of nodes) VOP function without incurring a major Mantra startup time hit (due to compiling the vop node network to vex every time). The purpose of this “uber” VOP function would be to generate basecolor and disp for the principled shader.

If anyone from the SideFX staff with experience in this area could chime in that would be greatly appreciated.

In general, Approach 1 is preferable because it keeps the original network with the VEX source code.

Ie, in the Approach 2, when you do Save > Save VEX Code, you detach VEX source code, and you need to keep track of the original .hip file that generated it.


Also, Mantra does look-aside byte-code caching (see `vexcache --help`). It it looks up the compiled byte-code based on the source code hash. When you use snippet and you "tweak" it, Mantra will include the large chunk, but the tweak will alter the hash, Mantra won't reuse bytecode and will need to re-compile the snippet, including the included large chunk of code.

With Material Builder approach, the large chunk is isolated from the main network (and if you make an HDA out of it and toggle on "save cached code" checkbox that VEX source code will be save to the HDA's section, like Principled Shader, for even more speed), which Mantra will cache to the look-aside table when it compiles it for the first time. So, when you use Material Builder in your new VOP network, the VOPs will generate a shader function call for it, so the tweaked VEX is very small: it has a function call (but no function body). The function body does not need to be re-compiled since the source code has not changed, so Mantra can reuse the look-aside VEX bytecodes.

As to your problem with bad normals, it almost sounds like Mantra bug? If it works for the displacement amount, then it should work for the normals too.

I'm looking at the VEX source code (RMB > VEX/VOP Options > View VEX Code...) for the principledshader_approach1 and 2 and I cannot see why the final `N` variable would be different between them. It's just the output (value of the export variable) of the of the final principled shader call.

One trick you can try is to select both the materialbuilder1 and principledshader_approach1 VOPs and then from the network pane menu do Edit > Collapse Selected into Material to see what the actual final shader looks like in terms of VOPs, and then see if you can tweak the wiring in that top-level network to get the correct normal.

Thank you for your response rafal! Given your explanation of Mantra’s look-aside byte-code caching, I can now see why I was observing the tweak time issue with Approach 2 and not Approach 1. I also agree, Approach 1 is preferable.

Here is some additional information I gathered on the bad normals issue with Approach 1 through further testing:

In my tests, the shading normals are incorrect if they are being calculated by a displace node which is outside the Material Builder node generating the height values (in the example hip I provided, the displace node is inside the Principled Shader). I was able to get correct normals by generating the displaced P and N inside the Material Builder with a displace node instead of relying on the displace node inside the Principled Shader. But this means that the Material Builder is outputting the displaced P and N instead of a height value, and unfortunately in my use case I need it to output a height value so the value can be used as part of a larger shader network.

Furthermore, I noticed that disabling "Add Bump to Ray Traced Displacements" on the Principled Shader also removes the bad normals, but I need that to be enabled so it doesn't solve the issue for me.

Note that this bad normals issue also shows up when “True Displacements” is disabled on the Principled Shader (the bump doesn’t show up at all).

I have attached an updated hip file to show the additional three tests above if you are interested.

Questions:
1. Does the additional information I provided above give you any more ideas of a potential workaround for the bad normal issue, or is my best hope to submit this as a bug to SideFX?
2. Since “Mantra is limited to critical bug fixes” [www.sidefx.com], can I still submit this as a bug, or would it be immediately rejected because it is not “critical”? It is critical to my current project…

I submitted a bug report for this. Hopefully SideFX fixes it soon.