I came across an interesting problem when I was doing some RnD for some toon shaders and effects. I want to render effectively only the geometry that is overlapping from 2 different sources. It's sort of a visual boolean, but I don't want the costly geometry operations from calculating that every frame.
I can understand a way of compositing this effect, it would be simply multiplying the alpha of the two objects together, but that takes extra time to render/comp, would have to be done individually for each set of objects, and does not work with reflections. I'd love to get this effect working purely in render.


This is my general idea, for the blue object to be rendered at all, a ray has to pass through both the pink and blue objects. The pink object does not contribute to any shading color, that is all contributed by the blue object. In the case of this diagram, rays A, B, and C would render the blue object and rays D and E would simply pass right through as they hit only the pink or only the blue object.

Unfortunately my knowledge of Houdini is more in geometry manipulation/simulation and not materials and rendering so this task is a bit difficult for me and I'm having trouble finding out where to start. If anyone has any ideas on where to start for this, I would love to hear them!