Hi, I'm working on a rather peculiar project where I need to output images that contain a deep description of the content they contain.

In other words, next to a color value for each pixel, I also need an array of descriptive terms, e.x (car, old, car-paint, red, scratch, … ). The pipeline is set up such that each 3d object has an array of terms store in an attribute, this array needs to be forwarded to the image upon rendering.

The terms does not need to be strings, I can make hash-tables and convert the words to integers.

What I need to know is if mantra support the rendering of an array-per-pixel? At best an array of arbitrary length.

And further, what image file-format could support this?

Either you need to encode this.
eg
Car = 00000001
Old = 00000010
Old and Car would be = 00000011
etc.
One bit per term

or you need to write your own output format. Even that would be possible. You could render into xml or whatever.

How many terms do you need?
Or how may terms can apply to one pixel?
How may combinations are there? Actually this would be the best approach.
First create a hash table with all combinations
But what about anti-aliasing? Is that important for you? Do you need to know if a pixel is 70% car 30% sky or so?
And maybe can i ask why you want to do this?

greetings

Olaf

It's for an artist, we're producing deep semantic images.

The number of terms is hard to judge, probably in the hundreds at least, and per pixel I could imagine four or five terms, however I'd like to avoid a fix limit to this.

Combinations are exponential, so a 1000 terms and 5 slots already means 1000^5. Though if doubles or longs are supported as output it could work.

Any clue on outputting custom file formats? OpenEXR supports arbitrary number of channels per pixel.

My current plan is to make n extra images planes, such that n = the size of longest array in the scene, and then per image plane render the i-th index of the arrays.

Gives me a bunch of images from which I can reconstruct the the arrays in post and store them in my own format.

From your description, it sounds like you're just asking to be able to add extra image planes, not to have the number of values vary per-pixel, so you wouldn't need to do anything complicated with deep-pixel EXRs.

You can add as many extra image planes as you'd like from the Mantra ROP under Images > Extra Image Planes. Click the plus to add some, set their VEX Variable names, their types, and make sure that your shaders output those variables. Is that it, or have I misinterpreted something?

@ndickson, thanks, it's a workaround.. though doesn't take me all the way.

To deal with anti-aliasing, reflection or transparency would need to accumulate values (as in adding to a list) rather than merging them (as with color-vectors).

For transparency a red car seen through a dirty window would be mean: {red, car} + {glass, window, dirt}. Say the car is reflective and surrounded by blush palm-trees. You might get ({red, car} + {blush, palm, tree, leaf, green}) + {glass, window, dirt} => {red,car,blush,palm,tree,leaf,green,glass,window,dirt}.

This is new grounds to me and I find hard to tell how long the array might end up. Which is also why I think the workaround doesn't seem so robust. Yet as of now, I have no idea how to preform such accumulation.

I want to know if it's possible to preform a ray-trace that accumulates the arrays of the objects it hits?
and how to save out such arrays? Can mantra render to say a .json file, or like Olaf mentioned, a xml file?

I suspect that although it may sound like a good idea, the results you'd get from including all secondary rays would probably not be what you'd like, because you'd basically get everything in every pixel, or at least very noisy results, especially if you have any very diffuse surfaces. You can technically do it, but I don't know if it'd be worth the effort to try. It'd also require more extra image planes, e.g. one for “car” and one for “glass”, even though they're mutually exclusive for a single ray, so if you were just taking a single ray's results, you could just have one plane where a 1 indicates “car” and a 2 indicates “glass”.

To reduce issues with anti-aliasing, you can set the Pixel Filter parameter for each extra image plane to “minmax idcover”, so that it'll just pick a single value, instead of trying to average them. If you do decide to have e.g. separate image planes for “car” and “glass”, where 1 indicates “yes” and 0 indicates “no”, you'll probably want to set the filter to “minmax max”, to indicate whether at least one ray in the pixel hit that type of surface, or “ubox”, to indicate what portion of rays in the pixel hit that type of surface.

Hi Smuseus,
he a link to a cracy project/tutorial:
http://tmdag.com/vopraytracer-pt1/ [tmdag.com]
It is a vex raytracer.

Now that you explained it in a bit more detail i would go about it by creating an image per term/objectgroup.
Transparencies and reflections will be the harder bit to deal with.
But you should include also depth(distance to camera) and transparency.
Now when testing a pixel you inspect all the available layers (one per term) from that image and get the array from that.
So one image per term makes the most sense, this keeps it flexible.

On top of that you create the beauty rendering.

greetings

Olaf

In case you're concerned about having a large number of extra image planes, people who want to adjust the lighting of complicated scenes when compositing often add over 100 extra image planes, sometimes one for each light and BSDF combination. It's a lot heavier-weight than just having a few image planes, but it's definitely doable.

After thinking about it, I was overly pessimistic about the prospect of supporting secondary rays, as long as you're looking for the proportion that each term contributes to the final pixels, since you'll be able to see that terms that only occurred in one secondary ray only contributed a little to the final pixel, since it'd be a value closer to 0. In that case, “ubox”, or even the default “gaussian -w 2” would work for a pixel filter to give an indication of how much each term contributed to each pixel, with anything greater than 0 indicating that it made some contribution.