Raytraced depth maps
Hi. this is a method i pretty much burnt my brain out trying to figure out. the method described here is to get a raytraced depth map for post-processing dof, which up til now is not possible. this is a post-processing method for use with post-processing filters, such as potoshop cs' lens blur.this tutorial is made using 3ds max 6, but it can literally be done in ANY 3d software that produces depth maps (sometimes called z-depth, z-buffer etc...) AND reflect refract passes. i've never used any other 3d program so i wouldn't know the process for those. you just need the greyscale z-map and reflect refract passes. Here is our image. don't tackle me with envy. it's a quicky to demonstrate this process.this tutorial is made using 3ds max 6, but it can literally be done in ANY 3d software that produces depth maps (sometimes called z-depth, z-buffer etc...) AND reflect refract passes. i've never used any other 3d program so i wouldn't know the process for those. you just need the greyscale z-map and reflect refract passes.
Above is our image. don't tackle me with envy. it's a quicky to demonstrate this process.
Step one is to prepare the renders. you need to take any object, that is in the reflection/refraction's path and assign each a unique color. not just any color but a blown out color. for example, make your diffuse a 100% blue color. now, add a gradient map to the self-illumination slot (max). both colors in the gradient should be the same exact color as the diffuse. i say "both" but some gradients have more then one color. anywho, the gradient is not used for it's gradiation effect but (in max) for the output function. therefore, you can use any map that allows you to pick the colors used AND has an output function. other 3d softwares must have something similiar, you just need to find out how. ok. now, turn up the output amount to like 6 or more. this ensures that your object will be easily distinguishable from any other object in the scene. repeat this for every object that's being reflected/refracted, using a different color for each object. do this also to all lights and atmospherics, too. they need not be blown out but just a unique color. a blown-out color would make it denser and that would not be desireable for this since we need it's visibility unaltered for this technique. If you have fog, do the same thing, but when you composite them, the reflection goes on top of the refraction. also, turn off your environment maps and make your background black. they are unnessesary and would actually make this harder. now, set the outputs for a depth map (z-depth, z-buffer etc...), and reflect, refract passes. the reflect/refract maps must be done in two seperate passes. one more thing, when you render the z-depth and reflect/refract passes, you MUST turn off anti-aliasing as depth maps are not meant to be anti-aliased. in max, i couldn't turn it off completely without disabling the reflect/refract passes so in order to bypass this, you set the anti-alias setting to it's lowest which gives pretty much the same result: no anti-aliasing. ok. render those and don't forget the actual render to which you will apply the dof to. Now that we've got all four of our maps (diffuse, z-depth, reflections, refractions), let's go to photoshop. Take all your maps and stack them on top of each other. the software-produced depth map should be on bottom.
Basically, in the reflection and refraction passes, we are trying to isolate anything other than black and assign it a depth level.
Now, let's start with our reflection pass. hide all other layers. select the reflection layer and do a color range selection (select menu > color range). (Above image).pick a color, any color so long as it's not black and using the fuzziness slider (maybe other graphics programs have a similar feature), select all of the color you chose, without selectiong any other color (including black). in this case, i chose blue to start with. (Below Image)
Now that you've selected all of the blue in the image, without going into any other color, make a layer for it and fill the selection with white. Name it blue and hide it.
Back to our reflection pass. repeat the previous step for any other object/color in the scene, naming them accordingly.
now, if you have anything other than an object that is in the reflection that is not black, do a color selection for it and fill it with white. these are usually light reflections or any other atmospheric in the scene. use the following steps to assign them a depth level.
Now for the refraction pass. it's pretty much the same thing. select all colors and, on their own layers, fill them with white. remember that reflections go on top of refractions.
Now let's start assigning depth values. starting with the floor. the floor, being so large, needs a gradiation since it has a vast depth. now use the eyedropper once on the depth map where the box and the floor visually "meet". and once on the depth map where the floor is farthest. (see below image) if you do not have a floor or any other reference object, simply approximate.
Now lock the layer's opacity and with the two eye-dropped colors, and the gradient tool, make the gradiation in the direction which is getting further with distance. use the depth map as a guide. the depth inside the refraction and outside shouldn't be too different. this applies to any object large enough that it needs a gradiation, including fog.
Now for the other objects, use the same technique. for example, if the object is visible both outside and inside the reflection/refraction, you can use the eye-dropper on the object in the software-produced depth map
If the object is not seen outside the reflection/refraction, you can use other objects which are relative to the object.
Now turn on all of the objects which were selected and colored in the previous steps and turn off all of the rendered layers except the software-produced depth map. our depth map is now ready.
Go to the top-most layer and select all (select menu > all) and copy merged (edit menu > copy merged). paste what you just copied into a blank alpha channel. now make your diffuse render active on the top-most layer and open the dof filter (filter > blur > lens blur in photoshop cs). in the filter parameters window, select your depth map. if there is only one alpha channel it will select it automatically. otherwise, select it from the drop-down. other filters make you select the depth before going into the filter so here's how. select the contents of the alpha channel by pressing the load channel as selection button on the bottom of the channels tab. enter the filter and bob's your uncle. from here on, you should know what to do. if you don't, refer to your dof filter's documentation.
this solution works but can be quite a hassle you're using it on an animation. so until our great software providers update their software, think... stills.