Before we start with the tutorial I want to make it clear that this tutorial uses both 3dmax and photoshop and its not designed for complete beginners at photoshop nor max since there are going to be things that I'm not going to explain such as the position of some tools, adjusting brush sizes, creating layers etc.. nor simple commands of max's user interface such as applying bitmaps nor other map types, exact position of map settings but if you get stuck in a certain area or don't know how to do a certain command don't give up quickly, make sure to take a quick look at the reference manual for either max or photoshop to answer your questions. Painting textures can be very time consuming but very rewarding and maybe you won't like how things are going in the beginning but it will start looking better as it progresses.
Even though this tutorial uses 3d max, I'm sure that people using other 3d packages would benefit from it since the tutorial is mostly dedicated to painting textures with photoshop, but I will also supply with a leaf model in other formats if you want to import it to your 3d package, note that the leaf model already has texture coordinates so you would simply have to take a screenshot of the uv map or bake the procedural color in order to follow along in the painting section.
A short summary of this tutorial would be that its goal is not to simply teach how to paint a leaf with photoshop but to teach texturing methods and techniques that could be applied to every texture you make. The goal is to learn more about texturing as a whole and learn the role that specific map types play on our 3d surfaces and how can we use those maps to make our 3d objects look and interact with virtual lights more realistically therefore our leaf object will include: A color map, bump map, specular map, diffusion map, dirt blend map, translucency map, a Reflection map, and a luminosity map. The map types will be explained more in depth in there specific section of the tutorial.
One of the most important keys when painting your own textures is to have lots of references, that way you can observe how the surface is composed. Break up the surface into multiple components such as color, bump, specularity, diffusion, dirt, luminosity or how much that surface self-illuminates, and translucency.
I think everyone already knows that when you make a color map what you try to capture is the surface's color but when painting textures always try to paint all the details in that object such as the faded areas in metal, rust, damage caused by the sun, even water drips and blood stains, scars etc.. we don't want to leave anything behind, you might think that a certain detail is not important or only makes a subtle difference and maybe not include it but they are important because these are the things that are going to separate your artwork from others and without those subtle differences your surface will look like its missing something to others and especially to yourself as a texture artist, I personally am not happy until I know that my surface acts and looks as its real life counterpart.
The things you try to capture for you bump map is the way that surface feels, you try to capture how that surface is distorted even with the smallest of details, like for example a real leaf my feel very smooth just by touching it but by looking closely you realize that that leaf is composed of very small bumps all throughout its surface and notice how damaged areas also have an impact on how the surface feels, you try to observe all of those details and include them in our bump map. Bump mapping should also be used to back up the information in our color map, like if you add drops of blood or water in the color map you would need to ad those details to the bump maps as well to make it appear as if the liquid had volume and if you add scratches or dents to the color map those should also be included in the bump map as well.
With specularity we try to capture the way that a given object reflects light, therefore its really important to have references of that object under different lighting conditions to get a better idea of how exactly that given object reflects and how the environment around it has affected its reflection. That given object may be wet, dry, dirty, clean, soft, hard, new, or old, all of these properties are important to include in your specular map. A specular map should be used to back up the information in the bump and color maps as well, like the scratches In the bump and color maps, normally light would get trapped more in a scratch or crack on the object's surface therefore that area would be less reflective therefore that should be included in your specular map, or if your adding drops of water or blood those areas would tend to reflect a light source more therefore that information should also be included in your specular map.
Here are the baked color maps, this contains the baked procedural color and the baked wire frame reference if you want to skip the texture baking sections and go straight to the texture painting sections, they
are jpg format with the resolution of 2048x2048 (you can resize of you desire). This was made with users with older versions of max in mind (can be used for other applications as well) since texture baking was implemented in version 5. I would recommend to delete the black background in the images using color range in photoshop in order to maintain alpha transparency in the layers (witch plays an important role in the tutorial).
Here's the 3dmax scene file with the unwrapped leaf model (max5).
Here's the obj, 3ds,xsi, and lwo format leaf model if you want to import and work with
it in another application.
Here are two fingerprint images that you can use for making custom fingerprint brushes that will come into play when making our specular map.
The tutorial is broken up into several parts:
Making the initial base color with procedural maps (3DS Max)
Baking Initial Base color (3DS Max)
Making the Color map (Photoshop)
Making the Bump map (Photoshop)
Making the Specular map (Photoshop)
Making the Diffusion map (Photoshop)
Making the Dirt Blend map (Photoshop)
Making the Reflection Level map (Photoshop)
Making the Translucency Level map (Photoshop)
Making the Luminosity Map (Photoshop)
Making seam layers (Photoshop)
Applying our maps (3DS Max)
Faking blurry refractions using reflections (3DS Max)
Faking Translucency (3DS Max)
Creating the dirt shader (3DS Max)
Rendering tips (3DS Max and Photoshop)
Final Renders and additional information
Make sure to visit the author's website to download this tutorial in pdf-format!
Making the colour map
Even though the instruction for making the procedural map and baking the map and use it for painting are max specific, I'm sure you can apply these techniques to other packages because of the nature of procedurals and simply because most of the surface editors in other packages have the noise and cloud/smoke procedural maps already included in them.
You may be asking yourself why are we going to make a procedural map instead of just taking a screen shot of the unwrapping editor? Simply because making a procedural map and then baking them it is a good way to insure that the edges of the object will blend together seamlessly and a procedural map could also provide us with a good initial base color that we could use as reference when painting details in photoshop.
To start off open the scene file included in the leaf zip file, you should see the fully unwrapped leaf model ready to be painted, but instead of taking a screenshot of the uv unwrap modifier were going to create a simple procedural texture and then bake it, as I mentioned before this way we ensure we start off with a seamless texture and some colors we can use as reference for painting as well.
Open the material editor and ad a smoke map to the diffuse channel of a new material, change the size to 10, give color 1 an rgb value of R:101 G:54 B:16 and give color 2 a value of R:81 Green:36 Blue:0, this will produce a brown tone witch is part of the discolored areas in our leaf.
Add the current smoke map to a noise by clicking on the smoke button and picking noise from the material/map browser (keep old map as sub-map), change the type to fractal, in the noise threshold give the high parameter a value of 0.6 and a low value of 0.25, and add another smoke map to the empty color 2 slot.
This time instead of browns were going to use yellow tones to add more variation to the faded areas in the leaf, change the size to 10 give the color 1 an rgb value of R:167 G:128 B:22 and color 2 a value of R:196 G:161 B:22.
Click the go to parent button once to go to the noise base level and add the noise map to another noise map by clicking at the noise button and picking noise from the material/map browser and keep old map as sub-map in order to create the third color layer, in the new noise map change the type to fractal, size to 30, noise threshold high value of 0.6 and low value of 0.25, phase to 10 just to give it a different pattern, and click the swap button in order for our old noise map to move to the second color slot.
Add another smoke map to the empty color channel (first color slot), change the size for that smoke map to 10, give color 1 a value of R:44 G:60 B:0 and color 2 a value of R:67 G:84 B:0
We're finished with this step, for now make a quick render of your scene, if you're not entirely happy with the pattern or color feel free to adjust the maps to your liking before moving to the next steps.
Baking the initial base colour
Now we're ready to bake the current texture and paint more details to it in photoshop, if your having trouble with baking the texture make sure to take a quick look at the reference files for 3d max, open the render to texture feature in the rendering menu, add the diffuse map, change the channel to channel 1 because that's were all of the unwrapping information is in but the size on the image depends on you, think about how much space will the leaf take of the screen when you make your final render, if its going to take about an area of 512x512 in your final render than you can make the size of the texture 1024x1024 to make sure none of the areas look blurry and to maintain a rather high level of detail in the textures, if your render is going to be much higher resolution and the leaf might use 1024x1024 pixels in the image than you might want to go as high as 2048x2048 but before you click render make sure your output file is a tga because it will have a transparent background we wont have to worry about separating the leaf with selection tools, I would recommend using 2048x2048 because that's the resolution I'm going to be using and in the painting sections I might give brush specifications that are the accurate pixel size for my canvas but might be smaller if you use a higher resolution image or the brush might be too big if you use a lower resolution image.
Before we start baking we need to make the material self illuminated otherwise you will see that some parts of the leaf may have highlights and other portions will be in shadow and that's not what we want for our baked texture, a quick way to do that is to add an output map the self illumination channel. Right now if you bake the texture you will see that the vain of the leaf does not show up, a quick way to go over this is to bake another texture but this time with wireframe checked in the material basic parameters in the material editor, baking a wireframe will also help remind us the exact position of certain details in the leaf when painting in photoshop, now disable mesh smooth in the leaf's modify panel and start baking the textures. Here's a screen of the baked color map and the baked wireframe map that we will use as reference in future sections.
Making the colour map
When you open the baked texture file in photoshop the first thing you might notice is that the top part of the leaf (the one on the right) and the bottom part (one on the left) have the exact same pattern that's because of the nature of procedural textures but to fix this use the heal tool to change the inner pattern on one of the leaves but make sure you don't change outer edges of the leaf because we still want the texture to remain seamless. In this case I find it better to use bigger brushes with their hardness parameter set to 0, that way the healing changes will be more subtle, after you finished healing one side of the leaf rename the layer to "base color".
Now we're going to begin painting some of the details into it but first open the wireframe map that we baked, copy and paste it to our image, position so its right on top of our base color layer and rename the wireframe map to "Wire Reference" and change the blending mode for it too multiply to be able to see it on top of our base color texture, now create a new layer on top of wire reference and rename it to "Leaf Pattern"
To make the leaf pattern you could make a new document and make it so it's about one fifth the size of your canvas, what we're going to do is make a custom brush for the pattern shape. Use the image on the left as a reference for the shape of the costume brush, to make the custom brush just select the entire canvas, then go to edit > define brush.
In the LeafPattern layer, use your new costume brush to paint the patterns on the left side of one of the leaves, then you can simple make a duplicate of the layer and flip it horizontally (image>adjust>flip horizontal) and position it on the right side of the leaf and adjust some of the strokes by lasso selecting them and moving so they come in contact with the edge center vain or you could make another custom brush with a pattern that is flipped horizontally and paint the patterns on the right side, after you finished making the patterns for one of the leaf sides, duplicate the layer and move it to the other side and make regular brush strokes to compensate for the differences in a few areas. Here's a screen of the leaf patterns with the wire reference layer hidden. After you finished positioning the patterns merge all the leaf pattern duplicates into a single layer.
Make a duplicate of the pattern layer and rename it to "pattern soft shadow" and apply a Gaussian blur of about 5, now change the blending mode to soft light, and move it so its a few pixels below the LeafPattern layer.
Change the blending mode of the leafpattern layer to "darken" and change the opacity value of the layer to 25 percent just as a temp color, were going to come back on this and make some changes to this layer later in the tutorial.
Duplicate the "pattern soft shadow" layer and rename to "pattern soft highlight", adjust the hue saturation for that layer, we're going to give this layer a dark green color tone, check colorize and give hue a value of about 115, saturation of 100 and lightness of 10. change the blending mode to hard light and to make the colors more apparent make a duplicate of the soft highlight layer and merge it down (ctrl+e).
Hide the leaf pattern layer and the soft shadow layer, make sure your wire reference is not hidden, now sample the dark green color in the soft highlight layer and paint in the outer edges of the center vain of the leaf, you might want to give your brush a little bit of scattering (in brushes tab) in order to make the brush strokes look a little more random than uniform and change the flow and opacity of the brush to about 50 percent. Use the following image as a reference for painting the dark green areas.
Create a new layer above the pattern soft highlight layer and name it pattern detail, for this were going to make a custom brush to add some detail to the pattern, with this brush it should look like the patterns have this root/lightning pattern on
them. Open the brushes tab in order to customize this brush, enable scattering and give the scatter amount a value of about 95, enable texture and give the scale a value of about 95 and a depth value of 90 in order to give the brush more detail, now pick the texture called "metal landscape" that has the exact effect were looking for, increase the brush size accordingly, also change the opacity value of the brush to about 75 and flow of about 60. Your brush strokes should look something similar to this:
Make sure you increase the size of the brush and the scale of the texture as you get closer to the center of the leaf and decrease it as you get closer to the bottom. To soften and add additional detail to the strokes after you finished going trough all the lines in the leaf pattern, increase the size of the brush and the scale of the texture even more and decrease the brush opacity to 30 percent and give the patterns another pass and that should complete the leaf patterns, now just change the blending mode for the pattern detail layer to overlay and opacity to 75 percent. Use this image as reference for painting the detail strokes.
We're going to go back to the base color layer and add more detail to it, we want the color to be darker in a few areas and lighter in others, we also want to make it look a little sharper since the procedural texture gave us good color tones to work with but not as much detail nor sharpness. We are going to use the dodge and burn tools to make areas darker and lighter and the sponge tool to remove and add color from a few areas as well. Pick the burn tool and change the brush size to about 50, an exposure value of 15, and in the brushes tan enable scattering and give it a value of about 260, enable dual brush and pick number "60" that brush will give us a good level of detail to work with. Start painting, its up to you too decide witch areas of the leaf you want to be dark or lighter, for now just paint the dark areas with the burn tool and when your happy with it make other areas lighter with the dodge tool, make sure the dodge has the same brush size and style parameters as the burn tool's current brush. Try not to paint on the edges.
For the sp