What is a Normal Map?

A Normal Map is usually used to fake high-res geometry detail on what is actually a low-res mesh. Each pixel of a normal map is used to transfer the normal that's on the high-res mesh onto the surface of the low-res mesh. The red, green, and blue channels of the texture are used to control the direction of each pixel's normal. The pixels in the normal map basically control what direction each of the pixels on the low-poly model will be facing, controlling how much lighting each pixel will receive, and thus creating the illusion of more surface detail or better curvature. The process of transferring normals from the high-res model to the in-game model is often called baking.

Rick Stirling wrote: "When explaining what a normal map actually was to someone, I explained the difference between a pixel and a polygon, and how UVmaps worked. Once this was understood I said that basically on a normal map each pixel was pretending to be a polygon."

Per Abrahamsen wrote: "What I say along the same lines is first teach them about vertex normals, and when they get that.. I say, well now imagine you have such normal at each texel rather than at each vertex."

A model with a normal map. Actual size
image byJames Ku

The low-resolution wireframe. Actual size
image byJames Ku

The high-resolution model used to create the normal map. Actual size
image byJames Ku

Tangent-Space vs. Object-Space

Normal maps can be made in either of two basic flavors: tangent-space or object-space. Object-space is also called local-space or model-space, same thing. World-space is basically the same as object-space, except it requires the model to remain in its original orientation, neither rotating nor deforming, thus world-space is rarely (if ever) used.

A tangent-space normal map.
Actual size
image byEric Chadwick

An object-space normal map.
Actual size
image byEric Chadwick

Tangent-space normal map

Object-space normal map

RGB Channels

Shaders can use different techniques to render tangent-space normal maps, but the normal map directions are usually consistent within a game. Usually the red channel of a tangent-space normal map stores the X axis (pointing the normals predominantly leftwards or rightwards), the green channel stores the Y axis (pointing the normals predominantly upwards or downwards), and the blue channel stores the Z axis (pointing the normals outwards away from the surface).

The red, green, and blue channels of a tangent-space normal map.
Actual size
image byEric Chadwick

If you see lighting coming from the wrong angle when you're looking at your normal-mapped model, and the model is using tangent-space normal maps, the normal map shader might be expecting the red or green channel (or both) to point in the opposite direction. To fix this either change the shader, or simply invert the appropriate color channels in an image editor, so that the black pixels become white and the white pixels become black.

Some shaders expect the color channels to be swapped or re-arranged to work with a particular compression format. For example the DXT5_nm format usually expects the X axis to be in the alpha channel, the Y axis to be in the green channel, and the red and blue channels to be empty.

Tangent Basis

Tangent-space normal mapping doesn't just use a map, it also uses a special kind of vertex data called the tangent basis. This is similar to UV coordinates except it provides directionality across the surface. Three vectors are created for each vertex: normal, tangent, and bitangent (aka binormal). These three vectors create an axis for each vertex, giving it a specific orientation. These axes are used to properly transform the incoming lighting from world space into tangent space, so your normal-mapped model will be lit correctly.

For example, when you look at a tangent-space normal map for a character, you typically see different colors along the UV seams. This is because the UV shells are often oriented at different angles on the mesh, a necessary evil when translating the 3D mesh into 2D textures. The body might be mapped with a vertical shell, and the arm mapped with a horizontal one. This requires the normals in the normal map to be twisted for the different orientations of those UV shells. The UVs are twisted, so the normals must be twisted in order to compensate. The tangent basis helps reorient (twist) the lighting as it comes into the surface's local space, so the lighting will then look uniform across the normal mapped mesh.

When shared edges are at different angles in UV space, different colors will show up
along the seam. The tangent basis uses these colors to light the model properly.
Actual size
image byEric Chadwick

When the renderer (or game engine) renders your game model, the shader must use the same tangent basis as the normal map baker, otherwise you'll get incorrect lighting. In fact you'll get seams all over the place without it. For the best lighting the shader should be written to extract the tangent basis stored in the mesh, and the mesh importer must either import the tangent basis created by the baker, or use the same method to recreate the tangent basis itself. If the shader doesn't use the same tangent basis as the baker, the lighting might be correct in some places but incorrect in others.

There are a few different ways programmers can use to calculate the tangent basis: DirectX, NVIDIA mesh mender, a custom solution, etc. The baking app xNormal supports custom tangent basis generators to make sure engine and baker match.

The tangent basis is calculated using the UV layout and the smoothing groups (hard edges), because typically each vertex in the tangent basis is a combination of three things: the vertex's normal (influenced by smoothing), the vertex's tangent (usually derived from the V texture coordinate), and the vertex's bitangent (derived by coders using that lovely thing called math). This means the UVs and the normals on the low-res mesh directly influence the coloring of a tangent-space normal map when it is baked. For this reason you should avoid changing the UVs without re-baking the map, because then the map probably won't match the tangent basis anymore, and you'll see lighting problems.

UV Coordinates

If you want to mirror the UVs, or you want to reuse parts of the normal map by overlaying multiple bits in the same UV space, then simply move all those overlapped/mirrored bits one unit away before you capture the normal map. Only one copy of the forward-facing UVs should remain in the 0-1 UV box at baking time.

The mirrored UVs (in red) are offset 1 unit before baking.
image byEric Chadwick

Normal map-baking tools will only capture normals within the 0-1 UV box, any UV bits outside this area are ignored. If you move all the overlaps exactly 1 UV unit away then you can leave them there after the bake and they will be mapped correctly. You can move them back if you want, it doesn't matter to most game engines. Be aware that ZBrush does use UV offsets to manage mesh visibility, however this usually doesn't matter because the ZBrush cage mesh is often a different mesh than the in-game mesh used for baking.

Many games have a difficult time solving the seam when a normal map is mirrored down the center of the mesh. This can be avoided by offsetting the mirror point. Ben Regimbal aka b1ll has some examples of offset-mirroring on his site.

If after baking you change the UV layout you should then re-capture the normal map, because rotating or mirroring UVs after baking may cause the normal map not to match the tangent basis anymore, which will likely cause lighting problems. Best advice is to experiment with your engine to see what works in your particular case. If your engine still shows correct lighting when you do rotate and/or mirror, you might want to use [Will Fuller's normal map actions] to rotate & flip areas of the normal map right inside Photoshop, without have to re-bake.

Modeling The High Poly Mesh

Subdivision Surface Modeling is the technique used most often for normal map modeling. The subdivision cage or basemesh is typically not the same as the in-game mesh used for baking. The in-game mesh usually needs to be carefully optimized to create a good silhouette, define edge-loops for better deformation, and minimize extreme changes between the vertex normals for better shading (see Smoothing Groups).

Some artists prefer to model the in-game mesh first, some prefer to model the high-res mesh first, and some start somewhere in the middle. The modeling order is ultimately a personal choice though, all three methods can produce excellent results.

If the in-game mesh is started from one of the subdivision levels of the basemesh sculpt, various edge loops can be collapsed or new edges can be cut to add/remove detail as necessary.

If you are using a sculpting app then you will want to create a specific cage or basemesh that is as close as possible to 100% quads. Try to avoid "poles" of 5+ edges running into each other since that can lead to "pinching" in the model when sculpting. Keep the quads a fairly uniform density over the model, and be clever about adding more detail where it will be necessary (eyes, ears, nose, etc.), since otherwise you will end up subdividing your mesh far more than necessary just to get enough polygons to sculpt detail into these areas.

It helps to add a quick UV to the basemesh before sculpting, since this can be used to bake ZBrush polypainting if you choose to do this at some point. This is much easier than trying to UV your basemesh later and trying to morph it or merge it with your sculpt.

In order to create an optimized in-game mesh including a good silhouette and loops for deforming in animation, you can start with the 2nd subdivision level of your sculpt, or in some cases with the base mesh itself. Then you can just collapse edge loops or cut in new edges to add/remove detail as necessary. Or you can re-toplogize from scratch if that works better for you.

JPG tutorial: Modeling High/Low Poly Models for Next Gen Games by João "Masakari" Costa

Smoothing Groups & Hard Edges

It is generally better not to use smoothing groups (hard edges) to add definition to the low-poly game mesh. Smoothing groups cause the vertex normals to be split along the hard edge, which both increases the number of vertices in-game, and can cause normal map baking errors because the split normals each "see" a different part of the high-res mesh during the raycasting process.

Smoothing groups split the vertex normals, causing ray misses (red area).
image byEric Chadwick

Bevels interpolate across the vertex normals, minimizing raycast errors.
image byEric Chadwick

In most cases a single smoothing group should be applied to the entire in-game mesh before baking. However this can produce extreme shading differences across the model, as the lighting is interpolated across the extreme differences between the vertex normals. It is usually better to reduce these extremes when you can because the tangent basis can only do so much to counteract the extreme lighting variations. Less extreme gradients are also better if your game engine doesn't use the same tangent basis as the baker (or doesn't make its own properly).

Hard edges can be added (via smoothing groups) where the model already has a natural seam. For example, you can add a hard edge along the rim of a car's wheel well, to prevent the inside of the wheel well from distorting the shading of the area on the outside of the car body.

Bevels can be used instead of smoothing groups, but they should be used with care because they double the vertex count along that edge. Smoothing groups also can double the vertex count for an edge, unless that edge is also a seam in UV space, or it's a seam between materials. For a good explanation of this, read Beautiful, Yet Friendly.

When you use object-space normal maps the vertex normal problem goes away since you're no longer relying on the crude vertex normals of the mesh. An object-space normal map completely ignores vertex normals. Object-space mapping allows you to use a single smoothing group and no bevels on the low-res mesh, without showing lighting errors.

Baking

When you use normal map baking software to create your normal map, it grabs the normals from your high-poly mesh and puts them into a normal map for the low-poly mesh. The baker usually starts projecting a certain numerical distance out from the low-poly mesh, and sends rays inwards towards the high-poly mesh. When a ray intersects the high-poly mesh, it records the mesh's surface normal into your normal map.

An in-game mesh with split normals causes ray misses (yellow) and ray overlaps (cyan).
image byDiego Castaño

An in-game mesh using a single smoothing group minimizes ray-casting errors.
image byDiego Castaño

To get an understanding of how all the options affect your normal map, do some test bakes on simple meshes like boxes. They generate quickly so you can experiment with UV mirroring, smoothing groups, etc. This helps you learn the settings that really matter.

Working with Cages

Cage has two meanings in the normal-mapping process: a low-poly base for subdivision surface modeling, or a ray-casting mesh used for normal map baking. In this section we'll talk about the latter.

Instead of using a numerical distance to start ray-casting from, some software allows you to use a ballooned-out copy of the low-poly mesh to control that starting distance. This ballooned-out mesh is the cage.

In 3ds Max the cage controls both the distance and the direction of the raycasting. In Maya the cage only controls the distance; the ray direction matches the vertex normals (inverted).

Solving Intersections

The projection process often causes problems like misses, or overlaps, or intersections. It can be difficult generating a clean normal map in areas where the high-poly mesh intersects or nearly intersects itself, like in between the fingers of a hand. Setting the ray distance too large will make the baker pick the other finger as the source normal, while setting the ray distance too small will lead to problems at other places on the mesh where the distances between in-game mesh and high-poly mesh are greater.

Fortunately there are several methods for solving these problems.

  1. Change the shape of the cage. Manually edit points on the projection cage to help solve tight bits like the gaps between fingers.
  2. Limit the projection to matching materials, or matching UVs.
  3. Explode the meshes.
  4. Bake two or more times using different cage sizes, and combine them in Photoshop.

Solving Wavy Lines

When capturing from a cylindrical shape, often the differences between the low-poly mesh and the high-poly mesh will create a wavy edge in the normal map. There are a couple ways to avoid this:

  1. Adjust the shape of the cage to influence the directions the rays will be cast. At the bottom of this page of his normal map tutorial, Ben Mathis aka poopinmymouth shows how to do this in 3ds Max. Same method can be seen in the image below.

  2. Subdivide the low-res mesh so it more closely matches the high-res mesh. Jeff Ross aka airbrush has a video tutorial that shows how to do this in Maya.

  3. Paint out the wavy line. The normal map process tutorial by Ben Mathis aka poopinmymouth includes an example of painting out wavy lines in a baked normal map.

  4. Use a separate planar-projected mesh for the details that wrap around the barrel area, so the ray-casting is more even. For example to add tread around a tire, the tread can be baked from a tread model that is laid out flat, then that bake can layered onto the bake from the cylindrical tire mesh in a paint program.
  5. The polycount thread "approach to techy stuff" has some good tips for normal-mapping cylindrical shapes.

Adjust the shape of the cage to remove distortion.
image byTimothy Evison aka tpe

Solving Pixel Artifacts

Random pixel artifacts in the bake.
image byEric Chadwick

If you are using 3ds Max's Render To Texture to bake from one UV layout to another, you may see stray pixels scattered across the bake. This only happens if you are using a copy of the original mesh in the Projection, and that mesh is using a different UV channel than the original mesh.

There are two solutions for this:

- or -

Baking Transparency

Sometimes you need to bake a normal map from an object that uses opacity maps, like a branch with opacity-mapped leaves. Unfortunately baking apps often completely ignore any transparency mapping on your high-poly mesh.

3ds Max's RTT baker causes transparency errors.
Actual size
image byJoe Wilson aka EarthQuake

The lighting method bakes perfect transparency.
Actual size
image byJoe Wilson aka EarthQuake

To solve this, render a Top view of the mesh. This only works if you're using a planar UV projection for your low-poly mesh and you're baking a tangent-space normal map.

The lighting setup for top-down rendering.
image byBen Cloward

Anti-Aliasing

Turning on super-sampling or anti-aliasing (or whatever multi-ray casting is called in your normal map baking tool) will help to fix any jagged edges where the high-res model overlaps itself within the UV borders of the low-poly mesh, or wherever the background shows through holes in the mesh. Unfortunately this tends to render much much slower, and takes more memory.

One trick to speed this up is to render 2x the intended image size then scale the normal map down 1/2 in a paint program like Photoshop. The reduction's pixel resampling will add anti-aliasing for you in a very quick process. After scaling, make sure to re-normalize the map if your game doesn't do that already, because the un-normalized pixels in your normalmap may cause pixelly artifacts in your specular highlights. Re-normalizing can be done with NVIDIA's normal map filter for Photoshop.

3ds Max's supersampling doesn't work nicely with edge padding, it produces dark streaks in the padded pixels. If so then turn off padding and re-do the padding later, either by re-baking without supersampling or by using a Photoshop filter like the one that comes with Xnormal.

Edge Padding

Need info here about padding.

Painting

Don't be afraid to edit normal maps in Photoshop. After all it is just a texture, so you can clone, blur, copy, blend all you want... as long as it looks good of course. Some understanding of the way colors work in normal maps will go a long way in helping you paint effectively.

A normal map sampled from a high-poly mesh will nearly always be better than one sampled from a texture, since you're actually grabbing "proper" normals from an accurate, highly detailed surface. That means your normal map's pixels will basically be recreating the surface angles of your high-poly mesh, resulting in a very believable look.

If you only convert an image into a normal-map, it can look very flat, and in some cases it can be completely wrong unless you're very careful about your value ranges. Most image conversion tools assume the input is a hightmap, where black is low and white is high. If you try to convert a diffuse texture that you've painted, the results are often very poor. Often the best results are obtained by baking the large and mid-level details from a high-poly mesh, and then combined with photo-sourced "fine detail" normals for surface details such as fabric weave, scratches and grain.

However... sometimes creating a high poly surface takes more time than your budget allows. For character or significant environment assets then that is the best route, but for less significant environment surfaces working from a heightmap-based texture will provide a good enough result for a much less commitment in time.

Some tutorials and tools for painting normal maps or creating them from 2D sources (paintings, photos, displacement maps):

Re-normalizing

Re-normalizing means resetting the length of each normal in the map to 1.

A normal map shader combines the three color channels of a normal map to create the direction and length of each pixel's normal. This information is necessary to apply the lighting. When you blend multiple normal maps together or edit them by hand this can cause the lengths to change. Some shaders are written to re-normalize the normal map, most shaders are not, they expect the length of the normals to be 1.

If the lengths of the normals are not normalized to 1, and the shader doesn't re-normalize, you may see artifacts on the shaded surface... the specular highlight may speckle like crazy, the surface may get patches of odd shadowing, etc.

NVIDIA's normal map filter for Photoshop provides an easy way to re-normalize a map after editing, just use the Normalize Only option.

The re-normalize option in the NVIDIA filter.
Actual size
image byScott Warren

An ambient occlusion pass can be multiplied onto the blue channel to store ambient occlusion in a tangent-space normal map, because it is shortening the normals in the crevices of the surface. However, the shader must be altered to actually use the lengths of your custom normals; most shaders just assume all normals are 1 in length.

Some shaders use compressed normal maps. Usually this means the blue channel is thrown away completely, and it's recalculated in the shader. The shader has to re-normalize in order to recreate that data, so any custom normal lengths that were edited into the map will probably be ignored completely.

Backlighting Example

If the shader doesn't re-normalize the normal map, you can customize the normal map for some interesting effects. If you invert the blue channel of a tangent-space map, the normals will be pointing to the opposite side of the surface, which can simulate backlighting.

Tree simulating subsurface scattering (front view).
Actual size
image byEric Chadwick

Tree simulating subsurface scattering (back view).
Actual size
image byEric Chadwick

The maps used for the leaves. The 2nd diffuse was simply color-inverted, hue-shifted 180°, and saturated.
image byEric Chadwick

The tree leaves use a shader than adds together two diffuse maps, one using a regular tangent-space normal map, the other using the same normal map but with the blue channel inverted. This causes the diffuse map using the regular normal map to only get lit on the side facing the light (front view), while the diffuse map using the inverted normal map only gets lit on the opposite side of the leaves (back view). The leaf geometry is 2-sided but uses the same shader on both sides, so the effect works no matter the lighting angle. As an added bonus, because the tree is self-shadowing the leaves in shadow do not receive direct lighting, which means their backsides do not show the inverted normal map, so the fake subsurface scatter effect only appears where the light directly hits the leaves. This wouldn't work for a whole forest, because of the computational cost of self-shadowing and double normal maps, but could be useful for a single "star" asset.

Shaders and Seams

You need to use the right kind of shader to avoid seeing seams. The shader must be written to extract the tangent basis stored in the mesh, and the mesh importer must either import the tangent basis created by the baker or use the same method to recreate the tangent basis itself. If the shader doesn't use the same tangent basis as the baker, the lighting will be inconsistent across the UV borders.

3ds Max Shaders

These shaders solve the seams properly:

Load the .FX files using the DirectX Shader material. Make sure to setup your lights in the shader, or it may just stay black. Often you also need to specify a diffuse bitmap too, you can't just leave that slot blank.

These shaders do not:

Maya Shaders

Normal Map Compression

Normal maps can take up a lot of memory. Compression can reduce the size of a map to 1/4 of what it was uncompressed, which means you can either increase the resolution or you can use more maps.

Usually the compression method is to throw away the Blue channel, because this can be re-computing at minimal cost in the shader code. Then the bitmap only has to store two color channels, instead of four (red, green, blue, and alpha).

DXT5nm Compression

DXT5nm is the same file format as DXT5 except before compression the red channel is moved into the alpha channel, the green channel is left as-is, and the red and blue channels are blanked with the same solid color. This re-arranging of the color channels is often called swizzling.

The Green and Alpha channels are used because in the DXT format they are compressed using somewhat higher bit depths than the Red and Blue channels. Red and Blue have to be filled with the same solid color because DXT uses a compression system that compares differences between the three color channels. If you try to store some kind of texture in Red and/or Blue (specular power, height map, etc.) then the compressor will create more compression artifacts because it has to compare all three channels.

There are some options in the NVIDIA DXT compressor that help reduce the artifacts if you want to add texture to the Red or Blue channels. The artifacts will be greater than if you keep Red and Blue empty, but it might be a tradeoff worth making. Some notes about this on the NVIDIA Developer Forums.

DXT1 Compression

DXT1 is also used sometimes for tangent-space normal maps, because it is half the size of a DXT5. The downside is it causes many compression artifacts, so much so that most people end up not using it. The upside though is that DXT stays compressed in video memory, meaning you can use more textures or larger textures within the same amount of RAM.

3DC Compression

3Dc compression works similar to DXT5nm, with some important differences. It yields the best results of any listed algorithm for tangent space normal map compression, and requires no extra processing time or unique hardware. Details can be found in the3Dc whitepaperfrom ATI.

A8L8 Compression

The DDS format A8L8 isn't actually compressed, it's just two 8bit grayscale channels (256 grays each). It does save you from having to store all three color channels. Your shader has to recompute the blue channel for it to work. However, A8L8 does not actually save any space in texture memory, it is typically converted to a four-channel 32bit texture when it's sent to the card. This format really only helps save disk space.

3D Tools

2D Tools

Tutorials

Discussion

Discuss this page on the Polycount forums. Suggestions welcome.


Normal Map (last edited 2008-10-19 20:50:21 by EricChadwick)