Cg Programming/Unity/Lighting of Bumpy Surfaces

This tutorial covers normal mapping.

It's the first in a series of tutorials about texturing techniques that go beyond two-dimensional surfaces (or layers of surfaces). Note that these tutorials are meant to teach you how these techniques work. If you want to actually use one of them in Unity, you should rather use a built-in shader or a Surface Shader.

In this tutorial, we start with normal mapping, which is a very well established technique to fake the lighting of small bumps and dents — even on coarse polygon meshes. The code of this tutorial is based on Section “Smooth Specular Highlights” and Section “Textured Spheres”.

Perceiving Shapes Based on Lighting

The painting by Caravaggio depicted to the left is about the incredulity of Saint Thomas, who did not believe in Christ's resurrection until he put his finger in Christ's side. The furrowed brows of the apostles not only symbolize this incredulity but clearly convey it by means of a common facial expression. However, why do we know that their foreheads are actually furrowed instead of being painted with some light and dark lines? After all, this is just a flat painting. In fact, viewers intuitively make the assumption that these are furrowed instead of painted brows — even though the painting itself allows for both interpretations. The lesson is: bumps on smooth surfaces can often be convincingly conveyed by the lighting alone without any other cues (shadows, occlusions, parallax effects, stereo, etc.).

Normal Mapping

Normal mapping tries to convey bumps on smooth surfaces (i.e. coarse triangle meshes with interpolated normals) by changing the surface normal vectors according to some virtual bumps. When the lighting is computed with these modified normal vectors, viewers will often perceive the virtual bumps — even though a perfectly flat triangle has been rendered. The illusion can certainly break down (in particular at silhouettes) but in many cases it is very convincing.

More specifically, the normal vectors that represent the virtual bumps are first encoded in a texture image (i.e. a normal map). A fragment shader then looks up these vectors in the texture image and computes the lighting based on them. That's about it. The problem, of course, is the encoding of the normal vectors in a texture image. There are different possibilities and the fragment shader has to be adapted to the specific encoding that was used to generate the normal map.

A typical example for the appearance of an encoded normal map.

Normal Mapping in Unity

The very good news is that you can easily create normal maps from gray-scale images with Unity: create a gray-scale image in your favorite paint program and use a specific gray for the regular height of the surface, lighter grays for bumps, and darker grays for dents. Make sure that the transitions between different grays are smooth, e.g. by blurring the image. When you import the image with Assets > Import New Asset change the Texture Type in the Inspector Window to Normal map and check Create from Grayscale. After clicking Apply, the preview should show a bluish image with reddish and greenish edges. Alternatively to generating a normal map, the encoded normal map to the left can be imported. (In this case, don't forget to uncheck the Create from Grayscale box).

The not so good news is that the fragment shader has to do some computations to decode the normals. First of all, the texture color is stored in a two-component texture image, i.e. there is only an alpha component $A$ and one color component available. The color component can be accessed as the red, green, or blue component — in all cases the same value is returned. Here, we use the green component $G$ since Unity also uses it. The two components, $G$ and $A$ , are stored as numbers between 0 and 1; however, they represent coordinates $n_{x}$ and $n_{y}$ between -1 and 1. The mapping is:

$n_{x}=2A-1$ and $n_{y}=2G-1$

From these two components, the third component $n_{z}$ of the three-dimensional normal vector n $=(n_{x},n_{y},n_{z})$ can be calculated because of the normalization to unit length:

${\sqrt {n_{x}^{2}+n_{y}^{2}+n_{z}^{2}}}=1\quad \Rightarrow$ $n_{z}=\pm {\sqrt {1-n_{x}^{2}-n_{y}^{2}}}$

Only the “+” solution is necessary if we choose the $z$ axis along the axis of the smooth normal vector (interpolated from the normal vectors that were set in the vertex shader) since we aren't able to render surfaces with an inwards pointing normal vector anyways. The code snippet from the fragment shader could look like this:

            float4 encodedNormal = tex2D(_BumpMap, 
               _BumpMap_ST.xy * input.tex.xy + _BumpMap_ST.zw);
            float3 localCoords = float3(2.0 * encodedNormal.a - 1.0, 
                2.0 * encodedNormal.g - 1.0, 0.0);
            localCoords.z = sqrt(1.0 - dot(localCoords, localCoords));
               // approximation without sqrt:  localCoords.z = 
               // 1.0 - 0.5 * dot(localCoords, localCoords);

The decoding for devices that use OpenGL ES is actually simpler since Unity doesn't use a two-component texture in this case. Thus, for mobile platforms the decoding becomes:

            float4 encodedNormal = tex2D(_BumpMap, 
               _BumpMap_ST.xy * input.tex.xy + _BumpMap_ST.zw);
            float3 localCoords = 
               2.0 * encodedNormal.rgb - float3(1.0, 1.0, 1.0);

However, the rest of this tutorial (and also Section “Projection of Bumpy Surfaces”) will cover only desktop platforms.

Tangent plane to a point on a sphere.

Unity uses a local surface coordinate systems for each point of the surface to specify normal vectors in the normal map. The $z$ axis of this local coordinates system is given by the smooth, interpolated normal vector N in world space and the $x-y$ plane is a tangent plane to the surface as illustrated in the image to the left. Specifically, the $x$ axis is specified by the tangent parameter T that Unity provides to vertices (see the discussion of vertex input parameters in Section “Debugging of Shaders”). Given the $x$ and $z$ axis, the $y$ axis can be computed by a cross product in the vertex shader, e.g. B = N × T. (The letter B refers to the traditional name “binormal” for this vector.)

Note that the normal vector N is transformed with the transpose of the inverse model matrix from object space to world space (because it is orthogonal to a surface; see Section “Applying Matrix Transformations”) while the tangent vector T specifies a direction between points on a surface and is therefore transformed with the model matrix. The binormal vector B represents a third class of vectors which are transformed differently. (If you really want to know: the skew-symmetric matrix corresponding to “B×” is transformed like a ternary quadratic form.) Thus, the best choice is to first transform N and T to world space, and then to compute B in world space using the cross product of the transformed vectors.

These computations are performed by the vertex shader, for example this way:

         struct vertexInput {
            float4 vertex : POSITION;
            float4 texcoord : TEXCOORD0;
            float3 normal : NORMAL;
            float4 tangent : TANGENT;
         };
         struct vertexOutput {
            float4 pos : SV_POSITION;
            float4 posWorld : TEXCOORD0;
               // position of the vertex (and fragment) in world space 
            float4 tex : TEXCOORD1;
            float3 tangentWorld : TEXCOORD2;  
            float3 normalWorld : TEXCOORD3;
            float3 binormalWorld : TEXCOORD4;
         };
 
         vertexOutput vert(vertexInput input) 
         {
            vertexOutput output;
            
            float4x4 modelMatrix = unity_ObjectToWorld;
            float4x4 modelMatrixInverse = unity_WorldToObject;
 
            output.tangentWorld = normalize(
               mul(modelMatrix, float4(input.tangent.xyz, 0.0)).xyz);
            output.normalWorld = normalize(
               mul(float4(input.normal, 0.0), modelMatrixInverse).xyz);
            output.binormalWorld = normalize(
               cross(output.normalWorld, output.tangentWorld) 
               * input.tangent.w); // tangent.w is specific to Unity

            output.posWorld = mul(modelMatrix, input.vertex);
            output.tex = input.texcoord;
            output.pos = UnityObjectToClipPos(input.vertex);
            return output;
         }

The factor input.tangent.w in the computation of binormalWorld is specific to Unity, i.e. Unity provides tangent vectors and normal maps such that we have to do this multiplication.

With the normalized directions T, B, and N in world space, we can easily form a matrix that maps any normal vector n of the normal map from the local surface coordinate system to world space because the columns of such a matrix are just the vectors of the axes; thus, the 3×3 matrix for the mapping of n to world space is:

$\mathrm {M} _{{\text{surface}}\to {\text{world}}}=\left[{\begin{matrix}T_{x}&B_{x}&N_{x}\\T_{y}&B_{y}&N_{y}\\T_{z}&B_{z}&N_{z}\end{matrix}}\right]$

In Cg, it is actually easier to construct the transposed matrix since matrices are constructed row by row

$\mathrm {M} _{{\text{surface}}\to {\text{world}}}^{T}=\left[{\begin{matrix}T_{x}&T_{y}&T_{z}\\B_{x}&B_{y}&B_{z}\\N_{x}&N_{y}&N_{z}\end{matrix}}\right]$

The construction is done in the fragment shader, e.g.:

            float3x3 local2WorldTranspose = float3x3(input.tangentWorld,
               input.binormalWorld, input.normalWorld);

We want to transform n with the transpose of local2WorldTranspose (i.e. the not transposed original matrix); therefore, we multiply n from the left with the matrix. For example, with this line:

            float3 normalDirection = 
               normalize(mul(localCoords, local2WorldTranspose));

With the new normal vector in world space, we can compute the lighting as in Section “Smooth Specular Highlights”.

Complete Shader Code

This shader code simply integrates all the snippets and uses our standard two-pass approach for pixel lights. It also demonstrates the use of a CGINCLUDE ... ENDCG block, which is implicitly shared by all passes of all subshaders.

Shader "Cg normal mapping" {
   Properties {
      _BumpMap ("Normal Map", 2D) = "bump" {}
      _Color ("Diffuse Material Color", Color) = (1,1,1,1) 
      _SpecColor ("Specular Material Color", Color) = (1,1,1,1) 
      _Shininess ("Shininess", Float) = 10
   }

   CGINCLUDE // common code for all passes of all subshaders

      #include "UnityCG.cginc"
      uniform float4 _LightColor0; 
      // color of light source (from "Lighting.cginc")

      // User-specified properties
      uniform sampler2D _BumpMap;   
      uniform float4 _BumpMap_ST;
      uniform float4 _Color; 
      uniform float4 _SpecColor; 
      uniform float _Shininess;

      struct vertexInput {
         float4 vertex : POSITION;
         float4 texcoord : TEXCOORD0;
         float3 normal : NORMAL;
         float4 tangent : TANGENT;
      };
      struct vertexOutput {
         float4 pos : SV_POSITION;
         float4 posWorld : TEXCOORD0;
         // position of the vertex (and fragment) in world space 
         float4 tex : TEXCOORD1;
         float3 tangentWorld : TEXCOORD2;  
         float3 normalWorld : TEXCOORD3;
         float3 binormalWorld : TEXCOORD4;
      };

      vertexOutput vert(vertexInput input) 
      {
         vertexOutput output;

         float4x4 modelMatrix = unity_ObjectToWorld;
         float4x4 modelMatrixInverse = unity_WorldToObject;

         output.tangentWorld = normalize(
            mul(modelMatrix, float4(input.tangent.xyz, 0.0)).xyz);
         output.normalWorld = normalize(
            mul(float4(input.normal, 0.0), modelMatrixInverse).xyz);
         output.binormalWorld = normalize(
            cross(output.normalWorld, output.tangentWorld) 
            * input.tangent.w); // tangent.w is specific to Unity

         output.posWorld = mul(modelMatrix, input.vertex);
         output.tex = input.texcoord;
         output.pos = UnityObjectToClipPos(input.vertex);
         return output;
      }

      // fragment shader with ambient lighting
      float4 fragWithAmbient(vertexOutput input) : COLOR
      {
         // in principle we have to normalize tangentWorld,
         // binormalWorld, and normalWorld again; however, the 
         // potential problems are small since we use this 
         // matrix only to compute "normalDirection", 
         // which we normalize anyways

         float4 encodedNormal = tex2D(_BumpMap, 
            _BumpMap_ST.xy * input.tex.xy + _BumpMap_ST.zw);
         float3 localCoords = float3(2.0 * encodedNormal.a - 1.0, 
             2.0 * encodedNormal.g - 1.0, 0.0);
         localCoords.z = sqrt(1.0 - dot(localCoords, localCoords));
         // approximation without sqrt:  localCoords.z = 
         // 1.0 - 0.5 * dot(localCoords, localCoords);

         float3x3 local2WorldTranspose = float3x3(
            input.tangentWorld, 
            input.binormalWorld, 
            input.normalWorld);
         float3 normalDirection = 
            normalize(mul(localCoords, local2WorldTranspose));

         float3 viewDirection = normalize(
            _WorldSpaceCameraPos - input.posWorld.xyz);
         float3 lightDirection;
         float attenuation;

         if (0.0 == _WorldSpaceLightPos0.w) // directional light?
         { 
            attenuation = 1.0; // no attenuation
            lightDirection = normalize(_WorldSpaceLightPos0.xyz);
         } 
         else // point or spot light
         {
            float3 vertexToLightSource = 
               _WorldSpaceLightPos0.xyz - input.posWorld.xyz;
            float distance = length(vertexToLightSource);
            attenuation = 1.0 / distance; // linear attenuation 
            lightDirection = normalize(vertexToLightSource);
         }

         float3 ambientLighting = 
            UNITY_LIGHTMODEL_AMBIENT.rgb * _Color.rgb;

         float3 diffuseReflection = 
            attenuation * _LightColor0.rgb * _Color.rgb
            * max(0.0, dot(normalDirection, lightDirection));

         float3 specularReflection;
         if (dot(normalDirection, lightDirection) < 0.0) 
            // light source on the wrong side?
         {
            specularReflection = float3(0.0, 0.0, 0.0); 
            // no specular reflection
         }
         else // light source on the right side
         {
            specularReflection = attenuation * _LightColor0.rgb 
               * _SpecColor.rgb * pow(max(0.0, dot(
               reflect(-lightDirection, normalDirection), 
               viewDirection)), _Shininess);
         }
         return float4(ambientLighting + diffuseReflection 
            + specularReflection, 1.0);
      }
      
      // fragment shader for pass 2 without ambient lighting 
      float4 fragWithoutAmbient(vertexOutput input) : COLOR
      {
        // in principle we have to normalize tangentWorld,
        // binormalWorld, and normalWorld again; however, the  
        // potential problems are small since we use this 
        // matrix only to compute "normalDirection", 
        // which we normalize anyways

        float4 encodedNormal = tex2D(_BumpMap, 
           _BumpMap_ST.xy * input.tex.xy + _BumpMap_ST.zw);
        float3 localCoords = float3(2.0 * encodedNormal.a - 1.0, 
            2.0 * encodedNormal.g - 1.0, 0.0);
        localCoords.z = sqrt(1.0 - dot(localCoords, localCoords));
           // approximation without sqrt:  localCoords.z = 
           // 1.0 - 0.5 * dot(localCoords, localCoords);

        float3x3 local2WorldTranspose = float3x3(
           input.tangentWorld,
           input.binormalWorld, 
           input.normalWorld);
        float3 normalDirection = 
           normalize(mul(localCoords, local2WorldTranspose));

        float3 viewDirection = normalize(
           _WorldSpaceCameraPos - input.posWorld.xyz);
        float3 lightDirection;
        float attenuation;

        if (0.0 == _WorldSpaceLightPos0.w) // directional light?
        {
           attenuation = 1.0; // no attenuation
           lightDirection = normalize(_WorldSpaceLightPos0.xyz);
        } 
        else // point or spot light
        {
           float3 vertexToLightSource = 
              _WorldSpaceLightPos0.xyz - input.posWorld.xyz;
           float distance = length(vertexToLightSource);
           attenuation = 1.0 / distance; // linear attenuation 
           lightDirection = normalize(vertexToLightSource);
        }

        float3 diffuseReflection = 
           attenuation * _LightColor0.rgb * _Color.rgb
           * max(0.0, dot(normalDirection, lightDirection));

        float3 specularReflection;
        if (dot(normalDirection, lightDirection) < 0.0) 
           // light source on the wrong side?
        {
           specularReflection = float3(0.0, 0.0, 0.0); 
              // no specular reflection
        }
        else // light source on the right side
        {
           specularReflection = attenuation * _LightColor0.rgb 
              * _SpecColor.rgb * pow(max(0.0, dot(
              reflect(-lightDirection, normalDirection), 
              viewDirection)), _Shininess);
        }
        return float4(diffuseReflection + specularReflection, 1.0);
      }
   ENDCG

   SubShader {
      Pass {      
         Tags { "LightMode" = "ForwardBase" } 
            // pass for ambient light and first light source
 
         CGPROGRAM
            #pragma vertex vert  
            #pragma fragment fragWithAmbient  
            // the functions are defined in the CGINCLUDE part
         ENDCG
      }
 
      Pass {      
         Tags { "LightMode" = "ForwardAdd" } 
            // pass for additional light sources
         Blend One One // additive blending 
 
         CGPROGRAM
            #pragma vertex vert  
            #pragma fragment fragWithoutAmbient
            // the functions are defined in the CGINCLUDE part
         ENDCG
      }
   }
}

Note that we have used the tiling and offset uniform _BumpMap_ST as explained in the Section “Textured Spheres” since this option is often particularly useful for bump maps.

Summary

Congratulations! You finished this tutorial! We have look at:

How human perception of shapes often relies on lighting.
What normal mapping is.
How Unity encodes normal maps.
How a fragment shader can decode Unity's normal maps and use them to per-pixel lighting.