Cg Programming/Unity/Many Light Sources
This tutorial introduces image-based lighting, in particular diffuse (irradiance) environment mapping and its implementation with cube maps. (Unity's Light Probes presumably work in a similar way but with dynamically rendered cube maps.)
This tutorial is based on Section “Reflecting Surfaces”. If you haven't read that tutorial, this would be a very good time to read it.
Diffuse Lighting by Many Lights
editConsider the lighting of the sculpture in the image to the left. There is natural light coming through the windows. Some of this light bounces off the floor, walls and visitors before reaching the sculpture. Additionally, there are artificial light sources, and their light is also shining directly and indirectly onto the sculpture. How many directional lights and point lights would be needed to simulate this kind of complex lighting environment convincingly? At least more than a handful (probably more than a dozen) and therefore the performance of the lighting computations is challenging.
This problem is addressed by image-based lighting. For static lighting environments that are described by an environment map, e.g. a cube map, image-based lighting allows us to compute the lighting by an arbitrary number of light sources with a single texture lookup in a cube map (see Section “Reflecting Surfaces” for a description of cube maps). How does it work?
In this section we focus on diffuse lighting. Assume that every texel (i.e. pixel) of a cube map acts as a directional light source. (Remember that cube maps are usually assumed to be infinitely large such that only directions matter, but positions don't.) The resulting lighting for a given surface normal direction can be computed as described in Section “Diffuse Reflection”. It's basically the cosine between the surface normal vector N and the vector to the light source L:
Since the texels are the light sources, L is just the direction from the center of the cube to the center of the texel in the cube map. A small cube map with 32×32 texels per face has already 32×32×6 = 6144 texels. Adding the illumination by thousands of light sources is not going to work in real time. However, for a static cube map we can compute the diffuse illumination for all possible surface normal vectors N in advance and store them in a lookup table. When lighting a point on a surface with a specific surface normal vector, we can then just look up the diffuse illumination for the specific surface normal vector N in that precomputed lookup table.
Thus, for a specific surface normal vector N we add (i.e. integrate) the diffuse illumination by all texels of the cube map. We store the resulting diffuse illumination for this surface normal vector in a second cube map (the “diffuse irradiance environment map” or “diffuse environment map” for short). This second cube map will act as a lookup table, where each direction (i.e. surface normal vector) is mapped to a color (i.e. diffuse illumination by potentially thousands of light sources). The fragment shader is therefore really simple (this one could use the vertex shader from Section “Reflecting Surfaces”):
float4 frag(vertexOutput input) : COLOR
{
return texCUBE(_Cube, input.normalDir);
}
It is just a lookup of the precomputed diffuse illumination using the surface normal vector of the rasterized surface point. However, the precomputation of the diffuse environment map is somewhat more complicated as described in the next section.
Computation of Diffuse Environment Maps
editThis section presents some C# code to illustrate the computation of cube maps for diffuse (irradiance) environment maps. In order to use it in Unity, choose Create > C# script in the Project Window and call is "ComputeDiffuseEnvironmentMap". Then open the script in Unity's text editor, copy the C# code into it, and attach the script to the game object that has a material with the shader presented below. When a new readable cube map of sufficiently small dimensions is specified for the shader property _OriginalCube (which is labeled Environment Map in the shader user interface), the script will update the shader property _Cube (i.e. Diffuse Environment Map in the user interface) with a corresponding diffuse environment map. Note that the cube map has to be "readable," i.e. you have to check Readable in the Inspector when creating the cube map. Also note that you should use small cube maps of face dimensions 32×32 or smaller because the computation time tends to be very long for larger cube maps. Thus, when creating a cube map in Unity, make sure to choose a sufficiently small size.
The script includes only a handful of functions: Awake() initializes the variables; Update() takes care of communicating with the user and the material (i.e. reading and writing shader properties); computeFilteredCubemap() does the actual work of computing the diffuse environment map; and getDirection() is a small utility function for computeFilteredCubemap() to compute the direction associated with each texel of a cube map. Note that computeFilteredCubemap() not only integrates the diffuse illumination but also avoids discontinuous seams between faces of the cube map by setting neighboring texels along the seams to the same averaged color.
Make sure to call the C# script file "ComputeDiffuseEnvironmentMap".
using UnityEngine;
using UnityEditor;
using System.Collections;
[ExecuteInEditMode]
public class ComputeDiffuseEnvironmentMap : MonoBehaviour
{
public Cubemap originalCubeMap;
// environment map specified in the shader by the user
//[System.Serializable]
// avoid being deleted by the garbage collector,
// and thus leaking
private Cubemap filteredCubeMap;
// the computed diffuse irradience environment map
private void Update()
{
Cubemap originalTexture = null;
try
{
originalTexture = GetComponent<Renderer>().sharedMaterial.GetTexture(
"_OriginalCube") as Cubemap;
}
catch (System.Exception)
{
Debug.LogError("'_OriginalCube' not found on shader. "
+ "Are you using the wrong shader?");
return;
}
if (originalTexture == null)
// did the user set "none" for the map?
{
if (originalCubeMap != null)
{
GetComponent<Renderer>().sharedMaterial.SetTexture("_Cube", null);
originalCubeMap = null;
filteredCubeMap = null;
return;
}
}
else if (originalTexture == originalCubeMap
&& filteredCubeMap != null
&& GetComponent<Renderer>().sharedMaterial.GetTexture("_Cube") == null)
{
GetComponent<Renderer>().sharedMaterial.SetTexture("_Cube",
filteredCubeMap); // set the computed
// diffuse environment map in the shader
}
else if (originalTexture != originalCubeMap
|| filteredCubeMap
!= GetComponent<Renderer>().sharedMaterial.GetTexture("_Cube"))
{
if (EditorUtility.DisplayDialog(
"Processing of Environment Map",
"Do you want to process the cube map of face size "
+ originalTexture.width + "x" + originalTexture.width
+ "? (This will take some time.)",
"OK", "Cancel"))
{
if (filteredCubeMap
!= GetComponent<Renderer>().sharedMaterial.GetTexture("_Cube"))
{
if (GetComponent<Renderer>().sharedMaterial.GetTexture("_Cube")
!= null)
{
DestroyImmediate(
GetComponent<Renderer>().sharedMaterial.GetTexture(
"_Cube")); // clean up
}
}
if (filteredCubeMap != null)
{
DestroyImmediate(filteredCubeMap); // clean up
}
originalCubeMap = originalTexture;
filteredCubeMap = computeFilteredCubeMap();
//computes the diffuse environment map
GetComponent<Renderer>().sharedMaterial.SetTexture("_Cube",
filteredCubeMap); // set the computed
// diffuse environment map in the shader
return;
}
else
{
originalCubeMap = null;
filteredCubeMap = null;
GetComponent<Renderer>().sharedMaterial.SetTexture("_Cube", null);
GetComponent<Renderer>().sharedMaterial.SetTexture(
"_OriginalCube", null);
}
}
}
// This function computes a diffuse environment map in
// "filteredCubemap" of the same dimensions as "originalCubemap"
// by integrating -- for each texel of "filteredCubemap" --
// the diffuse illumination from all texels of "originalCubemap"
// for the surface normal vector corresponding to the direction
// of each texel of "filteredCubemap".
private Cubemap computeFilteredCubeMap()
{
Cubemap filteredCubeMap = new Cubemap(originalCubeMap.width,
originalCubeMap.format, true);
int filteredSize = filteredCubeMap.width;
int originalSize = originalCubeMap.width;
// Compute all texels of the diffuse environment cube map
// by itterating over all of them
for (int filteredFace = 0; filteredFace < 6; filteredFace++)
// the six sides of the cube
{
for (int filteredI = 0; filteredI < filteredSize; filteredI++)
{
for (int filteredJ = 0; filteredJ < filteredSize; filteredJ++)
{
Vector3 filteredDirection =
getDirection(filteredFace,
filteredI, filteredJ, filteredSize).normalized;
float totalWeight = 0.0f;
Vector3 originalDirection;
Vector3 originalFaceDirection;
float weight;
Color filteredColor = new Color(0.0f, 0.0f, 0.0f);
// sum (i.e. integrate) the diffuse illumination
// by all texels in the original environment map
for (int originalFace = 0; originalFace < 6; originalFace++)
{
originalFaceDirection = getDirection(
originalFace, 1, 1, 3).normalized;
//the normal vector of the face
for (int originalI = 0; originalI < originalSize; originalI++)
{
for (int originalJ = 0; originalJ < originalSize; originalJ++)
{
originalDirection = getDirection(
originalFace, originalI,
originalJ, originalSize);
// direction to the texel
// (i.e. light source)
weight = 1.0f
/ originalDirection.sqrMagnitude;
// take smaller size of more
// distant texels into account
originalDirection =
originalDirection.normalized;
weight = weight * Vector3.Dot(
originalFaceDirection,
originalDirection);
// take tilt of texel compared
// to face into account
weight = weight * Mathf.Max(0.0f,
Vector3.Dot(filteredDirection,
originalDirection));
// directional filter
// for diffuse illumination
totalWeight = totalWeight + weight;
// instead of analytically
// normalization, we just normalize
// to the potential max illumination
filteredColor = filteredColor + weight
* originalCubeMap.GetPixel(
(CubemapFace)originalFace,
originalI, originalJ); // add the
// illumination by this texel
}
}
}
filteredCubeMap.SetPixel(
(CubemapFace)filteredFace, filteredI,
filteredJ, filteredColor / totalWeight);
// store the diffuse illumination of this texel
}
}
}
// Avoid seams between cube faces: average edge texels
// to the same color on each side of the seam
int maxI = filteredCubeMap.width - 1;
for (int i = 0; i < maxI; i++)
{
setFaceAverage(ref filteredCubeMap,
0, i, 0, 2, maxI, maxI - i);
setFaceAverage(ref filteredCubeMap,
0, 0, i, 4, maxI, i);
setFaceAverage(ref filteredCubeMap,
0, i, maxI, 3, maxI, i);
setFaceAverage(ref filteredCubeMap,
0, maxI, i, 5, 0, i);
setFaceAverage(ref filteredCubeMap,
1, i, 0, 2, 0, i);
setFaceAverage(ref filteredCubeMap,
1, 0, i, 5, maxI, i);
setFaceAverage(ref filteredCubeMap,
1, i, maxI, 3, 0, maxI - i);
setFaceAverage(ref filteredCubeMap,
1, maxI, i, 4, 0, i);
setFaceAverage(ref filteredCubeMap,
2, i, 0, 5, maxI - i, 0);
setFaceAverage(ref filteredCubeMap,
2, i, maxI, 4, i, 0);
setFaceAverage(ref filteredCubeMap,
3, i, 0, 4, i, maxI);
setFaceAverage(ref filteredCubeMap,
3, i, maxI, 5, maxI - i, maxI);
}
// Avoid seams between cube faces:
// average corner texels to the same color
// on all three faces meeting in one corner
setCornerAverage(ref filteredCubeMap,
0, 0, 0, 2, maxI, maxI, 4, maxI, 0);
setCornerAverage(ref filteredCubeMap,
0, maxI, 0, 2, maxI, 0, 5, 0, 0);
setCornerAverage(ref filteredCubeMap,
0, 0, maxI, 3, maxI, 0, 4, maxI, maxI);
setCornerAverage(ref filteredCubeMap,
0, maxI, maxI, 3, maxI, maxI, 5, 0, maxI);
setCornerAverage(ref filteredCubeMap,
1, 0, 0, 2, 0, 0, 5, maxI, 0);
setCornerAverage(ref filteredCubeMap,
1, maxI, 0, 2, 0, maxI, 4, 0, 0);
setCornerAverage(ref filteredCubeMap,
1, 0, maxI, 3, 0, maxI, 5, maxI, maxI);
setCornerAverage(ref filteredCubeMap,
1, maxI, maxI, 3, 0, 0, 4, 0, maxI);
filteredCubeMap.Apply(); //apply all SetPixel(..) commands
return filteredCubeMap;
}
private void setFaceAverage(ref Cubemap filteredCubeMap,
int a, int b, int c, int d, int e, int f)
{
Color average =
(filteredCubeMap.GetPixel((CubemapFace)a, b, c)
+ filteredCubeMap.GetPixel((CubemapFace)d, e, f)) / 2.0f;
filteredCubeMap.SetPixel((CubemapFace)a, b, c, average);
filteredCubeMap.SetPixel((CubemapFace)d, e, f, average);
}
private void setCornerAverage(ref Cubemap filteredCubeMap,
int a, int b, int c, int d, int e, int f, int g, int h, int i)
{
Color average =
(filteredCubeMap.GetPixel((CubemapFace)a, b, c)
+ filteredCubeMap.GetPixel((CubemapFace)d, e, f)
+ filteredCubeMap.GetPixel((CubemapFace)g, h, i)) / 3.0f;
filteredCubeMap.SetPixel((CubemapFace)a, b, c, average);
filteredCubeMap.SetPixel((CubemapFace)d, e, f, average);
filteredCubeMap.SetPixel((CubemapFace)g, h, i, average);
}
private Vector3 getDirection(int face, int i, int j, int size)
{
switch (face)
{
case 0:
return new Vector3(0.5f,
-((j + 0.5f) / size - 0.5f),
-((i + 0.5f) / size - 0.5f));
case 1:
return new Vector3(-0.5f,
-((j + 0.5f) / size - 0.5f),
((i + 0.5f) / size - 0.5f));
case 2:
return new Vector3(((i + 0.5f) / size - 0.5f),
0.5f, ((j + 0.5f) / size - 0.5f));
case 3:
return new Vector3(((i + 0.5f) / size - 0.5f),
-0.5f, -((j + 0.5f) / size - 0.5f));
case 4:
return new Vector3(((i + 0.5f) / size - 0.5f),
-((j + 0.5f) / size - 0.5f), 0.5f);
case 5:
return new Vector3(-((i + 0.5f) / size - 0.5f),
-((j + 0.5f) / size - 0.5f), -0.5f);
default:
return Vector3.zero;
}
}
}
Complete Shader Code
editAs promised, the actual shader code is very short; the vertex shader is a reduced version of the vertex shader of Section “Reflecting Surfaces”:
Shader "Cg shader with image-based diffuse lighting" {
Properties {
_OriginalCube ("Environment Map", Cube) = "" {}
_Cube ("Diffuse Environment Map", Cube) = "" {}
}
SubShader {
Pass {
CGPROGRAM
#pragma vertex vert
#pragma fragment frag
#include "UnityCG.cginc"
// User-specified uniforms
uniform samplerCUBE _Cube;
struct vertexInput {
float4 vertex : POSITION;
float3 normal : NORMAL;
};
struct vertexOutput {
float4 pos : SV_POSITION;
float3 normalDir : TEXCOORD0;
};
vertexOutput vert(vertexInput input)
{
vertexOutput output;
float4x4 modelMatrixInverse = unity_WorldToObject;
// multiplication with unity_Scale.w is unnecessary
// because we normalize transformed vectors
output.normalDir = normalize(
mul(float4(input.normal, 0.0), modelMatrixInverse).xyz);
output.pos = UnityObjectToClipPos(input.vertex);
return output;
}
float4 frag(vertexOutput input) : COLOR
{
return texCUBE(_Cube, input.normalDir);
}
ENDCG
}
}
}
Changes for Specular (i.e. Glossy) Reflection
editThe shader and script above are sufficient to compute diffuse illumination by a large number of static, directional light sources. But what about the specular illumination discussed in Section “Specular Highlights”, i.e.:
First, we have to rewrite this equation such that it depends only on the direction to the light source L and the reflected view vector R :
With this equation, we can compute a lookup table (i.e. a cube map) that contains the specular illumination by many light sources for any reflected view vector R . In order to look up the specular illumination with such a table, we just need to compute the reflected view vector and perform a texture lookup in a cube map. In fact, this is exactly what the shader code of Section “Reflecting Surfaces” does. Thus, we actually only need to compute the lookup table.
It turns out that the JavaScript code presented above can be easily adapted to compute such a lookup table. All we have to do is to change the line
weight = weight * Mathf.Max(0.0,
Vector3.Dot(filteredDirection, originalDirection));
// directional filter for diffuse illumination
to
weight = weight * Mathf.Pow(Mathf.Max(0.0,
Vector3.Dot(filteredDirection, originalDirection)), 50.0);
// directional filter for specular illumination
where 50.0 should be replaced by a variable for . This allows us to compute lookup tables for any specific shininess. (The same cube map could be used for varying values of the shininess if the mipmap-level was specified explicitly using the textureCubeLod instruction in the shader; however, this technique is beyond the scope of this tutorial.)
Summary
editCongratulations, you have reached the end of a rather advanced tutorial! We have seen:
- What image-based rendering is about.
- How to compute and use a cube map to implement a diffuse environment map.
- How to adapt the code for specular reflection.
Further reading
editIf you still want to know more
- about cube maps, you should read Section “Reflecting Surfaces”.
- about (dynamic) diffuse environment maps, you could read Chapter 10, “Real-Time Computation of Dynamic Irradiance Environment Maps” by Gary King of the book “GPU Gems 2” by Matt Pharr (editor) published 2005 by Addison-Wesley, which is available online.
- about Unity's built-in method for dynamic image-based lighting, you should read Unity's documentation of Light Probes.