Cg Programming/Unity/Outlining Objects
This tutorial covers a vertex transformation that moves each vertex along its surface normal vector to create a larger outline of an object. While this vertex transformation is quite simple, creating a reasonable rendering of the outline requires use of the stencil buffer, which is also discussed.
Creating an Outline by Moving Vertices along their Normal Vector edit
One method to create an outline that is around an object is to enlarge the object by moving its vertices along their surface normal vectors. Given the position and the normal vector in object coordinates, this is quite straightforward in a vertex shader:
uniform float _Thickness;
float4 vert(float4 vertex : POSITION, float3 normal : NORMAL) : SV_POSITION {
return mul(UNITY_MATRIX_MVP, float4(normal, 0.0) * _Thickness + vertex);
}
This vertex shader converts the surface normal vector normal into a float4 vector, then multiplies it with the user-specified uniform _Thickness (to allow for an adjustable thickness of the outline) and adds it to the position of the vertex in vertex. All this happens in object coordinates; thus, we have to convert to clip coordinates by multiplying with UNITY_MATRIX_MVP before returning the position.
There is not much more to say about this specific vertex transformation – except that it works best with smooth surfaces. Surfaces with hard edges (i.e., discontinuous surface normal vectors) do not work well. Sometimes it helps to use Cull Off to render frontfaces and backfaces, but in general the approach just doesn't work well with hard edges.
To give the outline a uniform green color, we can use a simple fragment shader like this:
float4 frag(void) : COLOR {
return float4(0.0, 1.0, 0.0, 1.0);
}
Avoiding Occlusions of the Outlined Object by its Outline edit
The more challenging part of rendering this kind of outline is to avoid occlusions of the outlined object by the outline: since we have enlarged the object to create the outline, the larger outline will usually occlude the object that we want to outline. On the other hand, the outline should occlude other objects in the background and it should be occluded by objects in the foreground, i.e., it should behave like any other opaque object, except when it is in front of the outlined object.
If we assume that we first render a regular version of the object that we want to outline, and after that render the outline, we can state the challenge in this way: the outline should only be rasterized for pixels that are not covered by the outlined object. In this way, the challenge sounds like something that can be solved with the stencil test, because the stencil test is used to limit the rasterization to certain parts of the framebuffer; in our case to the parts of the framebuffer that are not covered by the object that we want to outline.
Our strategy is then to first mark all pixels that are covered by the object that we want to outline in the stencil buffer. After that, when rendering the outline, we can use a stencil test to rasterize the outline only in pixels that haven't been marked.
To mark all pixels that are covered by an object with a value of 1 in the stencil buffer, we can use this ShaderLab syntax in a Pass block before CGPROGRAM:
Stencil {
Ref 1
Comp Always
Pass Replace
}
The Stencil keyword specifies that we want to activate a stencil test, which is necessary because writing to the stencil buffer technically is part of the stencil test. Ref 1 sets the reference value of the stencil test; in this case the value that we want to write into the stencil buffer.
Comp Always specifies that we want to work with all fragments regardless of the pixels' value in the stencil buffer. In general, Comp specifies a comparison for a pixel's value in the stencil buffer. If that comparison fails, the fragment is discarded and the pixel is not rasterized. Always specifies a comparison that always passes, i.e., none of the fragments are discarded.
Pass Replace specifies that the Replace "operation" should be applied to the stencil buffer if the comparison has been passed, i.e., the value of a pixel in the stencil buffer should be replaced by the reference value that is specified with Ref.
A complete pass to render a black object and at the same time mark the stencil buffer with 1 in all pixels that are covered by the object could look like this:
Pass {
Stencil {
Ref 1
Comp Always
Pass Replace
}
CGPROGRAM
#pragma vertex vert
#pragma fragment frag
float4 vert(float4 vertex : POSITION) : SV_POSITION
{
return mul(UNITY_MATRIX_MVP, vertex);
}
float4 frag(void) : COLOR {
return float4(0.0, 0.0, 0.0, 1.0);
}
ENDCG
}
The second part of our approach to to render the outline only where the stencil buffer has not been marked with a 1. Such a stencil test could be specified in this way:
Stencil {
Ref 1
Comp NotEqual
Pass Keep
}
Again, Ref 1 sets the reference value of the stencil test, but in this case the value is used for the comparison.
Comp NotEqual specifies a comparison for a pixel's value in the stencil buffer that is passed only if the value is not equal to the reference value. If that comparison fails (i.e., if the value in the stencil buffer is equal to 1) the fragment is discarded and the pixel is not rasterized, which is what we want for our outline such that it doesn't occlude the object that we want to outline.
Pass Keep specifies that the Keep "operation" should be applied to the stencil buffer if the comparison has been passed, i.e., we don't change the stencil buffer but simply keep whatever values are already in the stencil buffer. (This is important because multiple triangles of the mesh might cover the same pixel.)
This stencil test is applied to the shader above for the outline that had its vertices moved along the surface normal vector.
Complete Shader Code edit
Putting everything together and defining a properties block for the _Thickness uniform variable, we have this shader:
Shader "OutlinedObject" {
Properties {
_Thickness ("Thickness", Float) = 0.1
}
SubShader {
Pass {
Stencil {
Ref 1
Comp Always
Pass Replace
}
CGPROGRAM
#pragma vertex vert
#pragma fragment frag
float4 vert(float4 vertex : POSITION) : SV_POSITION
{
return mul(UNITY_MATRIX_MVP, vertex);
}
float4 frag(void) : COLOR {
return float4(0.0, 0.0, 0.0, 1.0);
}
ENDCG
}
Pass {
Cull Off
Stencil {
Ref 1
Comp NotEqual
Pass Keep
}
CGPROGRAM
#pragma vertex vert
#pragma fragment frag
uniform float _Thickness;
float4 vert(float4 vertex : POSITION, float3 normal : NORMAL) : SV_POSITION {
return mul(UNITY_MATRIX_MVP, vertex + normal * _Thickness);
}
float4 frag(void) : COLOR {
return float4(0.0, 1.0, 0.0, 1.0);
}
ENDCG
}
}
}
This shader renders opaque green outlines for uniformly black objects. However, you can easily replace the vertex and fragment shaders to render transparent outlines of other colors or shaded objects – as long as you keep the stencil tests and the outline in the 2nd pass is larger than the object in the 1st pass.
Further reading edit
If you still want to know more
- about how the stencil test fits into the OpenGL pipeline, you should read Section “Per-Fragment Operations”.
- about other ways of rendering silhouettes, you should read Section “Toon Shading”.
- about the specification of the stencil test, you should read Unity's documentation about “ShaderLab: Stencil”.