GLSL Programming/Unity/Nonlinear Deformations

This tutorial introduces vertex blending as an example of a nonlinear deformation. The main application is actually the rendering of skinned meshes.

While this tutorial is not based on any other specific tutorial, a good understanding of Section “Vertex Transformations” is very useful.

Blending between Two Model Transformations

Most deformations of meshes cannot be modeled by the affine transformations with 4×4 matrices that are discussed in Section “Vertex Transformations”. The deformation of bodies by tight corsets is just one example. A more important example in computer graphics is the deformation of meshes when joints are bent, e.g. elbows or knees.

This tutorial introduces vertex blending to implement some of these deformations. The basic idea is to apply multiple model transformations in the vertex shader (in this tutorial we use only two model transformations) and then blend the transformed vertices, i.e. compute a weighted average of them with weights that have to be specified for each vertex. For example, the deformation of the skin near a joint of a skeleton is mainly influenced by the position and orientation of the two (rigid) bones meeting in the joint. Thus, the positions and orientations of the two bones define two affine transformations. Different points on the skin are influenced differently by the two bones: points at the joint might be influenced equally by the two bones while points farther from the joint around one bone are more strongly influenced by that bone than the other. These different strengths of the influence of the two bones can be implemented by using different weights in the weighted average of the two transformations.

For the purpose of this tutorial, we use two uniform transformations mat4 _Trafo0 and mat4 _Trafo1, which are specified by the user. To this end a small JavaScript (which should be attached to the mesh that should be deformed) allows us to specify two other game objects and copies their model transformations to the uniforms of the shader:

@script ExecuteInEditMode()

public var bone0 : GameObject;
public var bone1 : GameObject;

function Update () 
{
   if (null != bone0)
   {
      renderer.sharedMaterial.SetMatrix("_Trafo0", 
         bone0.renderer.localToWorldMatrix);
   }
   if (null != bone1)
   {
      renderer.sharedMaterial.SetMatrix("_Trafo1", 
         bone1.renderer.localToWorldMatrix);
   }
   if (null != bone0 && null != bone1)
   {
      transform.position = 0.5 * (bone0.transform.position 
         + bone1.transform.position);
      transform.rotation = bone0.transform.rotation;
   }
}

The two other game objects could be anything — I like cubes with one of the built-in semitransparent shaders such that their position and orientation is visible but they don't occlude the deformed mesh.

In this tutorial, the weight for the blending with the transformation _Trafo0 is set to gl_Vertex.z + 0.5:

float weight0 = gl_Vertex.z + 0.5;

and the other weight is 1.0 - weight0. Thus, the part with positive gl_Vertex.z coordinates is influenced more by _Trafo0 and the other part is influenced more by _Trafo1. In general, the weights are application dependent and the user should be allowed to specify weights for each vertex.

The application of the two transformations and the weighted average can be written this way:

vec4 blendedVertex = weight0 * (_Trafo0 * gl_Vertex) + (1.0 - weight0) * (_Trafo1 * gl_Vertex);

Then the blended vertex has to be multiplied with the view matrix and the projection matrix. The view transformation is not available directly but it can be computed by multiplying the model-view matrix (which is the product of the view matrix and the model matrix) with the inverse model matrix (which is available as _World2Object times unity_Scale.w except for the bottom-right element, which is 1):

      mat4 modelMatrixInverse = _World2Object * unity_Scale.w;
      modelMatrixInverse[3][3] = 1.0; 
      mat4 viewMatrix = gl_ModelViewMatrix * modelMatrixInverse;

      gl_Position = gl_ProjectionMatrix * viewMatrix * blendedVertex;

In order to illustrate the different weights, we visualize weight0 by the red component and 1.0 - weight0 by the green component of a color (which is set in the fragment shader):

color = vec4(weight0, 1.0 - weight0, 0.0, 1.0);

For an actual application, we could also transform the normal vector by the two corresponding transposed inverse model transformations and perform per-pixel lighting in the fragment shader.

Complete Shader Code

All in all, the shader code looks like this:

Shader "GLSL shader for vertex blending" {
   SubShader {
      Pass {   
         GLSLPROGRAM

         // Uniforms set by a script
         uniform mat4 _Trafo0; // model transformation of bone0
         uniform mat4 _Trafo1; // model transformation of bone1

         // The following built-in uniforms 
         // are also defined in "UnityCG.glslinc", 
         // i.e. one could #include "UnityCG.glslinc" 
         uniform mat4 _Object2World; // model matrix
         uniform vec4 unity_Scale; // w = 1/scale
         uniform mat4 _World2Object; // inverse model matrix
            // all but the bottom-right element should be 
            // multiplied with unity_Scale.w
         
         // Varyings
         varying vec4 color;

         #ifdef VERTEX
                  
         void main()
         {
            float weight0 = gl_Vertex.z + 0.5; // depends on the mesh

            vec4 blendedVertex = weight0 * (_Trafo0 * gl_Vertex) 
               + (1.0 - weight0) * (_Trafo1 * gl_Vertex);

            mat4 modelMatrixInverse = _World2Object * unity_Scale.w;
            modelMatrixInverse[3][3] = 1.0; 
            mat4 viewMatrix = gl_ModelViewMatrix * modelMatrixInverse;
            
            gl_Position = 
               gl_ProjectionMatrix * viewMatrix * blendedVertex;

            color = vec4(weight0, 1.0 - weight0, 0.0, 1.0); 
               // visualize weight0 as red and weight1 as green
         }
         
         #endif


         #ifdef FRAGMENT
         
         void main()
         {
            gl_FragColor = color;
         }
         
         #endif

         ENDGLSL
      }
   }
}

This is, of course, only an illustration of the concept but it can already be used for some interesting nonlinear deformations such as twists around the $z$ axis.

For skinned meshes in skeletal animation, many more bones (i.e. model transformations) are necessary and each vertex has to specify which bone (using, for example, an index) contributes with which weight to the weighted average. However, Unity computes the blending of vertices in software; thus, this topic is less relevant for Unity programmers.

Summary

Congratulations, you have reached the end of another tutorial. We have seen:

How to blend vertices that are transformed by two model matrices.
How this technique can be used for nonlinear transformations and skinned meshes.

GLSL Programming/Unity/Nonlinear Deformations

Contents

Blending between Two Model Transformations

Complete Shader Code

Summary

Further Reading