When you draw something with OpenGL, you see colors on the screen, but keep in mind that there are other buffers than the color buffer. You are already familiar with the depth buffer, that prevent background pixels from being displayed if there is a closer pixel already. Now it's time to introduce the stencil buffer.

Concept

The stencil buffer is another buffer for your custom use: you can store per-pixel information there and direct OpenGL to act differently depending on that information.

There are quite a few points to understand to manipulate the stencil buffer correctly, so let's take time to learn them.

Bitplanes

This is mostly a matter of vocabulary: the per-pixel information is a number of bitplanes - that is, of bits.

For instance, it is common to have 8 bitplanes, which means one byte is associated with each pixel on screen, and you can store up to $2^{8}=256$ different values. You can inspect the number of bitplanes using glGetIntegerv(GL_STENCIL_BITS, &i).

Tests and Operations

The stencil buffer can be manipulated using glStencilFunc and glStencilOp: glStencilFunc specifies a test to apply to each pixel of the stencil buffer, then glStencilOp specifies the action to apply depending on the test result.

glStencilFunc takes 3 parameters, to build the following test (ref & mask) OP (stencil & mask):

OP: one of GL_NEVER, GL_ALWAYS, GL_EQUAL, GL_NOTEQUAL, GL_LESS, GL_LEQUAL, GL_GEQUAL, GL_GREATER
ref: a fixed integer used in the comparison
mask: a mask applied to both ref and the stencil pixel; you can use 0xFF (if you have 8 bitplanes) to disable the mask

glStencilOp takes 3 parameters:

sfail: the test from glStencilFunc failed
dpfail: the test from glStencilFunc passed, but the depth buffer test failed
dppass: the test from glStencilFunc passed, and the depth buffer passed or is disabled

Each of these 3 parameters is an action to perform on the stencil buffer, one of GL_KEEP, GL_ZERO, GL_REPLACE, GL_INCR, GL_INCR_WRAP, GL_DECR, GL_DECR_WRAP, GL_INVERT (default GL_KEEP).

So, we see that two different tests are chained, resulting in 3 situations:

first glStencilFunc, if it fails we apply sfail and stop
then depth-test (if depth buffer is available and enabled), if it fails we execute dpfail and stop
if both tests work, we evaluate the fragment shader in the color buffer and apply dppass

Note that with sfail and dpfail, the color buffer is left unmodified.

If you draw a volumetric shape, keep in mind that OpenGL may draw the same stencil pixel several times, depending on the order of overlapping triangles and/or the availability of the depth buffer; in such cases, your action will be applied several times.

Sample

Rotating cube, clipped

In our example, we'll draw a moving circle, and the scene will be clipped inside that circle:

/* main */
  glutInitDisplayMode(GLUT_RGBA|GLUT_ALPHA|GLUT_DOUBLE|GLUT_DEPTH|GLUT_STENCIL);

/* onDisplay */
  glClear(GL_DEPTH_BUFFER_BIT);
  glEnable(GL_STENCIL_TEST);
  glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
  glDepthMask(GL_FALSE);
  glStencilFunc(GL_NEVER, 1, 0xFF);
  glStencilOp(GL_REPLACE, GL_KEEP, GL_KEEP);  // draw 1s on test fail (always)

  // draw stencil pattern
  glStencilMask(0xFF);
  glClear(GL_STENCIL_BUFFER_BIT);  // needs mask=0xFF
  draw_circle();

  glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
  glDepthMask(GL_TRUE);
  glStencilMask(0x00);
  // draw where stencil's value is 0
  glStencilFunc(GL_EQUAL, 0, 0xFF);
  /* (nothing to draw) */
  // draw only where stencil's value is 1
  glStencilFunc(GL_EQUAL, 1, 0xFF);

  draw_scene();

  glDisable(GL_STENCIL_TEST);

draw_scene draws the color cube from tutorial 05, and draw_circle draws a circle in 2d mode. We won't discuss these functions because they are not related to the stencil feature, but you're invited to look at the source code (see at the link at the bottom of this page).

Note that there's no need to use a separate program to draw in the stencil buffer. In this case it makes sense, because we draw a 2D pattern, but it's perfectly possible to draw a 3D pattern in the same scene, as in the Mini-Portal tutorial.

Working on color, depth and/or stencil buffers specifically

In the sample, we see that we first worked exclusively on the stencil buffer to draw the stencil shape, then exclusively on the color buffer to use it as a mask.

When you want to modify the stencil buffer without modifying the color buffer and/or depth buffer, you can mask these buffers:

  glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
  glDepthMask(GL_FALSE);

Don't forget to set them back. You can save the previous values this way:

  GLboolean save_color_mask[4];
  GLboolean save_depth_mask;
  glGetBooleanv(GL_COLOR_WRITEMASK, save_color_mask);
  glGetBooleanv(GL_DEPTH_WRITEMASK, &save_depth_mask);

  /* Do something */

  glColorMask(save_color_mask[0], save_color_mask[1], save_color_mask[2], save_color_mask[3]);
  glDepthMask(save_depth_mask);

If you want to apply changes on the color buffer without modifying the stencil buffer, you can either:

set glStencilMask(0x00) - this means nothing will be written in any condition
set glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP) - which applies a no-op for all conditions

Debugging

It is difficult to understand what is wrong when working on the stencil buffer, because we have no direct way to see it.

If you use OpenGL non-ES, you can call glReadPixels to grab the whole stencil buffer and inspect it externally, but it's tedious.

Another method is to fill the screen using two triangles, and with a stencil test (for instance, to match only non-0 stencil pixels). You may need to enable/disable the color buffer when doing so.

Then swap the OpenGL color buffer and pause, so you can inspect the result visually:

      glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
      glClear(GL_COLOR_BUFFER_BIT);
      glStencilMask(0x00);
      glStencilFunc(GL_LEQUAL, 1, 0xFF);
      fill_screen();
      glutSwapBuffers();
      cout << "swap" << endl;
      sleep(1);
      glStencilMask(0xFF);
      glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);

Your fill_screen() function will preferably use a dedicated program that will involve no MVP matrix, though you can reuse an existing program and pass v=glm::mat4(1) and m=glm::inverse(projection).

Performances

We saw there are three possible actions in glStencilOp:

sfail
dpfail
dppass

It is true that it's counter-intuitive to express your action on the stencil buffer using the sfail action, because of the double-negation (instead of "do this when this condition is met", you have to write "don't do this when this condition is not met).

However, if you use dppass to express your action, keep in mind that OpenGL will check both the stencil and depth test, and will also compute the pixel value using the fragment shader. This is a terrible performance hit if you are only drawing a shape in the stencil buffer - because you don't need to call the fragment shader at all.

A suggestion is to first implement your drawing algorithm using dppass for clarity, and once it works, reverse the conditions in glStencilFunc and use the sfail. sfail will not try to evaluate the pixel fragment when the test fails, so it's faster.

Example:

First prototype with:

    glStencilFunc(GL_NOTEQUAL, 0, 0xFF);
    glStencilOp(GL_KEEP, GL_KEEP, GL_INCR);  // +1 when current pixel != 0

then optimize:

    glStencilFunc(GL_EQUAL, 0, 0xFF);
    glStencilOp(GL_INCR, GL_KEEP, GL_KEEP);  // +1 when !(current pixel == 0)

Limitations

The OpenGL 4.2 core profile specification mentions:

However, when both depth and stencil attachments are present, implementations are only required to support framebuffer objects where both attachments refer to the same image. ^[1]

This means that you probably cannot attach two renderbuffers to a single frame buffer.

In OpenGL ES 2.0 in particular, combined formats are not supported (GL_FRAMEBUFFER_UNSUPPORTED) in glRenderbufferStorage (only DEPTH_COMPONENT16 and STENCIL_INDEX8 are supported, typically not DEPTH32F_STENCIL8 or DEPTH24_STENCIL8).

If you need to combine post-processing and stenciling, you'll need to avoid framebuffer objects and use the glCopyTexSubImage2D technique instead.