OpenGL Programming/Stencil buffer

When you draw something with OpenGL, you see colors on the screen, but keep in mind that there are other buffers than the color buffer. You are already familiar with the depth buffer, that prevent background pixels from being displayed if there is a closer pixel already. Now it's time to introduce the stencil buffer.

ConceptEdit

The stencil buffer is another buffer for your custom use: you can store per-pixel information there and direct OpenGL to act differently depending on that information.

There are quite a few points to understand to manipulate the stencil buffer correctly, so let's take time to learn them.

BitplanesEdit

This is mostly a matter of vocabulary: the per-pixel information is a number of bitplanes - that is, of bits.

For instance, it is common to have 8 bitplanes, which means one byte is associated with each pixel on screen, and you can store up to ${\displaystyle 2^{8}=256}$ different values. You can inspect the number of bitplanes using `glGetIntegerv(GL_STENCIL_BITS, &i)`.

Tests and OperationsEdit

The stencil buffer can be manipulated using `glStencilFunc` and `glStencilOp`: `glStencilFunc` specifies a test to apply to each pixel of the stencil buffer, then `glStencilOp` specifies the action to apply depending on the test result.

`glStencilFunc` takes 3 parameters, to build the following test `(ref & mask) OP (stencil & mask)`:

• OP: one of GL_NEVER, GL_ALWAYS, GL_EQUAL, GL_NOTEQUAL, GL_LESS, GL_LEQUAL, GL_GEQUAL, GL_GREATER
• ref: a fixed integer used in the comparison
• mask: a mask applied to both ref and the stencil pixel; you can use 0xFF (if you have 8 bitplanes) to disable the mask

`glStencilOp` takes 3 parameters:

• sfail: the test from `glStencilFunc` failed
• dpfail: the test from `glStencilFunc` passed, but the depth buffer test failed
• dppass: the test from `glStencilFunc` passed, and the depth buffer passed or is disabled

Each of these 3 parameters is an action to perform on the stencil buffer, one of GL_KEEP, GL_ZERO, GL_REPLACE, GL_INCR, GL_INCR_WRAP, GL_DECR, GL_DECR_WRAP, GL_INVERT (default GL_KEEP).

So, we see that two different tests are chained, resulting in 3 situations:

• first `glStencilFunc`, if it fails we apply sfail and stop
• then depth-test (if depth buffer is available and enabled), if it fails we execute dpfail and stop
• if both tests work, we evaluate the fragment buffer in the color buffer and apply dppass

Note that with sfail and dpfail, the color buffer is left unmodified.

If you draw a volumetric shape, keep in mind that OpenGL may draw the same stencil pixel several times, depending on the order of overlapping triangles and/or the availability of the depth buffer; in such cases, your action will be applied several times.

SampleEdit

Rotating cube, clipped

In our example, we'll draw a moving circle, and the scene will be clipped inside that circle:

```/* main */
glutInitDisplayMode(GLUT_RGBA|GLUT_ALPHA|GLUT_DOUBLE|GLUT_DEPTH|GLUT_STENCIL);
```
```/* onDisplay */
glClear(GL_DEPTH_BUFFER_BIT);
glEnable(GL_STENCIL_TEST);
glStencilFunc(GL_NEVER, 1, 0xFF);
glStencilOp(GL_REPLACE, GL_KEEP, GL_KEEP);  // draw 1s on test fail (always)

// draw stencil pattern
draw_circle();

// draw where stencil's value is 0
glStencilFunc(GL_EQUAL, 0, 0xFF);
/* (nothing to draw) */
// draw only where stencil's value is 1
glStencilFunc(GL_EQUAL, 1, 0xFF);

draw_scene();

glDisable(GL_STENCIL_TEST);
```

`draw_scene` draws the color cube from tutorial 05, and `draw_circle` draws a circle in 2d mode. We won't discuss these functions because they are not related to the stencil feature, but you're invited to look at the source code (see at the link at the bottom of this page).

Note that there's no need to use a separate program to draw in the stencil buffer. In this case it makes sense, because we draw a 2D pattern, but it's perfectly possible to draw a 3D pattern in the same scene, as in the Mini-Portal tutorial.

Working on color, depth and/or stencil buffers specificallyEdit

In the sample, we see that we first worked exclusively on the stencil buffer to draw the stencil shape, then exclusively on the color buffer to use it as a mask.

When you want to modify the stencil buffer without modifying the color buffer and/or depth buffer, you can mask these buffers:

```  glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
```

Don't forget to set them back. You can save the previous values this way:

```  GLboolean save_color_mask[4];

/* Do something */

```

If you want to apply changes on the color buffer without modifying the stencil buffer, you can either:

• set `glStencilMask(0x00)` - this means nothing will be written in any condition
• set `glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP)` - which applies a no-op for all conditions

DebuggingEdit

It is difficult to understand what is wrong when working on the stencil buffer, because we have no direct way to see it.

If you use OpenGL non-ES, you can call `glReadPixels` to grab the whole stencil buffer and inspect it externally, but it's tedious.

Another method is to fill the screen using two triangles, and with a stencil test (for instance, to match only non-0 stencil pixels). You may need to enable/disable the color buffer when doing so.

Then swap the OpenGL color buffer and pause, so you can inspect the result visually:

```      glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
glClear(GL_COLOR_BUFFER_BIT);
glStencilFunc(GL_LEQUAL, 1, 0xFF);
fill_screen();
glutSwapBuffers();
cout << "swap" << endl;
sleep(1);
```

Your `fill_screen()` function will preferably use a dedicated program that will involve no MVP matrix, though you can reuse an existing program and pass `v=glm::mat4(1)` and `m=glm::inverse(projection)`.

PerformancesEdit

We saw there are three possible actions in `glStencilOp`:

• sfail
• dpfail
• dppass

It is true that it's counter-intuitive to express your action on the stencil buffer using the sfail action, because of the double-negation (instead of "do this when this condition is met", you have to write "don't do this when this condition is not met).

However, if you use dppass to express your action, keep in mind that OpenGL will check both the stencil and depth test, and will also compute the pixel value using the fragment shader. This is a terrible performance hit if you are only drawing a shape in the stencil buffer - because you don't need to call the fragment shader at all.

A suggestion is to first implement your drawing algorithm using dppass for clarity, and once it works, reverse the conditions in `glStencilFunc` and use the sfail. sfail will not try to evaluate the pixel fragment when the test fails, so it's faster.

Example:

• First prototype with:
```    glStencilFunc(GL_NOTEQUAL, 0, 0xFF);
glStencilOp(GL_KEEP, GL_KEEP, GL_INCR);  // +1 when current pixel != 0
```
• then optimize:
```    glStencilFunc(GL_EQUAL, 0, 0xFF);
glStencilOp(GL_INCR, GL_KEEP, GL_KEEP);  // +1 when !(current pixel == 0)
```

LimitationsEdit

The OpenGL 4.2 core profile specification mentions:

However, when both depth and stencil attachments are present, implementations are only required to support framebuffer objects where both attachments refer to the same image. [1]

This means that you probably cannot attach two renderbuffers to a single frame buffer.

In OpenGL ES 2.0 in particular, combined formats are not supported (`GL_FRAMEBUFFER_UNSUPPORTED`) in `glRenderbufferStorage` (only `DEPTH_COMPONENT16` and `STENCIL_INDEX8` are supported, typically not `DEPTH32F_STENCIL8` or `DEPTH24_STENCIL8`).

If you need to combine post-processing and stenciling, you'll need to avoid framebuffer objects and use the `glCopyTexSubImage2D` technique instead.

ReferencesEdit

1. "The OpenGL Graphics System: A Specification - Version 4.2 (Core Profile)". Khronos.org. 2011-08-22. Retrieved 2011-10-05.

< OpenGL Programming