What is a stencil buffer? Wikipedia states that a stencil buffer is some arbitrary buffer a shader can use.

However, it hints that it's used for clipping, or otherwise "tightly binding" the depth and pixel buffers, slightly contradicting itself.

What does the stencil buffer really do, and how is it practically used in modern applications?
The stencil buffer definition by Wikipedia is indeed not great, it focuses too much on the details of modern implementations (OpenGL). I find the disambiguated version easier to understand:


  A stencil is a template used to draw or paint identical letters, symbols, shapes, or patterns every time it is used. The design produced by such a template is also called a stencil.


That's what stencil meant before Computer Graphics. If you type stencil on Google Images, this is one of the first results:



As you can see, it is simply a mask or pattern that can be used to "paint" the negative of the pattern onto something.

The stencil buffer works in the exact same way. One can fill the stencil buffer with a selected pattern by doing a stencil render pass, then set the appropriate stencil function which will define how the pattern is to be interpreted on subsequent drawings, then render the final scene. Pixels that fall into the rejected areas of the stencil mask, according to the compare function, are not drawn.



When it comes to implementing the stencil buffer, sometimes it is indeed coupled with the depth buffer. Most graphics hardware uses a 1 byte (8 bits) stencil buffer, which is enough for most applications. Depth buffers are usually implemented using 3 bytes (24 bits), which again is normally enough for most kinds of 3D rendering. So it is only logical to pack the 8 bits of the stencil buffer with the other 24 of the depth buffer, making it possible to store each depth + stencil pixel into a 32 bit integer. That's what Wikipedia meant by:


  The depth buffer and stencil buffer often share the same area in the RAM of the graphics hardware.




One application in which the stencil buffer used to be king was for shadow rendering, in a technique called shadow volumes, or sometimes also appropriately called stencil shadows. This was a very clever use of the buffer, but nowadays most of the rendering field seems to have shifted towards depth-based shadow maps.