Honouring Depth Test with Stencil Test I'm using OpenGL's stencil test to carve out a semicircular hole in a transition frustum with a cylinder oriented perpendicularly. However as one can see below, while this objective is achieved, the cylinder is still visible through the frustum which shouldn't be the case.




I'm using the following code fragment that I found at
https://en.wikibooks.org/wiki/OpenGL_Programming/Mini-Portal

glClear(GL_DEPTH_BUFFER_BIT);
glEnable(GL_STENCIL_TEST);
glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
glDepthMask(GL_FALSE);
glStencilFunc(GL_NEVER, 1, 0xFF);
glStencilOp(GL_REPLACE, GL_KEEP, GL_KEEP);  // draw 1s on test fail (always)

// draw stencil pattern
glStencilMask(0xFF);
glClear(GL_STENCIL_BUFFER_BIT);  // needs mask=0xFF
DRAW_CYLINDER ();

glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
glDepthMask(GL_TRUE);
glStencilMask(0x00);
// draw where stencil's value is 0
glStencilFunc(GL_EQUAL, 0, 0xFF);
/* (nothing to draw) */
// draw only where stencil's value is 1
glStencilFunc(GL_EQUAL, 1, 0xFF);

DRAW_FRUSTUM();

glDisable(GL_STENCIL_TEST);


Moreover I've found that the results are insensitive to the parameters of the glStencilOp  and DepthMask commands. 

I seem to vaguely remember from several years ag that a multi-pass rendering may be required. If so, could someone point me to how to accomplish this with the proper set of command parameters?

thanks in advance.



This is an example of what I'm expecting - The cylinder should cut a hole through the frustum. I've rendered it here by trimming the black cylinder within the frustum so that it just just out past the frustum surface.
When cylinder and frustum intersect the hole will be a view into the internals of the cylinder. The hole is where external of the cylinder occlude frustum and internal of the cylinder does not occlude the frustum.

Knowing this you should simply do the following to find the surface (or rather volume) made from the hole.

Render frustum without color and set depth accordingly. Set a stencil bit where the cylinder (external) occludes the frustum, without setting new depth. Clear the previously set bit where the internal of the cylinder occludes the frustum, without setting new depth. 

Alternate between internal and external using CullFace.

The stencil mask you now have is the surface where the frustum hole is. Render the frustum (in color) masking it with the stencil. Make sure to use LEQUALS depth test to allow the frustum to render on top of itself. Voila. Note that the depth buffer holds the value of the unmasked frustum so If you need to render something within the hole you would have to render the whole scene again and mask it to only render in the hole unless depth test passes against the frustum. Left as student exercise.