Layer rendering in Vulkan
I think you may be overcomplicating things with your idea of using the stencil. In 2D graphics layers, we usually do not need to use the depth buffer to resolve depth order between 2D elements (their depth order is determined by draw order instead). Therefore, we can reserve the nearest depth value for all the foreground elements, and the farthest depth value for all background elements.
This would work as follows, assuming depth 0.0 is the far plane and depth 1.0 is the near plane.

Render opaque foreground elements in front-to-back order, using depth test VK_COMPARE_OP_GREATER, and arranging in the vertex shader for all elements to be placed at depth 1.0.
Render opaque scene objects roughly front-to-back.
Render opaque background elements in front-to-back order, placing them at depth 0.0.
Render transparent background elements back-to-front, now using depth test VK_COMPARE_OP_GREATER_OR_EQUAL, disabling depth writes, and placing them at depth 0.0.
Render transparent scene objects back-to-front.
Render transparent foreground elements back-to-front, placing them at depth 1.0.

The switch from GREATER to GREATER_OR_EQUAL is important here. Since the opaque elements are going front-to-back, you want to exclude the &quot;equal&quot; case, since you want later elements to be occluded by earlier elements. Then, when coming back-to-front for the transparents, you want to include the &quot;equal&quot; so that later elements will correctly render on top of earlier ones in the foreground/background layers.
Edit: Now that I'm thinking about this more, this may not work since transparent elements in foreground/background won't be able to be occluded by opaque elements in the same layer. This could be addressed by pre-clipping the transparent element geometry, or if this is important to support on the GPU, it could be addressed by rendering opaque elements depth-only (no fragment shader) in the first passes and then rendering all elements, both opaque and transparent, back-to-front in the later passes.