title: How to draw concave shape using Stencil test on Metal
body: <p>This is the first time I'm trying to use Stencil Test but I have seen some examples using OpenGL and a few on Metal but focused on the Depth test instead. I understand the theory behind the Stencil test but I don't know how to set it up on Metal.</p>
  
  <p>I want to draw irregular shapes. For the sake of simplicity lets consider the following 2D polygon:</p>
  
  <p><img src="https://s2.postimg.org/twz5zwnk9/Full_Size_Render.jpg" alt="concave shape"></p>
  
  <p>I want the stencil to pass where the number of overlapping triangles is odd, so that I can reach something like this, where the white area is the area to be ignored:</p>
  
  <p><img src="https://s18.postimg.org/6mgonv9ft/Full_Size_Render_copy.png" alt="Area to be ignored"></p>
  
  <p>I'm doing the following steps in the exact order:</p>
  
  <p>Setting the depthStencilPixelFormat:</p>
  
  <pre><code>mtkView.depthStencilPixelFormat = .stencil8
  mtkView.clearStencil = .allZeros
  </code></pre>
  
  <p>Stencil attachment:</p>
  
  <pre><code>let textureDescriptor = MTLTextureDescriptor.texture2DDescriptor(pixelFormat: .stencil8, width: drawable.texture.width, height: drawable.texture.height, mipmapped: true)
  
  textureDescriptor.textureType = .type2D
  textureDescriptor.storageMode = .private
  textureDescriptor.usage = [.renderTarget, .shaderRead, .shaderWrite]
  mainPassStencilTexture = device.makeTexture(descriptor: textureDescriptor)
  
  let stencilAttachment = MTLRenderPassStencilAttachmentDescriptor()
  
  stencilAttachment.texture = mainPassStencilTexture
  stencilAttachment.clearStencil = 0
  stencilAttachment.loadAction = .clear
  stencilAttachment.storeAction = .store
  renderPassDescriptor.stencilAttachment = stencilAttachment
  </code></pre>
  
  <p>Stencil descriptor:</p>
  
  <pre><code>stencilDescriptor.depthCompareFunction = MTLCompareFunction.always
  stencilDescriptor.isDepthWriteEnabled = true
  
  stencilDescriptor.frontFaceStencil.stencilCompareFunction = MTLCompareFunction.equal
  stencilDescriptor.frontFaceStencil.stencilFailureOperation = MTLStencilOperation.keep
  stencilDescriptor.frontFaceStencil.depthFailureOperation = MTLStencilOperation.keep
  stencilDescriptor.frontFaceStencil.depthStencilPassOperation = MTLStencilOperation.invert
  
  stencilDescriptor.frontFaceStencil.readMask = 0x1
  stencilDescriptor.frontFaceStencil.writeMask = 0x1
  stencilDescriptor.backFaceStencil = nil
  
  depthStencilState =  device.makeDepthStencilState(descriptor: stencilDescriptor)
  </code></pre>
  
  <p>and lastly, Im setting the reference value and the stencil state in the main pass:</p>
  
  <pre><code>renderEncoder.setStencilReferenceValue(0x1)
  renderEncoder.setDepthStencilState(self.depthStencilState)
  </code></pre>
  
  <p>Am I missing something because the result I got is just like there is no stencil at all. I can see some differences when changing the settings of the depth test but nothing happens when changing the settings of the stencil ...</p>
  
  <p>Any clue?</p>
  
  <p>Thank you in advance</p>