Compositing Transparency / Layer Masking Problem I have a scene with a number of Render Layers. In the screenshot I am only showing two to focus on the problem (called "main ground" and "bushes").

Each render layer has all the other layers set as a mask (I am not sure if this is correct, but it was the only way I could see to put them back together again as one image without visible parts that should have been obscured).

I am using Alpha Over nodes, but oddly I am getting strange halos around some of the layers. You can see this on the left of the image where there is a strange transparent halo around the bush, cutting through to the sky background.

It seems like Blender is cutting the wrong sized mask "hole" in the main ground layer for the bush layer, but I don't understand why. I suspect I somehow need to tell Blender to better calculate the bush layer mask, but I don't know how.
What you are experiencing is the result of antialiasing, more explicitly two overlapping masks with antialiasing, so the transparency of the upper mask is reveals some of the antialiasing of the underlaying mask.

You could get rid of anitaliasing it by switching off the AA samples using branched path tracing. That will give you perfect edges for the mask but it will affect the overall quality by having an aliased image that will look blocky...

 

A simpler and higher quality solution is not using a layer mask on the render layer for the grass.



Keep the layer mask on the layer for the bushes only.