Where do the white lines come from in my composite? I'm having this issue for years now, and I'm not sure if I am doing something wrong on my end, or if what I would like to achieve - or expect to see - is even possible. My question is about compositing artifacts when using Blenders mask layers. Let me explain:



In the Blend file provided, there are two objects: Dragon A and B. Each of them is on a separate layer, rendered on a separate render layer, and masked with the other one. So A is masked out by B, and B is masked out by A. Like this:

scene setup



RenderLayers A and B



When I render, I write out two MultiLayer EXRs with 16bit depth. That Stuff, I feed into the following comp:



and it gives me this:



You can clearly see the white lines around the areas where the dragons mask each other.

If I comp in a weird fashion like this:



I get this:



It's not perfect, but still a lot better than the first one. What I am doing is kind of cutting off the bright alpha values, making a 0.75 alpha value equal 1.

My question is: Why do I see the white background shine through after all? Is it something which is fundamentally unchangeable and we have to live with? Am I doing something wrong on my side? Do you have an alternative solution?

Edit

The solution Ray Mairlot came up with solves the issue by only masking one of the two, the other one would exist un-masked underneath, and thus provides a perfect overlay. However, I'd like to still mask one against the other for specific reasons: I'm sending images to a web application, which layers them dynamically (a configurator in essence). There can be 30 or more image layers, each of them needs to be downloaded by the website visitor. If I could mask one part against the other, the individual file size would be smaller. Not really a necessary reason, but that's why the idea came up. The other reason is, if I mask the layers, the individual layers render faster. This can save me some 30% of the render time per configurator (which renders on average for 12 hours).
Those two images you're trying to composite together have "correlated" alphas. The Alpha over operation isn't the appropriate operation for this situation.

The solution is simple: you just have to add those images together, then add their alphas together and assign the resulting alpha to the result of the first addition.

Here's how:



Why alpha-over doesn't work

Alpha over is, like its name states, an "over" operation. Something is over something else, covering it. The alpha channel determines the occlusion of the background by the foreground.

When you inspect the alpha-over formula for two associated (premultiplied) images, you'll see it's just an addition, but before that addition the alpha channel of the foreground is inverted and multiplied on the background, casting a shadow on the pixels that the foreground occludes.

Associated Alpha Over Operation:

Foreground.RGB + ( ( 1.0 - Foreground.Alpha ) * Background.RGB )


Porter Duff Alpha Merge Operation Uncorrelated Geometry:

Alpha.A + Alpha.B - ( Alpha.A * Alpha.B )


Correlated Geometry:

Alpha.A + Alpha.B


When alphas are correlated, like in your example, there is no foreground-background situation. It's rather filling a hole in one image with the other image.

The hole and the image that will cover it fit perfectly, so there is no reason for casting a shadow on the background. There's already a hole in it.
If you use alpha over, the "shadow" of the foreground will be multiplied again on the antialias pixels that are already shadowed by the mask, causing a doubled value of transparency, and that's why you get those semi-transparent pixels where they should be completely opaque.

So, skipping the "over" part by just adding the two plates together works perfectly. In Blender you have to add alphas too and assign the result to the mix.