How do I composite a transparent texture over a masked Render Layer I tried to composite a transparent texture over a masked Render Layer, but the render layer mask doesn't detect the transparency of the texture.



The layer with the transparent texture has transparency, and the texture too, but the render layer mask doesn't detect the transparency.
Your "skydome" renderlayer is masked, and it shouldn't.

Remove the mask from the background renderlayer and the alpha over operation will produce the right result.

The skydome renderlayer should look fully opaque, without any transparency (your screenshot shows that it is masked by the objects of the foreground renderlayer).

Keep in mind that masks in the renderlayer panel are z-masks, created by geometry, and not affected by alpha transparency.