Difference Blending Option
You are interested in blending mode "Difference"

You can use it in "Blending mode" of layer, or on masks.

UPD: Adding screenshot, as asked


Red — Mask Blending

Blue — Layer Blending