Baked normal map partially inside out (Blender/Unity)
So apparently a thing that can happen is that the unwrap algorithm puts down some islands mirrored. A way to see this is to view the model with the grid texture and look for flipped letters/numbers. 

Hope that helps someone out there!