Can anyone explain this strange 3-way wiring? Trying to install a motion sensor switch
3-way switches can seem a little bit like magic, I suppose. ;-)

What happens is that power comes into one switch on the "common," travels to the other switch over one of the "travelers," depending on the position of the switch, and then depending on the position of the switch on the far end, power will travel out of that common to the light(s). If the 14-3 or 12-3 wire only runs between the switches (not through the light fixture), then the "common" in that cable will be the hot conductor coming back from the far switch, in which case it should be connected to one of the black wires, which supplies power to the light. But if the red/black/white cable runs through the light fixture, it might be wired up differently.

In your case, it looks like power is probably coming into the box with the multiple "romex" cables entering it. One of the incoming 14-2 or 12-2 cables brings power in, and the other takes power to the light. That means the white wires in those black/white cables are neutrals, while none of the 3 wires connected to the switches are neutrals.

Your occupancy switch requires a neutral connection in order to function.