Some power across a relay even when open?
Are you driving load? Even a simple load such as a lamp will do for testing.
I expect then that you will find the voltage drops to zero when the relay is off.
The voltage you are seeing is probably caused by capacitance either in the relay and wiring or in the cabling to the relay. (or a snubber as suggested by @Transistor).
Your meter is sensitive enough on the AC voltage ranges to detect even the few microamps of current that flows through the capacitance. It is negligible compared to the current required by normal AC loads.
I often see this in house wiring, especially where the live and switched circuit run together in the same cable.
Solid State Relays (SSRs) require a load to operate correctly and can pass a few milliamps when in the off condition. Conventional mechanical relays only pass a a few microamps due to the proximity of the contacts and wiring.