Masking Radio Waves in After Effects
Yeah, that's a weird thing with the radio waves effect. It seems to be applied after the layer masks, and thus ignores the masks for the layer. This happens for a few effects, particularly ones that generate alpha. It might have something to do with that effect dating back at least to the Bronze Age—you'll have noticed how dog slow it is.

The only way to get the masks to do what you want is to precomp the layer with the effect on it (cut the mask off the layer, then hit cnd / ctrlshiftc and choose move all properties into new composition), and then paste the mask on to the precomp in your original composition.