Adding encryption to a ham radio
Digital audio data that is transmitted wirelessly is usually encoded in some efficient format. It sounds like you're talking about broadcasting raw PCM samples that are encrypted before transmission. That's certainly unusual. 

Unless you are already following some kind of standard that I'm not aware of, you're probably best suited by packing your audio data into the payload some higher-level protocol that includes synchronization data to recover framing, the encryption that you desire, and most likely error correction. Your approach of just encrypting raw PCM samples with no error correction has the unpleasant side effect that a single bit error in the encrypted data will turn your entire block of samples into garbage.