Filtering a signal to have a constant exponential moving average
I'll draw on my (limited) experience in audio mixing and mastering. Speech recorded through a microphone requires the same statistical properties (constant moving average and constant variance), and two straightforward processing stages ensure this.

An "exponential moving average" is another word for a 1-pole low-pass filter. This moving average can be brought close to zero by passing the signal through any high-pass filter, such as a "DC killer" or "remove subsonic rumble" filter that an audio processing package such as Audacity or Audition may provide. The stronger the high-pass characteristic, the closer the moving average will stick to the center line.

Second, you want to make the variance constant. That's a job for a level compressor. Find the RMS (square root of variance) of the signal over the next few milliseconds, and adjust the gain (amplification) of the signal over that period to bring the variance up or down to the variance you want. Audio processing packages provide this as well.

You mention numerical stability. The standard DSP literature describes ways to implement a robust high-pass filter. For instance, FIR filters tend to be very stable.