Take FFT of audio represented in PDM
In a PDM microphone, the sigma-delta convertor pushes the noise to frequency regions above the 0-20kHz spectrum. So if you'd take the FFT of the signal that comes directly from the PDM microphone, the straight zeros and ones, and only look at the bins that fall in the 0-20kHz region, you'd get the data that you need.
You can see this here, where a 16kHz sine wave has been converted to a PDM signal:

The noise floor on the left of the green 20kHz line is well below -70dB, but quickly increases to the right of it.
In the plot above, the PDM data rate is 768kHz. A common PDM microphone data rate will be more on the order of 3MHz. The problem with this is that you'll doing FFTs of a large amount of data and that you're throwing away a lot of unused data.
A moving average filter would indeed quickly reduce the number of data samples that go into the FFT. It will result in attenuation in the pass band that is too high for good audio quality, but it should be totally fine for an LED display.
I wrote about my own PDM microphone learnings in a series of blog posts.