Audio Visualizer (not working properly)
It sounds like a dry joint; the fact that the LED matrix seems to work 'better' when you're fiddling with the microphone board makes me think that the problem lies entirely with that little board.

Try adding more solder; the more the merrier.