What is the purpose of the capacitor in this transformer circuit?
In theory, no connection to the center tap of the primary is actually needed.  The "balanced" signal is differential, so only the voltage difference between the two lines matters.  The transformer inherently works as a differential front end, since only voltage across its input coil causes and magnetic field that is the coupled to the output coil.

However, you don't want the common mode voltage of the balanced signal to go too far from ground.  If totally insulated, it could pick up static electricity and build up to 1000s of volts, eventually arcing over something and causing a crackle sound in the audio.  The resistor provides a relatively weak path to ground, but it's plenty low enough to prevent static buildup.  The capacitor does the same thing but more so at higher frequencies.  At too high common mode frequencies, the inevitable capacitive coupling in the transformer will cause some of the common mode signal on the input to show up as signal on the output.  The capacitor provides a more serious connection to ground for AC, while the resistor only a weak connection for DC to avoid ground loops.

Note that since this connection to ground is halfway thru the primary of the transformer, the magnetic field caused by the common mode voltage across one half the winding is offset by the magnetic field caused by the common mode voltage across the other half of the winding.

Personally, 22 &micro;F seems really high to me.  I would want to let the common mode float more easily at audio frequencies.  I would have been more concerned about squashing the pickup of radio station, while letting pickup from the power line be a common mode signal that the transformer should do a nice job rejecting.  The two halves of the primary aren't going to balance exactly, so you don't want to push more common mode signal thru there than you have to.