Solid State Relay Capacitance Bad for Audio?
Using SSR or discrete FET switches for an audio attenuator is not the best idea when far better solutions exist. You should investigate the use of a digital pot for the attenuation core of the design. If you search at a site like http://www.mouser.com for digital pot you will find thousands of parts to pick from.

To eliminate the effects of capacitance on the wiper of the digital pot it is highly recommended to buffer the wiper of the pot directly into the high impedance input of an op-amp configured as a voltage follower. 

Use of a higher total resistance digital pot can help to reduce loading on the source driving circuit. 

To take care of the various types of source signals, that could be in a wide range of characteristics, I would recommend that you provide an input connector for each general voltage range of signal. Wire each of these input connectors to a suitable fixed input attenuator that scales the maximum input signal to be within range of the digital pot. Each scaled input would then be fed through a wafer style rotary input selector switch.

The selector switch output would then connect to the digital pot for further scaling based upon the actual signal level at the selected input. It would also be wise to use a very low capacitance diode clamp chip on the signal line going into the digital pot. This will help to protect the low voltage circuits (digital pot and voltage follower) if a user plugs a high voltage source signal into the incorrect input jack. The diode clamp device wants to one made with low forward voltage drop Schottky diodes and then connected into the GND and SUPPLY rails that define the operating range of the digital pot. 

Depending upon the nature of the source signal attenuators it may be necessary to place some series resistance in the signal line before it gets to the clamp so as to limit the maximum current flow that could happen for incorrectly connected input signal selections.