ICL7660 - Cannot provide stable negative voltage for audio amplifier
As stated in the comments, ICL7660 and ICL7660A series switched capacitor voltage inverteres shouldn't be used for current requirements more than 20mA because of high output impedance (around 100 Ohms). You can check this from the datasheet. The most common usage for this type of inverters is providing a supply for opamp-based circuits and RS-232 circuits. You might know that these circuits draw only a few mA of current.

If required voltage swing is not higher than ±2V then you can still use the single +5V supply with virtual grounding.

If you really need a negative supply with higher current then you can build a classical voltage inverter with MC34063 or LM2776 or equivalent: