Audio amplifier provides the capability to drive 660Ω loads directly without buffering
The output of the PGA2311 is a voltage source that, if powered with a +/- 5 volt supply, has an output range of -3.75 volts to +3.75 volts. The output can drive up to 50 mA of current.

This means that if you connect a 100 ohm load, your worst case current will be 37.5 mA, well within the rated output current. You should perform a thermal dissipation calculation, however, to make certain you do not require a heatsink since the device power dissipation has gone up by a factor of 6.6 from their reference design.

The device is designed for a 660 ohm load and unfortunately they do not specify the effect on THD based on load impedance. You can call an application engineer at TI if you are concerned about this parameter.