Lack of bass with higher sampling rates
I'm not familiar with the parts, but some devices include built-in high-pass and low-pass filtering with characteristics that are all affected by master clock rate.  Doubling the clock rate will double the cutoff frequency of all the high-pass and low-pass filters.  In some cases, there may be some options to e.g. control whether cutoff frequency for the high-pass filter should be at clock/256, clock/512, or clock/1024, or clock/2048.  In other cases, the high-pass and low-pass filters may have a fixed frequency relationship.

I wouldn't expect a 1.5x change in the high-pass filter cutoff to turn a signal all the way from "big and boomy" to "wimpy", but without knowing the spectral content of the sound you're listening to and the behavior of the parts, it's hard to say for sure.