Downsides to terminating a line level audio signal into an amplifier without an output resistor
The PCM5122 datasheet says it supports both DC and AC coupled loads, if the load is within specification. Minimum load resistance is 1k and typical is 10k, but there is no maximum resistance so 18k AC coupled into inverting PAM8406 input should be fine. But when the audio output to PAM8406 is taken directly from the chip instead of taking it after the output filter, there will be more high frequency modulator noise present, and in general this should not be fed into the next stage. But if this fixes your issue it might be fine. My only advice is that carefully design the best balance between digital signal amplitude, digital gain, analog output amplitude and analog gain, so that digital domain and analog domain do not clip, and so that least analog gain is used not to amplify noise. And as I've used that chip before, it was hard to make pop-free (they even removed that from datasheet because it wasn't pop free), everytime it gets stable clocks after a pause, there is a pop on analog output which you don't want to amplify.