Where is my audio noise coming from?
My best guess is this: -

The DC-DC converter is producing common mode noise on its output power feed to the RaPi. That CM noise will be referenced to 0 volts on the amplifier power port and, the amplifier is unable to cope with that CM noise applied to its differential inputs. To prove this, if you grab hold of an audio transformer and feed the RaPi signal via it to the left or right channel of the amplifier, the noise should reduce significantly.
If this doesn't work, then at least you've ruled out one suspicious mechanism that might have been the problem.