Ceramic capacitor 4kHz sound
Check whether during "PWM on" condition and with sound emission the LDO is stable, i.e. it does
not go oscillating at output. If it oscillates, that might be the root cause.
Try to use another capacitors in similar packaging, for example tantalum
ones. But first check whether your LDO allows that.
Which dielectric your capacitors are made of? Probably swapping X5R
to X7R, or going for other capacitor manufacturers
might help. Also try to select capacitor series (at the specific manufacturer) that are less susceptible   to electrostriction AKA microphone effect.