Headphone output: DC Bias?
simulate this circuit &ndash; Schematic created using CircuitLab

I don't think horta's answer will help.  From the description, this happens every time the radio starts receiving rather than just when the radio is turned on.

DC IS what you want to get rid of.  A capacitor has a high resistance to low frequencies, and a coil (the speaker) has a low resistance to low frequencies.  

When the amp kicks in, you get a "step" in DC voltage from zero to something less than the battery voltage of the radio.  You need to block the DC.  The coil itself has a high resistance to the higher frequency components of the step - that's why it is a "thump" instead of a sharper "click."  

A capacitor in series with your speaker should minimize the "thump."

I would try at least 20uF.  You'll need an electrolytic capacitor, so check which way the speaker output is polarized and measure how high the DC Voltage is.  You'll need to observe the polarity when connecting the capacitor, and you'll need to get a capacitor rated for more DC voltage than you measure.

If you simulate the diagram, you will find that it reduces the "thump" to just a few millivolts.

I've added a simulation of an audio signal feeding the LC circuit in the diagramm.  The audio is 3 dB down at 300Hz, which is where I was shooting for.  There's a 40dB hump at 500Hz which I didn't expect.  Less than 20uF would push the hump up higher as well as the cutoff.  You'll probably want to go higher than 20uF.