pam8403 using low wattage speaker
Besides distorting rather badly at full volume (10% distortion), four ohm speakers thermally stress this little chip.
Consider that power transfer efficiency is about 82% with 4-ohm load. Data sheet doesn't make clear if this is one channel operating or both channels operating. When 3W is delivered to a 4-ohm speaker, the chip must dissipate 0.54W. In addition, quiescent power is 0.016A * 5v = .08W
Thermal resistance (junction-to-air) is 110 degrees per watt. That means the chip junction is about 60 degrees above ambient temperature. If both channels deliver three watts, the chip is over-stressed.

When chip operation degrades after awhile, thermal stress should come to mind as a possible cause. Two approaches might be taken:


add a heat sink to the chip to dissipate heat.
increase load resistance on each channel.




Edit:
0.5W speakers driven by an amplifier that can deliver 3W invites speaker destruction.