Amplifying PWM Audio with an OpAmp
Ah, specmanship!  Look closely at the LM358 data sheet section:



The manufacturer gets to the "typical" 20 ma sink current when there is a one-volt difference between Vin+ and Vin- (not usually typical for an op amp), and with a 15-volt supply.  The sink current values are different at different voltages, dropping down to microamps as you approach the rail.  So the signal will be badly distorted.  The fact that you can recognize the sounds coming out is a testament to the effectiveness of the human brain and ear.

What you need is a push-pull driver circuit that goes all the way to both rails.  If you don't want to build one from discrete components, you might try a gate driver.