Single supply op-amp voltage follower - negative clipping with AC-coupled load
You'll never get am LM324 to do what you want so why not bite the bullet and use an LM386. It's minimum supply is 4 volts and at 5 volts can produce 4 volts peak to peak. It's intended for audio applications as well and can drive a 4 ohm load (or greater): -



Read the data sheet if you want more details and other application circuits.