Unkown distortion in low power push-pull amplifier
I suspect your problem is crossover distortion, but coming from the op amp rather than the output transistors. The LM358 datasheet says:-


  To reduce the power supply current drain, the amplifiers have a class
  A output stage for small signal levels which converts to class B in a
  large signal mode... 
  
  For ac applications, where the load is capacitively coupled to the
  output of the amplifier, a resistor should be used, from the output of
  the amplifier to ground to increase the class A bias current and
  prevent crossover distortion.


I built your circuit and it exhibited crossover distortion approximately 0.2V below the signal center line, which explains why it suddenly appears as the volume is increased. I was able to move the distortion to the top or bottom of the waveform (where it is only barely noticeable at maximum volume) by connecting a 1k&ohm; resistor from the op amp output to Vcc or Ground. 

Biasing the op amp into class A reduces peak drive voltage in one direction which causes the amplifier to clip asymmetrically, but it is still capable of producing over 2V peak undistorted output into 32&ohm;.