Grounding apparently causes volume increase in speaker
As said in the comments, it was a mechanical problem in the potentiometer. Melting the solder on the pins and squeezing the pot into the circuitboard as the solder resolidfied fixed the problem.