Wiring a 3.5mm Audio out jack where Speakers were previously
You are assuming the speakers are driven with single phase and ground. That may be a BAD assumption. The speakers may be driven in push-pull mode.

See this cross post.

Try connecting ONLY the L, and R and find a different ground.