Why are 3.5mm audio jacks unreliable?
I do not experience any unreliable connections with 3.5mm jacks, unless they are of a very cheap and flimsy quality.

(Maybe post a picture the jacks you experience a problem with)

As corrected by ‘Justme’ in the comments. You will only hear the left or right panned sound if the jack is not correctly inserted leaving the ground (sleeve) connection floating.