Low voltage 2 channel stereo analog switch
Your options basically fall into two broad catagories:  Some sort of semiconductor device, like an opamp or analog CMOS switch, or some sort of mechanical switch (a.k.a. relay).

The problem with using a semiconductor device is that you really want your power rail to be higher than the audio.  For a headphone that would put your power rail somewhere in the +/- 5 to 12v range.  

A latching relay would be nice, and they are available.  I doubt that you can find one that works at 1.2v, but it would only consume power when actually switching.  A DC/DC converter might be required, but it only has to be on when switching the relay.  Plus, it is very hard to beat the audio quality of what's effectively a piece of wire.

Of course, let me ask the obvious question:  What's the problem with a simple switch?  Zero power consumption.  No power rail issues.  Super reliable.  Great sound quality.  If a simple switch will work then this is what you want to use.