RTC Battery Voltage Error; Non-Ideal Op Amp Error?
the problem is not the op-amp per se, but engineer's assumption of how the input and output pins of ICs behave when power is not applied to that IC.

Many engineers do not know that there are either ESD protection diodes or parasitic diodes (or both) at input and outputs of ICs. Datasheets do not make this obvious. Take a look at this article.

Thus, when VCC is removed from your ADC and opamp, there is a diode path from VBATT to the VCC net due to the ESD diode inside the opamp IC. Nearly all ICs will have this architecture. Go ahead and measure VCC with the connector removed and you will find it is about 1 diode drop below the VBATT voltage. This means your coin cell is powering the ADC and opamp when the connector is removed! That is why the coin cell voltage is drooping.

The simplest but a bit cheesy solution is to put say 100k resistance at the opamp IN+. This will reduce the current pushed into ADC and opamp. It would be about (2.8V - 0.6V) / 100k = 20 microAmps. It would increase the ADC error by whatever the input bias current of your opamp is times the 100k series resistance, and this would still be extra current draining your coin cell.

A better solution would be to use a small MOSFET to disconnect the opamp input whenever VCC is disconnected. But this would add 2 transistors and some resistors. I can sketch up a circuit if you want.

There really is not a simple, easy solution (other than the series resistor). It is actually quite a design challenge when you do this type of "hot swap" circuit, where you have to switch power to your circuit to different supplies on the fly or you have to handle non zero inputs even when power to your circuit is removed. 

Hope that helps, -Vince