Lithium button-cell battery discharging really quickly
I would suspect the diode.  Try disconnecting the 3.3 volts from your buck regulator and see if your problem goes away.  If D1 is shorted, the battery would be heavily loaded.

You should put a resistor in series with the battery to limit the instantaneous current when you close the switch.  The BAT diodes only handle small currents.  10K would be about right.  As a bonus, you can then use the resistor to measure the current from the battery in case your VDDBU input is damaged, which is the less likely possibility.

Failing those two, it would have to be leaky caps, and you can remove them one at a time and use the resistor to help you figure it out.  But you need the resistor in any case.