Leakage from MOSFET Switch To Ground
When you switch SW2 to GND with C1 fully charged, the voltage applied to the drain of M1 is about -5V. (capacitors "dislikes" voltage changes across them)

Because the voltage between M1 and M2 is bigger than that, D1 is conducting.

Thus you have something around, let say, -4V at the source of M2.

The gate voltage of M2 is 0V.

Thus you have a Vgs of ~4V at M2. This is probably above it's threshold and M2 is conducting and thus discharging your cap.

This is just intuitive reasoning, you may do some calculus to get the correct figures.