Why is the discharge longer
There's something odd with your readings and others have addressed the effect of the diode. There are a couple of things that might be worth reviewing.

Figure 1. Capacitor discharge voltage curve (top) and capacitor current (bottom). The negative current is because current is leaving the capacitor. Image source: Electronics-Tutorials.
The concept of time constant is useful in RC or RL circuits. In the case of the RC discharge it is the time taken to discharge by 63% from an initial value and is assigned the Greek letter tau, τ, and τ = RC.
There are a few values worth remembering:

The capacitor will discharge by 63% after 1τ.
The capacitor will discharge by 95% after 3τ.
The capacitor will discharge by 99% after 5τ.
The capacitor will never completely discharge! (In reality it will get close enough to zero that you won't be able to measure it anymore.)

For your tests and given that you have a diode in there I would suggest that you do the following:

Measure the battery voltage and calculate 63% of that.
Connect your test circuit and time how long it takes to charge to 63%.
Calculate your τ and see if it matches. (Don't forget to allow for capacitor tolerance.)
Let it charge fully - 10τ at least.
Calculate 37% of the fully charged voltage and time how long it takes to discharge to that value.

The two experiments should give fairly close times but the 0.7 V drop of the diode will affect the charge time. If you want you can modify the charge measurement to be 63% of Vbat - 0.7 and see if that improves the comparison.