How to improve I2C RTC accuracy
3 min per day is:

$$ 24\cdot60= 1440min$$

$$ \frac {3}{1440}= 2083ppm$$

Your Xtal is more than 2000ppm off its nominal frequency. Even here there is an issue. But 3 hours per day is more than 4% ! There is a design issue here. For sure.

The Xtal frequency depends on the load capacitance and this is the common root cause of a wrong frequency.

Are you sure you computed your C1 and C2 correctly? The formula is:

$$ C_L = \frac {C_1 C_2}{C_1+C_2}+C_{stray} $$

With


CL = the load capacitance given by the manufacturer
C1 and C2 = The added capacitors on the xtal pins
Cstray = the stray capacitance of the tracks, pins etc...


If you have access to a spectrum analyzer, you may measure the frequency of the xtal using a near field probe (don't touch the xtal tracks with a probe, it changes the capacitance and thus the frequency) and adjust the load capacitors to get the correct frequency.

EDIT: 

In the PCF8583 datasheet of NXP: "Chapter 11.1 Quartz frequency adjustment". It is explained here that you should add an external capacitor to your Xtal to fine tune the frequency.