Max value of Y capacitor
First you have to know the leakage current limit for your class of equipment.  For IEC60950 IT equipment it is 250uA.  Then the leakage current will be given by: Cy = (Il/ ωV) – Ciw where Cy is the Y-capacitor value, Ciw is the interwinding capacitance of your transformer (specified or measured) ω is the line frequency and V is the line voltage.

Of course you need to build in margin for all the tolerances involved (check temperature variation as well) and you need to measure the leakage current to be sure the calculation is correct.