5v relay to be driven from a limited supply of 20mA which requires 200mA of latching current
I miss an important parameter: "must-activate voltage". This will (and must!) be less than the nominal 5 V, as it will determine how much the voltage may droop during the 50 ms to activate.  

You won't need a charge pump; a large capacitor will probably do. If you switch on the relay the capacitor voltage will droop exponentially, and the must activate voltage will be reached later for larger capacitors.  

Let's assume must-activate voltage is 4 V. Then

\$ 4 V = e^{\left(\dfrac{-t}{R C}\right)} \cdot 5 V \$

If \$t\$ = 50 mS, and \$R\$ = 24 &Omega; we find \$C\$ by rearranging the equation:

\$ C = \dfrac{- 50 ms}{24 \Omega \cdot ln\left(\dfrac{4 V}{5 V}\right)} = 10000 \mu F \$