Time to charge a capacitor with a power limiting supply and ideally no resistance
With the simplifying assumptions that:


the solar cell is an ideal power source, which produce a constant power at any combination of voltage and current
there are no other losses


then the calculation is quite simple. The energy, voltage, and capacitance of a capacitor are related by:

$$ E = \frac{1}{2}CV^2 $$

Using this, you can calculate the energy already stored in the capacitor at the initial voltage, and the final energy required at your target voltage. The difference is the total work required, and the time required to do that much work is the work divided by the power of your charger.

$$ E_0 = \frac{1}{2}C{V_0}^2 \\
E_{end} = \frac{1}{2}C{V_{end}}^2 \\
\Delta E = \frac{1}{2}C{V_{end}}^2 - \frac{1}{2}C{V_0}^2 \\
t = \frac{\frac{1}{2}C{V_{end}}^2 - \frac{1}{2}C{V_0}^2}{P} \\
t = \frac{\frac{1}{2}C ({V_{end}}^2 - {V_0}^2)}{P}
$$

Given


\$P=5mW\$
\$C=1F\$
\$V_0=1V\$
\$V_{end}=5V\$


then:

$$ t = \frac{\frac{1}{2}1F ((5V)^2 - (1V)^2)}{5mW} $$

$$ t = \frac{\frac{1}{2}1F (25 - 1)V^2}{5mW} $$

\$F=J/V^2\$ and \$W=J/s\$ by definition of the farad and watt so:

$$ \require{cancel} \begin{align}
t &amp;= \frac{\frac{1}{2}1\cancel{J}/\cancel{V^2} (25 - 1) \cancel{V^2}}{0.005\cancel{J}/s} \\
&amp;= \frac{\frac{1}{2}1 (24)}{0.005/s} \\
&amp;= \frac{12s}{0.005} \\
&amp;= 2400s
\end{align}
$$