Charging a capacitor with low current
Take this datasheet as an example.

The leakage current used in their example is 2&mu;A, though I cannot find any real tabular data. This is because for most of these types of caps it is in the micro ampere range. (Although from unknown sources, I would also not be surprised with dozens of &mu;A or more)

That means your source needs to supply only 5&mu;A to be sure to eventually charge it.

What is a much bigger problem/risk, with these kinds of capacitors you can also see in the datasheet: They do not like high currents!

If you take more than 1 or 2mA out of one the life will be very significantly reduced and if you charge it directly with a voltage source without a series resistor it will also not last long, due to the initial peak currents when charging.

The datasheet shows you how long it will take to charge for a 30 Ohm and a 100 Ohm resistor for 5.5V and 1F.