Solar Powered MintyBoost
Since the batteries are in series, the current running through the one goes through the other, you do not get 20,000 mAh, you still only have 10,000 but at double the voltage. If you charged them at 320mA, it would only take 31.25 hrs as the 320mA runs through them both at the same time.

What you need is a Buck converter. If you don't know or want to design one, Digikey sells them for about a $1. Digikey Buck Converter. That will convert the 4 - 4.5 V of the solar cell to 2.5 volts, but at much higher current. 2.5V is good as it should charge the batteries easily.