Does a boost converter lower the source current of a solar panel?
I'm confused -- if you have two 4V solar panels in series, your open-circuit solar cell voltage is going to be 8V, and you want to have a buck converter to get 5V for charging a phone. Hooking up 8V to that Amazon module will likely damage it.

If you have two 4V solar panels in parallel, now a boost converter would make sense to get 5V, and that module could potentially work for you.

Regardless, think about your problem in terms of power. A solar panel with 4V open-circuit voltage, and let's say your 50mA number is short-circuit current, we can back-of-the-envelope as a 200mW device. If you put these both in parallel, you have about 400mW (theoretical) you can deliver to your device, assuming perfect power transfer (which of course we cannot do).

Now, assuming that you are not doing any peak-power tracking and are applying the output of the cells straight into a boost converter to get 5V, assuming perfect everything (which again, is not true), your best case delivery to a phone is 5V @ 80mA...which is not very much. I doubt that's enough to even trickle charge a modern smartphone. If we assume that your energy conversion process is 80% efficient, that's even less transfer -- only 64mA delivered ((0.8 * 400mW) / 5V).

In the scenario where you have placed them in series, you would have the same amount of power available, but you would use a buck converter instead of a boost converter. 

So: 


Don't hook up the panels in series to a boost converter. 
You don't have enough power in either scheme to do what you want and charge a phone sufficiently (best case, you charge a phone that is off very, very slowly).