Voltage drops to 3V when connecting USB to 5V Source
Sounds like you need a bigger solar panel or more light, and the voltage is dropping because not enough current is being supplied. Also, if the regulator is linear you are throwing away most of the power of the solar panel when dropping the voltage to 5V 

You need the diode after the regulator