Takes days to charge phone from 10W solar panel
First make sure everything is working as intended.  Point the solar panel at full sun and see what kind of current you can get at 5 V.  Try a 5 &Omega; power resistor.  That should draw 1 A, which takes 5 W of power at 5 V.  You should be able to get at least that according to your specs.

If that works, it may be that the phone is expecting a "smart" charger.  It uses the USB charging protocol to communicate with the charger to find how much current it can draw.  If it can't talk to the charger, the phone falls back to a rather low current.

Measure the current with the phone connected, first making sure the output really is at 5 V.