Charging mobile with solar panel Issue
In short: To signal the usb device that you are attempting to charge, that it can charge at a current higher than 500mA ,if if wants to, you need to short D+ and D- of the USB connection to that device.
I have read through the comments once again and it seems to me as though this is a misunderstanding of your expected operation of what you have put together.

You are putting a load on the output of your regulator which you do not have control over, so it will load the circuit as its internal circuitry defines it to.

Phone charging, especially a smart phone, now a days, has communication protocols in place for the phone to &quot;negotiate&quot; with the charger what it is capable of receiving and what the charger is capable of delivering.
Charging o phone via a constant voltage source, that is, the output of your step down regulator, will result in a load which is probably around 500mA, depending on the phone and the state of charge of its cell, it may attempt to pull more current but I'm guessing that since most old USB outputs were limited to 500mA max current, the phones' firmware may have defined 500mA as a limit to make sure that the USB hosts port does not disable the output as a protective measure because a device connected is attempting to sink more than 500mA



From a consumption perspective : I would say, 5V, 400mA , i.e. 2W load is what I would expect from such a circuit that does not negotiate via any charging protocols to the phone.

A PV Panel is not a battery, it resembles more a current source, but not really a constant one..
Here is a quick and dirty diagram of what your panel would be like. Note that it is not to scale but to bring a point across.
In Red, the power output vs Voltage
In Orange/yellow the current vs voltage

Assuming that your regulator is operating at 80% efficiency, then for 2W output, this means that it is consuming 2.5W at its input.

There are two operational points of the solar panel for 2.5W delivery, more or less around the Vpv = 5V and around Vpv = 20V.

since you normally start at open circuit voltage and then the regulator starts pulling current you will be at around those Vpv = 20V and at 2.5W you will be consuming 125mA.



So far everything is as expected. As you say, the regulator is not hot, Panel voltage at 20V.. seems fine. you should check the current coming from the PV Panel it will probably be at 125mA or so, not 560mA or anywhere near the MPP since your load is not requesting to go there.
Just because a PV Panel has a maximum power available does not mean that you will operate at that point.

Signalling the Connected Device that you can provide more than 500mA @ 5V
It appears as though part of the usb Power delivery standard is; if you short (resistance below 200 Ohms) between the D+ and D- pins of the connection to the device you are attempting to charge, you are telling the device to pull more than 500mA
5V out ---- Vbus
    +----- D+
    |
    +----- D-
Gnd ------- Gnd

I have verified this on two off the shelf chargers, one of them 5V 1A capable and the othe 5V 2A capable.
My measurements were as follows:
           | 5V1A charger  |  5V2A Charger
---------------------------------------------
V+ to D+   | Open Circuit  |  Open Circuit  
V+ to D-   | Open Circuit  |  Open Circuit  
GND to D+  | Open Circuit  |  Open Circuit  
GND to D-  | Open Circuit  |  Open Circuit  
D+ to D-   |   0.27 Ohms   |   0.27 Ohms

So, to extend your operation up to 5V 1A you would short the D+ D- connections on the USB cable you are connecting to the output of your regulator.
This would be a power delivery of 5W, assuming 80% efficiency of your regulator that would be 6.25W power in from the panel. Maybe Vpv = 20V or 19V or even 18V depending on the environmental conditions and 300 to 350mA current from the PV Panel.
As you can see from the calculations, even at rated output of the device 5V, 1A, i.e. 5W you will not be consuming all the available power from the PV panel. And there is no problem with that.
Here is a question/answer which discusses the Data pins and USB charging, here is another,