Step Down Regulator - Output voltage unstable/voltage drop with load
The input must not affect the output, it's a closed loop system, which means that in wide range of input voltage, it will regulate the output to what you configured. 

And since i haven't seen such unregulated DC/DC for a long time, i suggest to first understand how do you connect it and what are you measuring. Specifically, you have to measure on the module itself to ensure regulation. The wires must be thick enough to hold charge current. Of course, in the end the measurement should be taken near the phone. If the cable is poor, you may need to raise the DC/DC output voltage.