3x AA battery phone charger input voltage drop
how can I fix this problem?

Go back to high school physics.

Modern converters give you lots of flexibility in changing one voltage and current combination to another. However, they can't cheat physics and give a higher power out than in. Ultimately, some part of the three links in your chain can't handle the power that the phone requires.  The three links are the batteries, the 5 V supply, and the phone charger.

Since you measured at the output of the 5 V supply and found the voltage collapsed, you know that it's either the batteries or the 5 V supply that can't handle the power.