LM2577 5V Boost Circuit does not boost voltage
Your compensation network is wrong.

As shown in the typical app in the datasheet, compensation should be done with a series RC block instead of a single resistor (R4 in your circuit).

Without proper compensation, the circuit will not be stabilized.