MAX1771 MAXIM 4.2V TO 5V
For your application diagram 2C in the data sheet with the feedback resistors changed to give 5V output would be better. Figure 5 is for where the input voltage can either be lower or higher than the output voltage.  If you wish to get 5V from l LiIon cell then the input voltage will never be higher than 4.2V so the simpler circuit will work better.

We don't see the oscilloscope capture so we cannot comment on the problem you are having.

You do need to use good decoupling on the input and output (e.g C4 on fig 2c).  Normal aluminium electrolytics often have too high a series resistance to work well.  Tantalums or ceramics are good.

You also need very good low impedance grounding or you will get voltages across the ground inductance and resistance - a plug-in breadboard will probably not work acceptably.

It looks like the ESR of the output capacitor is too high.  What type of capacitor is it? What is its ESR?

As I mentioned above you need a low ESR type - ceramic, tantalum or aluminum polymer.  A normal aluminum electrolytic is often not good enough.

When Q1 is turned off the current from L1 is transferred into the output capacitor - any ESR will cause a voltage spike.  The gradual decline for each cycle is the ripple, you can reduce that by increasing the output capacitor.