Unstable Lower And Upper Voltage Charger Wall Adapter For Charging Battery
The fluctuations are called "ripple" and the amount that you described look normal to me. 

According to Battery Charging Specification v1.2 the allowed bus voltage tolerance is 5% in DCP mode (the mode that most wall chargers have). Both chargers' outputs are in that range. So there shouldn't be any problems.

Your concern should be the max output current of the charger and the charge current need of the device to be charged. In any ways, since most wall chargers provide at least 500mA, there shouldn't be any problems.