A regulator drops on high current
When the LM1117 is providing 100 mA load current, the maximum drop-out voltage can be as high as 1.2 volts. This means that the input voltage to the regulator needs to be at least 4.5 volts (all the time) or you will see a lowering of the output voltage.

If the 5 volts is in fact (say) 4.7 volts and drops just a little bit under load this could explain the problem. Alternatively, the 5 volts may look like a solid voltage but whatever external current it is providing may occur in regular bursts and there may be an instantaneous voltage that drops below 4.5 volts that your meter cannot see.

You may need to use an oscilloscope to see this.

Check the input voltage to the LM1117 rather than the output voltage from whatever generates the 5 volts because there could be additional volt drop you haven't factored-in.