Simple 24V to 5V converter circuit (Almost complete)
Taken straight from the datasheet:

8.2.1.2.1 Input Capacitor
An input capacitor is required if the regulator is placed more than 3 inches from the power supply filter. A 0.33-µF
capacitor on the input is suitable for most applications.
8.2.1.2.2 Output Capacitor
TI recommends a minimum load capacitance of 0.01 µF to limit high-frequency noise.

Really for something like this, I don't think it will make a massive amount of difference, however, maybe 100uF is a bit excessive. It'll still work, but it does seem a tad large! Maybe you used those capacitors because they are the ones you happened to have to hand at the time? For a circuit using only 50mA, if it is working and continues to work, then I don't think it would be a problem.