Unity Gain Op-amp Buffer Noise
I don't think you chose the wrong opamp, at least not based on what you are showing here. It looks like power supply noise to me based on the obvious correlation between supply and output. Also the noise is periodic at about 15-20ms which would match 50/60hz.

Opamps have a specification power supply rejection ratio, which says how well the opamp can isolate the output from the power supply. Looking at the datasheet for this opamp, PSRR is very good, about 140dB, but that is at DC and PSRR is frequency dependent. 

Looking at your second picture, I see about 1V p-p noise on the power rail, and 20mV on the output (not 200mV - wrong probe setting?).  You are seeing maybe 35dB PSRR in this configuration, from 1V to 20mV. Looking at Fig 12 in the datasheet (PSRR vs Frequency) this ratio happens around 10-100kHz. It wouldn't be surprising if you have that sort of noise on your power rail, switching converters typically run 50khz upwards.

You should probably add some filtering on both power rails to ground using L-C filters. There is no easy answer on what exactly to use and it may not be simple to do this 'properly' - this PDF has some good information.  However if it were me, I would grab some power inductors and capacitors, check the cutoff frequency using the formula, and give it a try to see if it helps reduce the noise. For example I might try 22uH/47uF for 5kHz cutoff.