Extremely noisy signal after voltage follower/buffer
Looking at the datasheet, the AD8004 does not seem to be a great choice for a voltage follower, and some special consideration might be needed if you are going to use it that way.


Nowhere in the datasheet is it claimed that the amplifier is gain-of-1 stable. However some specs are given for gain-of-+1 configurations, so it's not entirely impossible to use it this way.
Where gain-of-+1 specs are given, a feedback resistance (Rf) of 700 - 1100 ohms is specified. There are no specs for how the part will operate in the usual voltage-follower configuration with a short circuit (Rf = 0 ohms) connection from output to inverting input.
The frequency response in the G=+1 configuration shows substantial peaking.


My conclusion from this is that the amplifier is "just barely" stable in unity-gain configurations. Very possibly your amplifier is oscillating if you have hooked up the standard "voltage follower" configuration. 

If you do want to use AD8004 as a follower, you should definitely include a resistor in the feedback path (806 or 1.21k ohms are recommended in table I of the data sheet).

As a comment suggested, you also must be sure to connect the pins of any unused amps in the package in such a way that they will maintain a steady voltage. If you leave them open, the unused outputs are likely to fluctuate randomly. And this fluctuation is likely to couple (for example, through the power supply) to the used amplifier and exacerbate any problem you have with stability.

But even safer, especially given you have only a 1 MHz input signal, would be to find a more appropriate op-amp for this application.