ADC EVAL_AD7984_PMDZ voltage source and signal source sharing common ground
The eval board consists of a unity gain buffer on each input, the output of which goes into the ADC.  Barring some details about current and input impedance, it is as though the board inputs are connected to the ADC inputs.  For the ADC to work, the input voltages need to be between the ADC's ground pin and the ADC's vref pin (see datasheet table 2 -> ANALOG INPUT -> Voltage Range).

If you are powering it from a DC power supply with the eval board's -2.5V input at mains ground, then the eval board ground and hence the ADC's ground pin are at mains ground +2.5V. The ADC's vref pin is held at eval board ground + 5V by the voltage reference IC.

So when either input drops below mains + 2.5V, the corresponding ADC input drops below ADC ground.  That takes the ADC out of spec, so you can't expect it to work properly. Worse, it goes below ADC ground pin -0.3V, which is outside the absolute maximum rating for the chip (datasheet page 6) which could damage the chip. Fortunately, the board includes some 22 ohm resistors which will limit the current and probably protect the ADC from damage.

You seem to be expecting the fact that the amplifier is differential to mean it can take any differential input. That's not the case. Differential means the output of the ADC is the difference between the inputs, but for the ADC to work the inputs need to be within the input limits. Every ADC and differential amplifier has input limits - you wouldn't expect to connect 1000000V to one input and 1000001V to the other and read one volt without fireworks. For this ADC, the limits are ground to Vref (which is 5V on the eval board). Very few differential amps or ADCs accept input voltages outside their power supply voltage range. If you really need that, then you need to look at either an isolating amplifier or a transformer.