Finding a suitable ADC driver (SE to DIFF VCM+-0.5V)
The whole misunderstanding was caused by the following sentence on page 42 of the ADC datasheet:

Each input pin (INP, INM) must swing symmetrically between (VCM + 0.5 V) and (VCM – 0.5 V), resulting in a 2-VPP (default) differential input swing.

So it was just a basic understanding of differential signals that was lacking. There is no offset, only gain and VCM.