offset binary from ADC to Decimal
How would I check it? 

I would write (VHDL testbench code) to translate from the real data (sinewave) I put into my ADC simulation model, into the binary format it produced and transmitted to the FPGA, and the testbench would check that the value computed and the value inside the FPGA were identical.

If they were not always identical I would then look at the simulation waveforms around the times that the simulation reported errors, to identify what caused the differences, and correct whatever was wrong. Modelsim can display analog waveforms, which make spotting some forms of error easy.

Some types of error are easier to see using a ramp waveform, which should generate a monotonic count, so not all tests have to be sinewaves.

(as ADCs aren't always provided with good simulation models, you often have to write the models yourself. But that's just behavioural code, so it's not too difficult, and it's well worthwhile in the long run)