Interfacing AVR microcontroller to ADC, waveform generator and other peripherals
Your block daigram shows no relationship between the Oscillator/Divider and the Waveform generator, but these must be optimized for synchronous mixing.

Can you try a simple approach?

To mix the the sine input and output, may I suggest the classic 180deg range XOR mixer which is commonly known as a Type I PLL phase detector. The inputs are the same frequency and the output is the difference phase and sum frequency ( + harmonics) so a LPF must be chosen to easily suppress the 2f components yet provide fast tracking of phase vs voltage.


let R1C1 = R2C2 = R3/C3 = 1000/f = 1000/250KHz = 4ms
let R1 = 1~10 MΩ, Let R3 ~ 1KΩ to drive ADC
use buffered inverters '04 and '86 XOR 

Average ADC readings to achieve desired resolution and noise reduction by √n samples.