A Fast Power Grid Frequency Estimation Approach Using Frequency Shift Filtering
Basically they cascade moving-average filters, or CAF.
a Moving-average filter has the following coefficients
$h_{av}(m) = 1/M, m = 0, 1.. M-1$
In Matlab it would look like this
h = ones(1,M)./M;
y = filter(h,1,x);

If you want to cascade three moving-average filters
h = ones(1,M)./M;
y1 = filter(h,1,x);
y2 = filter(h,1,y1);
y3 = filter(h,1,y2);

M should be equal to the number of samples per period. So if your frequency is approximately 50 Hz and your sampling frequency is 12800 Hz then M = 256;.
Edit : I modified your script to make it work. There were 2 problems, first of all you only processed 256 samples, your couldn't measure anything significant because the transient response of your 3 cascaded filters is 768 samples long. Secondly, you filtered the reference and not your complex signal...
clear
clc
%ADC sin wave, it represent the data that I have captured with the ADC
%and want to estimate the frequency
ADCsin = dsp.SineWave(1,50.1);
ADCsin.ComplexOutput = false;
ADCsin.SampleRate  = 12800;
ADCsin.SamplesPerFrame = 12800;
ADC = ADCsin();
plot(ADC);
grid on
hold on

%ref sin wave with 50Hz complex value
Refsin = dsp.SineWave(1,50);
Refsin.ComplexOutput = true;
Refsin.SampleRate  = 12800;
Refsin.SamplesPerFrame = 12800;
ref = Refsin();
plot(imag(ref))

%calculating  xs
xs = ref .* ADC;

%calculate the CAF
M = 256;
h = ones(1,M)./M;
y1 = filter(h,1,xs);
y2 = filter(h,1,y1);
y3 = filter(h,1,y2);


%calculating the final result
Fest = 50 - (angle(y3(end)) - angle(y3(end-1))) * 12800/(2*pi)