NCO loses lock after a while for certain values of alpha EDIT: So the transfer function for my system is

$$ F_{nco}(z)=\beta E(z)/(z-1)$$
$$ z\Theta_{nco} (z)=\Theta_{nco} (z)+\alpha E(z)+\beta E(z)/(z-1)$$

So I would like to obtain $\Theta_{nco} (z) / \Theta(z)$ transfer function cause I think that would allow me to calculate whic range of $\alpha$ would be good for my PLL to lock. The thing is I don't really know how to obtain $ E(z)$ from $e[n]= \arg \big\{ x[n] \cdot (y[n])^* \big\} $



I'm trying to code my own NCO in order to use it for my own Costas loop (since I couldn't get this to work Implementing Costas loop for 4-PAM) and so far the NCO is working good except for it losing lock after a while. These are the update equations I'm using:

$$\theta_{nco}[n+1]= \theta_{nco}[n] + \alpha e[n] + f_{nco}[n]$$

$$f_{nco}[n+1] = f_{nco}[n] + \beta e[n]$$

So I generated this input with $f=50 Hz$ and $f_s = 1 kHz$ during $0.2 s$ (that is, 200 samples):

$$\begin{align}
x[n] &amp;= e^{j 2\pi n f/f_s+\theta} \\
  &amp;= \cos(2\pi n f/f_s+\theta)+j\sin(2\pi n f/f_s+\theta) \\
\end{align}$$

Then I'm running a for loop to update the NCO, which initially has 0 phase and frequency ($\theta_{nco}$ and $f_{nco}$). The parameters being $\alpha = 0.1$ and $\beta = 0.0025$. 

The way I obtain the error is 

$$e[n]= \arg \big\{ x[n] \cdot (y[n])^* \big\} $$

being $y[n]$ the output of the NCO:

$$y[n]=e^{j (2 \pi n f_{nco}[n] + \theta_{nco}[n])}$$

The thing is, it locks quite fast but it loses lock towards the few last samples (at around 0.2 s) (see attached picture). If I increase the number of samples, it loses lock even more.

I managed to fix it by changing alpha and beta to $\alpha = 0.05$ and $\beta = 0.000625$

So basically what I'd like to know is how I can know which are the best values for $\alpha$ and $\beta$ since I'm just randomly picking a low number for $\alpha$ and then doing $\beta=\alpha ^2 /4$. Thanks!
I will assume that your PLL is purely digital. Is that correct?

I assume your PLL consists of 3 blocks

The first block calculates the phase of your incoming signal (the signal you wanna lock to).

The second one will compare the phase of the signal to the phase of your "internal oscillator" yielding the error e[n]

The third one is your "controller" which I assume is some kind of PI controller with alpha being the proportionnal gain and beta being the integral gain. 

Your phase detection algorithm that detects the phase of your input signal has some kind of low-pass effect. This creates some kind of delay between the output your PI controller and the measurement. This delay puts a limitation on the values of your alpha and beta parameters. I.e. if you increase the values too much, your PLL can be unstable and you will lose the lock.

If you change the phase of your incoming signal, how many samples does it take for the change to be effective? This will tell you approximately what your delay is. From this delay, you should be able to find alpha and beta gains that will give you a good gain and phase margin. You should try at least 6 dB and 45 degrees.

edit : How you obtain the value y*? Is is some kind of delay chain? Hilbert Filter? I assume there is some kind of delay involved...

edit 2 : Your controller seems ok, your gains are low. Your bandwidth is low and thus the closed-loop should be stable. I suspect that your error happens when the phase measured by algorithm transitions between pi and - pi. Picture this case

Phase NCO = (pi - 0.001)

Phase ref = (pi + 0.001)

The phase difference should be 0.002 radian right?

However, phase ref will not actually be (pi + 0.001) but rather (- pi + 0.001)

Yielding error = (-2*pi + 0.002). Which is not the real error, this has the effect of suddenly increasing the bandwidth of your controller, which is something you do not want to do. This is probably why it is stable at lower values of alpha. The glitch would be there but not as big. 

I'm assuming your algorithm yields a phase between -pi and pi, if the algorithm yields a phase between 0 and 2*pi, the thinking remains the same.