What does the phase discriminator portion of the Costas Receiver do mathematically? What does the phase discriminator portion of the Costas Receiver do mathematically? 




The output of the I-channel is $ \frac{1}{2}A_C \cos \phi \, m(t) $. Which means for small deviation of phase $ \phi $ , $ \frac{1}{2}A_C \cos \phi \, m(t) \approx \frac{1}{2}A_C \, m(t) $. Or it can be said that $  \frac{1}{2}A_C \, m(t) $ would be attenuated by a small amount. However, we have to keep in mind as the text says (as it not might be as innocuous as it seems): 


The output of the Q-channel is $ \frac{1}{2}A_C \sin \phi \, m(t) $. Which means for small deviation of phase $ \phi $ , $ \frac{1}{2}A_C \sin \phi \, m(t) \approx \frac{1}{2}A_C \phi \,  m(t) $.
As per the text, the phase discriminator consists of a multiplier followed by a low pass filter.
Which means multiplication of $ \frac{1}{2}A_C \cos \phi \, m(t) $ and 
$ \frac{1}{2}A_C \phi \, m(t) $ would yield:

$ g(t)=\frac{1}{2}A_C \cos \phi \, m(t) * \frac{1}{2}A_C \phi \,  m(t) $

$ g(t)= \frac{1}{4}A^2_C \phi \cos \phi \, m^2(t)$

Now, let's say $ m(t) $ is band limited to $ M $. then the term $ m^2(t) $ in the frequency domain would spread across  $ -2M $ to $ +2M $ centered around $ f=0 $ and the term $ \frac{1}{4}A^2_C \phi \cos \phi $ is a constant. 
Now, if $ g(t) $ is subjected to a LPF, then the term $ \frac{1}{4}A^2_C \phi \cos \phi \,  m^2(t) $ would be retained as it is(if the cutoff frequency of the LPF is slightly > $ 2M $).
So, what does phase discriminator in Costas receiver do mathematically? 

Also, if the phase error is significant (that is we have no idea at all about the probable phase of the carrier), then how does the analysis change? Because this is a possibility that we might have no idea about the phase of the carrier.


Text Used: Communication Systems By Simon Haykin
This pdf has the answer for you:
The short answer is that you have conflicting assumptions in number 4 above. You use the small angle approximation for sine but not cosine. If $\phi$ is small then $cos(\phi) \approx 1$. So you would end up with only $\phi$ in that term times the amplitude and $m^2(t)$. Once you integrate, or low pass filter, the signal component $m^2(t)$ for long enough that becomes basically constant and you're left with the result of something that is proportional to $\phi$. That makes the VCO advance or retard the frequency momentarily based on the phase error.
I should add that the small angle approximation generally is a good model even for when there are large errors. Generally when the error is large the error term and VCO push things in the right direction, even if the precise amount isn't perfect. Keep iterating on that and you eventually get close enough for the small angle approximation to be valid.