What does the phase discriminator portion of the Costas Receiver do mathematically?
This pdf has the answer for you:
The short answer is that you have conflicting assumptions in number 4 above. You use the small angle approximation for sine but not cosine. If $\phi$ is small then $cos(\phi) \approx 1$. So you would end up with only $\phi$ in that term times the amplitude and $m^2(t)$. Once you integrate, or low pass filter, the signal component $m^2(t)$ for long enough that becomes basically constant and you're left with the result of something that is proportional to $\phi$. That makes the VCO advance or retard the frequency momentarily based on the phase error.
I should add that the small angle approximation generally is a good model even for when there are large errors. Generally when the error is large the error term and VCO push things in the right direction, even if the precise amount isn't perfect. Keep iterating on that and you eventually get close enough for the small angle approximation to be valid.