Carrier and timing not properly locking?
I recommend that the OP confirm via testing each of the loop parameters separately in terms of convergence time and lock-in range. Step response testing is a good approach for convergence time and the range of the step can be increased to both evaluate lock-in range and close-in "linear" response that would affect tracking versus further-out "non-linear" response that would affect acquisition.

To do these one at a time I suggest for example the AGC: with the system locked in carrier and timing, introduce a very small step in amplitude and evaluate the time domain response and how long it takes to converge. Increase the range of this step and the response time should stay the same when in linear operation, and then start to increase or change as the operation becomes non-linear (if it does), and then eventually lose lock if the step is outside of the acquisition range (or more likely causes the carrier and timing to lose lock).  Ideally this is done in simulation such that the other parameters can be forced to remain locked. 

Carrier and Timing would follow the same approach: for carrier the level should remain constant and carrier and timing acquired and then introduce a small carrier offset, etc...