Testing an Encoder's Accuracy
Initialize a software counter to 0 when Index is detected once. 

Use the Z index signal as a "parity or null count check" for up/down counts in any direction or speed and when Index is detected the count should be = zero.

If a fault is detected, then determine the error value and cause of error by direction or speed or noise interference.  

Test Engineer advice


define acceptable immunity specs. , conducted, radiated and encoder speed, temperature, humidity, vibration, shock etc.
Test to (non-destructive) failure levels of signal integrity to verify your SNR margin.  e.g. Impulse current generator on cables nearby or brushed motor under load., ESD tests. etc
since signal integrity is statistical there may be a probability of error which depends on SNR and margin.