FPGA encoder counter running away randomly
Capture the input rather than the counter to see if bounce is your problem. If you have storage scope, use edge trigger to get the capture. Otherwise try adding debounce between Reg1/Reg2 and between Reg4/5.

To implement this use an 8 or 16 element shift register fed from synchronised A. Take the AND and the ~OR from between all the shift register bits, these give you are "all set" and "all clear" signals. Use this as the set and reset of an output register, which would feed REG2. Same for B -> REG 5.

If you find this still glitches you can either try a lower clock, or extend the shift register. Above 16 stages it's probably better to remodel as a binary counter and last-state bit. On each cycle if the state is different from last, reset counter and update last-state. If the counter overflows without being cleared, that's your trigger to set/reset the output.