De-bouncing a (Gray) Rotary Encoder Switch in C
I don't see why bother with clipping averages. If something was less than ROTSW_MIN, it is also less than ROTSW_LOWER.
count is useless, you may operate directly on rotsw_count.




The moving averages could be computed a bit faster in assembly. The rotsw_samples array effectively contains 8 2-bit entries, and hence could be stored in a 16-bit register, say R. Now consider something along the lines of

            rlc R             ; Shift the ra2 of the "old sample" into carry
            sbc avg_ra2, $0   ; Subtract carry
            rlc R             ; Same for ra3
            sbc avg_ra3, $0

            or R, sample      ; New sample is added to the "array"

            rrc sample        ; Shift ra3 of the "new sample" into carry
            adc avg_ra3, $0   ; Add carry
            rrc sample        ; Same for ra2
            adc avg_ra2, $0 


The exact syntax depends on the architecture and an assembler; it also assumes that the sample is already shifted right by 2.

Your target architecture may allow even faster variants.

Note that the similar tricks could be used to speed up the final rotsw_count computation.



There is not enough context to address your worries about the switch being too speedy. In any case, the code is already quite fast. If doesn't keep up with the peripheral, the best answer is to upgrade the CPU.