Nick Gammon "Camera shutter speed timer"
Can you tell me what is happening [...]?


Not really, but here is a wild guess: your input signal is bouncy, the
ISR fires too often, your loop cannot keep up with the rate, and you end
up accessing endTime in the ISR and loop simultaneously.

If you have a scope, look at your encoder signal. Is it clean?


  would it be better to use a counter instead of interrupts?


Using the input capture feature of a timer will give you the best
possible accuracy, as you will have single-cycle resolution. In
contrast, micros() has only 4&nbsp;µs resolution (64 times worse) and
the ISR may be delayed by up to ≈&nbsp;16&nbsp;µs by the TIMER0_OVF
ISR.

You can use the FreqPeriod library you linked to as an inspiration for
writing your own timer-based code, but it will require some changes:


As pointed out by Nick Gammon in a comment, you don't need the analog
comparator, nor the conditioning RC network, as the encoder's outputs
are already digital. Or you may use a low-pass filter if the encoder
is bouncy.
You will use the input capture pins of either timer 4 (digital pin 49)
or timer 5 (digital pin 48). The other input capture pins are not
available on the Mega's pin headers.
You will replace ISR(ANALOG_COMP_vect) by either ISR(TIMER4_CAPT)
or ISR(TIMER5_CAPT) (the relevant “capture event” ISR). No need to
adjust a comparator threshold in the ISR.
In the begin() method, you will leave the analog comparator alone,
and you will enable the capture event interrupt.


Obviously you will have to study the
datasheet
before start coding.