How to check if my quadrature encoder is broken or not?
plus what the heck is the orange one?


Quite possibly a "zero position" output, for those types with a zero-position sensor. For your variant, this is likely unconnected. If it was connected, you get a single pulse for every shaft rotation.



The electrical interface does not use mechanical switches, but does use transistor switches for the quadrature signals...(from the data sheet):

 If you want to check it out using LEDs, try this circuit. Use any LED you have handy...rotating the shaft should blink the LED on-and-off quickly:







simulate this circuit &ndash; Schematic created using CircuitLab