Why the encoder isn't giving a proper square wave?
Check that you haven't enabled AC coupling on your oscilloscope for CH1. It looks like there's an RC decay on each cycle of your encoder, which would fit with AC coupling on the oscilloscope.