Op-amp input voltage gets clipped off
The datasheet of the chip you linked to has the answer.
In the nominal operating conditions, the inputs must be withing supply voltage limits, between 0V and 5V in your case. You are exceeding that by going below 0V and witnessing the results of it.
In the absolute maximum ratings, the inputs have a limit of -0.3V.  Below that value, the chip does not have to work, but it also won't get permanent damage. Exceeding that can cause permant damage to the chip.