Inverse Kinematics constraint with pole target
Maybe I'm wrong and someone will find a solution with the IK limits but it looks like you can't prevent the knee twisting this way.

You need to use your Pole Target bone to fix the problem, and anyway the Pole Target is here for that.

So, in Pose mode when you see that the leg rotation make your knee twist in the bad direction, just move the Pole Target and give it a Keyframe so that the knee comes back to the right bending.