Is my Inverse Kinematics not working correctly or is it working correctly?
If you make a 2 step vertical IK chain (controlled by a target bone), without any pole bone, and then in edit mode you move slightly the "knee" forward, in pose mode the bending of the chain will follow this slightly moved direction (tipically forward).

With a pole bone you can further control the direction of the bending.



Make sure that as seen in the image, the resting armature's knee (in edit mode not pose mode) is forward of the rest of the leg, ie: bent forward just a little. That way IK will bend it in the right direction.