How can I set an IK limit on a bone's Y axis?
With no pole, locking the Y axis works, and can be seen by locking/unlocking it in pose mode.  However, what exactly it means to lock the Y axis isn't necessarily intuitive; in reality, orientations aren't sets of 3 independent axes.
There is one situation where locking the Y axis (of the first bone in the chain) will do nothing: when you are using a pole target.  In this case, the calculations used to track the pole will replace the limited Y axis rotation of the first (only) bone in the chain.
Note that pole targets aren't necessary for IK; they're an option, and there are alternative controls.
If I wanted to create an IK thigh twist bone with a pole target, here's what I'd do:

The thigh bone of the IK chain is non-deforming.  The deforming thigh twist bone is parented to it.  Note the hierarchy: the deforming (upper, non-twist) thigh bone is not parented to it.  Instead, it's parented to the hip (or whatever the IK thigh's parent is), and then acquires the proper orientation via a damped track constraint, in this case, targeting the head of the thigh twist bone.