12V -> 3.3V DC step down at low current
Basically, your problem is that one impulse of energy through the inductor/capacitor is far more than you need to keep the output voltage "in regulation". You have chosen to short out the current limit resistor but, this is fundamental to controlling the energy per pulse.

Energy per pulse is I^2*L/2 and you have no control on what I limits at. The power delivered to the load is this energy multiplied by the number of times you pulse per second.

Because it is only "doing" 500Hz it certainly is in discontinuous mode and this, given what I've just said, shouldn't surprise you any more. You need to choose a value of current limit resistor that stops too much energy being delivered to the load.

You might also try reducing the value of L to about 100uH as well - this reduces the energy per pulse by a factor of 6.8.