How can I tell if this boost converter is in continuous or dis-cont. mode?
From your description of the inductor, it seems like you used a filter inductor rather than a power inductor. You need an inductor that has a high DC saturation current. Something like a Murata 12RS105C. 1 mH will be easier for your experiments than 100 uH.

[Repeat of my comment, so everything is in one place: To see what is going on in the circuit, you need to measure the current in the inductor or the FET. Since you probably don't have a current probe, put a small (~0.2 ohm) resistor in series with the ground side of the FET and put a scope probe on it.]

There are many boost converter equations online. But the most basic is the inductor "charge" equation. If you put a constant current into a capacitor, the voltage increases linearly. If you put a constant voltage across an inductor, the current increases linearly (until it saturates). 



You can use this equation to estimate the MOSFET ON time before saturation.

If you put a small resistor in series with the MOSFET source pin, you can see the MOSFET current. You can infer the inductor current from this.

These circuits will often have nasty ringing that can make the plots hard to interpret. You may need to add an RC snubber across the MOSFET.



Edit: And like I said in my comment, you need a load.

Edit2: Measurements on actual circuit added.

I don't have access to a current probe since I retired. I thought that I had a suitable small resistor to measure the current at the MOSFET source, but I don't. It needs to be a non-inductive resistor. But, you can use the lousy contact resistance of the breadboard to your advantage. Each contact can have a few tenths of an ohm, maybe more. So, channel 2 has 2 breadboard contacts in series, nothing else. It isn't calibrated, but you can get an idea of the signal shape.

According to the inductor equation, the current peak should be about 400 mA.

5V * 80 uS / 1 mH = 400 mA.

The circuit puts out about 18 volts with a 50% duty cycle. If you vary the duty cycle, you should be able to get it into continuous mode.







simulate this circuit &ndash; Schematic created using CircuitLab

Channel 1 is the gate drive. Channel 2 is the source resistor voltage.



Edit3: Added continuous mode plot.

I increased the duty cycle until it went into continuous mode. Note that I had to increase the frequency to prevent my inductor from saturating.