How to know if a flyback converter is working in CCM or DCM
The easiest way is to look at the MOSFET drain waveform, \$v_{DS}(t)\$. If this is a high-voltage design, you can even avoid clipping the probe ground and simply carefully approach the probe tip close enough to the transformer and "see" the waveform by radiation. If you clip the oscilloscope ground and connect the probe tip to the MOSFET drain, make sure you have inserted an isolation transformer or supply the converter with an isolated dc supply. Observing the waveforms, reveals several cases:


in discontinuous conduction mode or DCM, the transformer demagnetizes cycle by cycle. It means that at some point, the secondary-side diode stops conducting and the reflected voltage - the flyback voltage - disappears in the primary side. The capacitance lumped at the drain is thus charged at the reflected output voltage scaled by the transformer turns ratio plus the input voltage and needs to discharge to return to the input voltage. The energy stored in this capacitor is now oscillating back and forth between the capacitor and the primary inductance. This oscillation is typical of a DCM-operated flyback converter regardless of its control mode. The below simulation shows a typical shot:




The sense voltage (or the primary switch current) starts from zero and ramps up to the peak value, \$I_p\$.


in continuous conduction mode (CCM), the transformer does not demagnetize cycle by cycle and some energy remains stored in the core. The secondary-side diode conducts during the whole off-time period and the reflected voltage does not disappear. The primary current no longer starts from zero but from some pedestal called the valley current \$I_v\$. The below shot shows the typical waveforms observed in CCM. The square drain-source waveform is typical of the CCM operation for the flyback but also for many other converters (as the DCM ringing):





and finally, there is a third possible case which is also DCM but optimized in such way that the switch always turns on in the minimum of the drain-source voltage. It minimizes turn-on switching losses and can even bring them down to zero with true zero-voltage switching (ZVS). It also limits EMI radiations by reducing or even cancelling the brutal discharge of the lumped capacitance. This mode can either be a specific operating point at a certain load or input voltage or it can be the operating strategy: if you see the switching frequency changing with the input voltage or load but still observe the MOSFET switching right in valley switching, then this is a quasi-square-wave resonant converter, often named quasi-resonant converter or QR. Other names include borderline or boundary conduction mode (BCM). The below shot is the typical signature of this converter:




There are several reasons why operating a flyback in DCM or CCM but usually, if I take the example of a notebook adapter, the converter is designed to operated in CCM at low line and nominal power where conduction losses are smaller and the ac ripple in the output capacitor is minimized while DCM is naturally entered in high input voltage conditions. For a high-voltage power supply, DCM or BCM is usually adopted because the secondary-diode spontaneously turns off, without losses on the primary-side switch and you can use a lazy high-voltage diode for the rectifying job.