FSK and IQ modulation
What is the advantage of performing the FSK using IQ modulation?



You only need one RF oscillator operating at a single frequency, instead of having 2 (or more in the case of M-ary FSK) oscillators operating at separate frequencies for each bit/symbol.
Since you only have one oscillator, you don't have to worry about discontinuities in the phase of the transmitted RF signal, like this:




Sharp discontinuities contain high frequency which widens the bandwidth of the signal. With I/Q modulation you'll have a much easier time producing a continuous-phase FSK signal (CPFSK).


  How to demodulate the FSK signal?


First lets assume we are trying to demodulate a simple FSK signal where the center frequency is $f_c$, a '1' is represented by the frequency $f_c+\Delta f$, and a '0' is represented by the frequency $f_c-\Delta f$.

Before IQ sampling, you tune your RF front end receiver to $f_c$. Then when you sample you have a baseband I/Q representation of the signal, centered around 0 Hz. The only thing to do is figure out if you have a positive frequency (+$\Delta f$) or negative frequency (-$\Delta f$). The easiest way to do that is to monitor the change in phase of the baseband IQ signal. A positive phase change corresponds to a positive frequency, and a negative phase change corresponds to a negative frequency.

So monitor the phase (the angle of $I + jQ$) and watch how it changes over one symbol period. Positive change = '1', negative change = '0'.


  Does the FSK signal look triangular wave or sinewave?


If you're talking about the RF signal, it looks like a sinewave of varying frequency.

If you're talking about baseband IQ signal, it also looks sinusoidal, but with a bunch of phase shifts, like this (blue = I, orange = Q):



In this particular example, a '1' corresponds to a phase change of $+\pi/2$, and a '0' corresponds to a phase change of $-\pi/2$.