SDR and amplitude modulated signals
Every signal transmitted by an SDR is a quadrature signal: the sum of two amplitude-modulated signals, with carriers that are in quadrature (one is at 90 degrees from the other).
A bit of trigonometry shows that this is equivalent to an amplitude and phase modulated signal; in other words, a quadrature signal is equivalent to $A(t)\cos(2\pi f_c t + \phi(t))$.
As long as you can express your signal with one of these two (equivalent) expressions, you can do anything! You can certainly do FM on top of AM if you want to. For many of these &quot;exotic&quot; signals, the challenge will be in figuring out a demodulation process, but don't let that stop you from experimenting.