Generating a signature to identify RF modulation types
Yes, this is quite a studied problem in certain fields. I co-own a patent on it and we used one out of many approaches. We broke the problem into a feature extraction step and then a classification step. Some features can be the bandwidth, the continuity of the phase, the amount of phase clusters for digital modulation, etc. Then we used a Mahalanobis distance comparison to known modulations. I've also seen linear discriminants and neural networks used to process the features.

Certain modulations also exhibit certain shapes once you square them, cube them, etc. and then compute the power spectral density. Often if you do an FM or AM demodulation you can see spectral lines for digital modulations. Cyclostationary computations can also be helpful in digital modulation detection. The post above is right in regards to the search topic: "modulation classification"