How to measure/estimate symbol rate of a signal with unknown ( but probably QAM ) modulation? Is it possible to estimate the symbol rate of a signal whose modulation is not know exactly but it can only be guessed that it is probably some kind of QAM (probably 64 or 256). 

What is the best way to get an estimate for the  symbol rate of a signal of unknown modulation?

For example by feeding the signal through some non-linearity and then looking at it's spectra, should that reveal something about the symbol rate ?

More precisely, I have a 10 MHz wide signal recorded with an SDR at 2.45GHz. It is emitted by a wireless camera.
The nonlinear trick is in fact often used to recover the symbol clock of $\frac \pi {2N}$ phase-shifted signals, e.g. BPSK, QPSK: Squaring samples $x[n]$ (or even taking $x^4$) leads to a removal of phase information, and you can then optimize timing with something that is pseudo-derivate based like Müller&amp;Muller.

Now, the problem with QAM is usually that there's no two constellation points with the same phase; so an arbitrary amount of squaring can not reduce the number of phases to 1.

For such signals synchronization is complicated even with exact knowledge of what to expect; it's pretty common to see preamble-correlation based synchronization in receivers. Upside of that is that you can not only get timing, but also a fair amount of CSI that way, which leads to the possibility of equalization.

As a first step, however, some kind of autocorrelation-based detector surely sounds promising: Assuming that with a finite set of constellation points, there's also only finitely many sequences of symbols within the influence of a single pulse shaping filter impulse response, which also means that one might expect the signal to repeat after that amount of time. Hence, a peak on the autocorrelation¹ function of your signal probably signifies the length of $N,\, N\in\mathbb N$ symbols.

With that known, you'd still be on the lookout for $N$; remember that the spectrum of a data-modulated signal is the pulse shape convolved with a diraccomb with an inter-dirac distance of $\frac{1}{f_{symbol}}$. Hoping that symbol timing coincides with the pulse shape filter's time domain roots, you could just look at a sufficiently oversampled signal and count the spectral "peaks"; in reality, this will typically not be as clear.

A few other hints: If the spectral shape is wideband and rectangular, possibly even with a bit of "sincy" sidelobe on either side, it's probably OFDM. That'll make your life a little harder, but: OFDM typically uses cyclic prefixes, which means that you can find the OFDM symbol duration using autocorrelation; since the cyclic prefix needs to be a whole number of subsymbols (IDFT bins), you can narrow down the possible OFDM configurations quite a bit.

¹In fact, any periodicity-based estimator might work; have a look at thegr-specest toolbox. It has some parametric spectrum estimators, of which MTM might be very interesting, here.