Help in analytical expression of BER for M-QAM I am new in learning deriving BER expression. I am familiar for the case of BPSK in AWGN but cannot understand how the BER expression in Eq(12) in this paper, An Enhanced Spectral Efficiency Chaos-Based Symbolic Dynamics Transceiver Design
download link

comes. http://www.raymaps.com/index.php/theoretical-ber-of-m-qam-in-rayleigh-fading/

tutorial presents the BER expression for M-QAM in AWGN. 

Do you have any reference on how to analytically derive the BER for QAM and Rayleigh in AWGN for this case? How can I derive?
You need to read a good book to form a solid foundation first and then read research papers to understand advanced concepts. Eq (12) comes through a standard procedure for BER calculations. See Digital Communications by John Proakis, for example.