Help in analytical expression of BER for M-QAM
You need to read a good book to form a solid foundation first and then read research papers to understand advanced concepts. Eq (12) comes through a standard procedure for BER calculations. See Digital Communications by John Proakis, for example.