coding gain and shaping gain in SCMA
You can read about shaping gain in any book about digital communication. For example, John G. Proakis, "Digital Communications".
The theory is presented in J. H. Conway and N. J. A. Sloane, "Sphere Packings, Lattices and Groups".

In SCMA case, the multidimensional mother constellation only potential can increase shaping gain. There are many different methods for design good SCMA codebooks. I conducted a study comparing known codebooks and presented method for design codebooks based on genetic algorithm. For AWGN channel, there are codebooks based on QPSK alphabet with very good performance (in comparison with codebooks based on good multidimensional constellation).

In addition to a good multidimensional mother constellation, it is necessary to optimize the rotation factors, power variation coefficients, permutation, etc... Only a good multidimensional mother constellation (it has a shaping gain) does not guarantee the good performance of SCMA. So, QPSK mother constellation without shaping with good optimization of the rotation factors may provide the good performance of SCMA system.