Aliasing in digital communication system In a digital communication system, is it possible to counteract the aliasing problem by properly tuning the constellation order $M$ and/or the carrier frequency?

Here is my train of thought, please correct me if I'm wrong:

The sampling rate needs to be 2 times the signal bandwidth. This is because we need to cover all the frequencies in which the signal operates. If the signal breaks the Nyquist criterion we'll only get a part of the signal. So if I look at a fix sampling rate and try to tune the constellation order $M$ up and down it won't help, because the signal will still have the same bandwidth. The only thing that will change is the transfer rate $r_b = \log_2(M)/T_s$, where $T_s$ is the sampling rate. Also tinkering around with carrier frequency will only move the signal up or down the frequency. So neither of these solutions will counteract the aliasing problem. The only solution is to somehow reduce the signal bandwidth. I'm thinking of using an RRC filter on the transmitter (to shrink the bandwidth) and then using another RRC filter on the receiver to counteract the ISI.
If the signal breaks the Nyquist criterion we'll only get a part of the signal.


This needs to be clarified: the frequencies above Nyquist will fold back into the Nyquist band, so it's not just that you lose part of the signal; it's that the signal is corrupted (usually) beyond hope of reconstruction.

As an example, consider a 4 kHz real signal sampled at 4 kHz -- the frequencies between 2 and 4 kHz will fold back into the 0 to 2 kHz range; not a single "part" of the signal will survive.

You're correct that this can't be solved by modifying the constellation order $M$ or the carrier frequency. As you correctly assume, you have two options. You can reduce the bandwidth through pulse shaping; if the symbol rate is $R$, then the minimum bandwidth $B=R/2$ is achieved with sinc pulses. The second option is to reduce the rate.

(As a side note, recently there have been proposals to increase the rate without increasing the bandwidth (search for "Faster than Nyquist signaling"), but I think these ideas are outside the scope of your studies at the moment.)