How is the cut-off frequency is set relatively to the signal of interest in a simple RC filter?
Depends on how much "tilt", how much variation in sideband energy your receiver will tolerate.

In AM, the sidebands are counter-rotating in phase; the vector summation produces only Amplitude variation; any slight imbalance results in slight phase modulation as well.

People often are concerned about AM--PM conversion. Any LowPassFilter (and ANY circuit is ultimately becomes LowPass at the higher frequencies) thus causes some AM to PM conversion.

Summary: if your "signal" is just a single tone, with no need to preserve sidebands (the modulation information), and the exact amplitude does not matter, you can set the F3dB for best SNR, or flattest amplitude, or best pulse response (Bessel). If you have a data-eye, where phase modulation contains TIMING OF EDGES and thus affects the data-eye opening, you need to run some simulations or some back-of-envelope harmonic-analysis and phaseshift impacts.