What kind of modulation technique and carrier frequency is used for interplanetary communication such as between the Earth and Mars?
For science data these days, the preferred frequency band for Earth-bound links is X-band for numerous reasons.


Relatively unaffected by atmospheric conditions (on Earth), unlike Ku/Ka which are subject to severe attenuation through rain.
Higher frequency yields higher antenna gain at the probe (rocket fairing constraints limit diameter of antenna, and nobody wants a repeat of what happened to the Galileo probe).  X-band again is a decent compromise.
For the Mars Menagerie there are multiple frequency bands in use from UHF to Ka band, primarily between the ground-based probes and the orbiters which can serve as store-and-forward systems.  Again the primary Earth link is X-band.
The X-band technology is pretty robust, but there have been tests of optical (laser) communications links, however no interplanetary probe has used this as the primary data link.  The Psyche probe is scheduled to have optical capability for deep space testing.  Again the problem with optical wavelengths is the atmospheric uncertainty, which would put such a station up on a mountain range such as Atacama desert for utilization.  Most likely the practical configuration will be to have a satellite (network) receive the deep space signal and retransmit to the ground using a more powerful signal (since the path length is much shorter, and the local satellite can have rather large solar panel efficiency).
It would be difficult to locate enough atmospherically clear areas around the world to provide the same coverage as the existing DSN system.  On the other hand the sites would not take up nearly as much room.
There can be other narrow-band links at other frequencies for telemetry, command and control, emergency operations.