Grounding one port of SPDT analog switch
It is possible to use an analog switch in this way. For digital 20 MHz signals, the switch resistance and capacitance will not have any noticeable effect.
However, analog switches are more complexity than actually required, and when the D/E signals already are grounded, you do not need connect them to anything. The output is either the same as the input, or disabled; you can implement this logic with a three-state buffer. There are two-channel devices like the (SN)74LVC2G125; use 2× (SN)74AHCT1G125 if the control signal uses TTL voltage levels.