can $e^{j\phi_0}$ be incorporated into the sine and cosine terms?
Your problem might be the unmentioned fact that the given frequency response $(4)$ in Carlson's book is only valid for positive frequencies. The complete definition should be
$$H(f)=\begin{cases}Ae^{j\phi_0}e^{-j2\pi ft_g},&amp;f&gt;0\\Ae^{-j\phi_0}e^{-j2\pi ft_g},&amp;f&lt;0\end{cases}\tag{1}$$
Now we have $H(f)=H^*(-f)$, which is necessary for the corresponding system (channel) to be real-valued.
If you split the system into two subsystems $H_1(f)=Ae^{j\textrm{sgn}(f)\phi_0}$ and $H_2(f)=e^{-j2\pi ft_g}$, where $H_1(f)$ is a (scaled) phase shifter, and $H_2(f)$ is a pure delay, it's easy to see the impact of the total system on the input signal.
Consider the input signal $x_1(t)\cos(\omega_ct)$. The output of the scaled phase shifter is $Ax_1(t)\cos(\omega_ct+\phi_0)$, if the signal $x_1(t)$ is a lowpass signal with a maximum frequency smaller than the carrier frequency $\omega_c$. Delaying that signal by $t_g$ gives
$$y_1(t)=Ax_1(t-t_g)\cos[\omega_c(t-t_g)+\phi_0]\tag{2}$$
The result for the signal $x_2(t)\sin(\omega_ct)$ follows in a completely analogous way.
In sum, the two necessary conditions for the given formula to be true are

$H(f)$ corresponds to a real-valued system, i.e., $H(f)=H^*(-f)$, and
$x_1(t)$ and $x_2(t)$ are lowpass signals with maximum frequencies smaller than the carrier frequency $\omega_c$.