How do I take the real part of this bandpass filter system's output?
The signal $f(t)$ is a complex bandpass signal. Note that if $H(e^{j\omega})$ is an ideal lowpass filter (and $\omega_c&gt;\omega_0$) then $f(t)$ only has positive frequency components. For a real-valued signal, the spectrum shows conjugate symmetry, so there have to be negative frequency components as well.

By taking the real part of $f(t)$ you get $y(t)$:

$$\Re\{f(t)\}=\frac12\left[f(t)+f^*(t)\right]\Longleftrightarrow
\frac12\left[F(j\omega)+F^*(-j\omega)\right]=\\=\frac12\left[X(j\omega)H(j(\omega+\omega_c))+X^*(-j\omega)H^*(j(\omega_c-\omega))\right]\tag{1}$$ 

(where '$\Longleftrightarrow$' means Fourier transform pair).

Since $x(t)$ and $h(t)$ (the impulse response of the ideal lowpass filter) are real-valued, their Fourier transforms satisfy $X(j\omega)=X^*(-j\omega)$ and $H(j\omega)=H^*(-j\omega)$. Using this property, Eq. (1) can be rewritten as

$$\Re\{f(t)\}\Longleftrightarrow\frac12 X(j\omega)\left[H(j(\omega+\omega_c))+H(j(\omega-\omega_c))\right]=Y(j\omega)$$