Complex low pass filters
A low pass filter has a frequency response that satisfies

$$|H(\omega)|\approx 0,\quad |\omega|&gt;\omega_c\tag{1}$$

where $\omega_c$ is the cut-off frequency. A complex low pass filter must also satisfy

$$ H(\omega)\neq H^*(-\omega)\tag{2}$$

which causes its impulse response to be complex-valued. So the frequency response of a complex low pass filter is characterized by Eqs $(1)$ and $(2)$.

In discrete-time, you have the usual convolution sum

$$y[n]=\sum_{k=-\infty}^{\infty}h[k]x[n-k]\tag{3}$$

where $y[n]$ is the output sequence, $x[n]$ is the input sequence, and $h[n]$ is the filter's impulse response. Note that all sequences in $(3)$ are generally complex-valued. If you split up (3) into its real and imaginary part you get

$$y_R[n]=\sum_{k=-\infty}^{\infty}\left(h_R[k]x_R[n-k]-h_I[k]x_I[n-k]\right)\\
y_I[n]=\sum_{k=-\infty}^{\infty}\left(h_R[k]x_I[n-k]+h_I[k]x_R[n-k]\right)\tag{4}$$

where the subscripts $_R$ and $_I$ denote the real and imaginary parts, respectively. From $(4)$ you see that you have to implement two filters, one with impulse response $h_R[n]$ and the other with impulse response $h_I[n]$, and that you have to filter the real part as well as the imaginary part of the input signal with both filters.

As for applications in digital communications, anywhere where you have in-phase and quadrature components, you can model these two signals as a complex signal, and any filtering on this complex signal requires a complex filter (or, at least, a filter with complex input and output). However, in practice you most often encounter two special cases:


The filter is actually real-valued but has a complex input and output, i.e. you simply filter the real and imaginary parts of the input signal independently with the same filter. In Eq. $(4)$ that means that $h_I[n]=0$. This is the case when you apply a low pass filter after complex demodulation. 
The filter is complex-valued but the input signal is real-valued, so you filter one (real-valued) signal with two filters (the real and imaginary parts of the impulse response). In Eq. $(4)$ that means that $x_I[n]=0$. This occurs when you filter a real-valued passband signal with an analytic band pass filter before complex demodulation. (Note that here the filter is a band pass, not a low pass).


A complete complex filtering operation with complex-valued inputs and outputs, and a complex impulse response (as specified by Eq. $(4)$) is used for modelling a digital communication system using an equivalent discrete-time baseband channel. The corresponding baseband channel filter is generally complex-valued.