Contradiction between complex baseband and real-valued baseband Suppose a direct-conversion transmitter+receiver (ideal transmitter) along with its filters. In complex baseband, the filters in the signal chain can be modeled as a complex-valued FIR filter ($y$, $c$, $x$ are complex-valued!):

$$
y = \sum_{k=0}^K c_k x[n-k]
$$

Rewriting this in cartesian coordinates gives:

$$
y_i + j y_q = \underbrace{\sum_{k=0}^K \left(a_k x_i[n-k] - b_k x_q[n-k]\right)}_{y_i} + j \sum_{k=0}^K \left( b_k x_i[n-k] + a_k x_q[n-k] \right)
$$

Now only considering the real part, it can be seen that it consists of two different filters with which the real and imaginary input signals are filtered.

Now consider the same system from a practical perspective: After the DAC and upconversion, the RF signal looks as follows:

$$
x_{rf} = x_i \cos \omega_c t + x_q \sin \omega_c t
$$

The output sequence $y_i$ is obtained by multiplying $x_{rf}$ with $\cos(\omega_c t +\phi)$ and filtering it with a filter called $H$:

$$
y_i = H( x_{rf} \cos(\omega_c t +\phi) ) \\
\approx H(x_i/2 \cos\phi + x_q/2 \sin\phi) \\
= \frac{\cos\phi}{2} H(x_i + x_q \tan\phi)
= G(x_i + x_q \tan\phi)
$$

The filter $H$ (or $G$) is modeled as a FIR filter:

$$
y_i = \sum_{k=0}^K \left( g_k x_i[n-k] + g_k \tan\phi x_q [n-k] \right)
$$

From the equation above it can be seen that the real and imaginary part are filtered through only one filter (they differ just by the constant $\tan\phi$ !).

Where does this contradiction come from?

PS: I know that the first approach is the correct one because it gives the correct results. I do not understand why it is not consistent with my second approach
As a sidenote:

When you demodulate your signal $x_{\mathrm{rf}}(t)$  you say that there is some phase difference $\phi$. This translates to crosstalk (i.e., the quadrature and in-phase components influence each other at the receiver). It's easy to see that if $\phi = 0$ (namely, we know exactly when to sample the received signal), there is no crosstalk at all:

$$y_i = \sum_{k=0}^K \left( g_k x_i[n-k] + g_k \tan(0) x_q [n-k] \right) =  \sum_{k=0}^K  g_k x_i[n-k]$$

In the baseband domain, this means that the filter $c_k$ is purely real (if it was purely imaginary, then there wouldn't be crosstalk either, you just have to invert the channels to retrieve the information correctly).

The answer:

Whether there is crosstalk or not, both of your equations get to the same result, but with different names. Using your notation, $a_k$ in the first equation would be equivalent to $g_k$ in the second one, and $b_k$ to $-g_k\tan(\phi)$. I believe that your confusion arose from thinking of $a_k$ and $b_k$ as two different filters (which is correct), instead of thinking of them as the real and imaginary part of a single filter ($c_k$).