M-Array DPSK decoding and encoding
As vaz's answer suggests, your differential encoding is incorrect. In QPSK
regarded as a phase-shift-keyed signaling scheme as opposed to
a 4-QAM scheme (see this previous answer of mine for some details), the input dibit $(b_I,b_Q)$ is treated as being the Gray-coded integer representation of
the integers $\{0, 1, 2, 3\}$ and the phase of the RF carrier in the
corresponding signaling interval is $0$ or $\pi/2$ or $\pi$ or $3\pi/2$. In tabular form, we have
$$
\begin{array}{|c|c|c|c|c|c|}
\hline
(b_I,b_Q) &amp; \text{normal value} ~k &amp; \text{Gray code value} ~\ell  &amp;\text{phase-modulated signal}\\
\hline
(0,0) &amp; 0 &amp; 0 
&amp; \sqrt{2}\cos\left(2\pi f_c t - 0\frac{\pi}{2}\right)\\
(0,1) &amp; 1 &amp; 1  
&amp; \sqrt{2}\cos\left(2\pi f_c t - 1\frac{\pi}{2}\right)\\
(1,1) &amp; 3 &amp; 2 
&amp; \sqrt{2}\cos\left(2\pi f_c t - 2\frac{\pi}{2}\right)\\
(1,0) &amp; 2 &amp; 3  
&amp; \sqrt{2}\cos\left(2\pi f_c t - 3\frac{\pi}{2}\right)\\
\hline
\end{array}
$$
That is, we can say that the QPSK modulator input
$(b_I,b_Q)$ is regarded as the Gray code representation
of the integer $\ell \in \{0,1,2,3\}$ and the modulator produces the
output
$$\sqrt{2}\cos\left(2\pi f_c t - \ell\frac{\pi}{2}\right).$$

In differential QPSK (DQPSK), the input dibit Gray-code value
is subtracted (differential, get it?) from the previous transmitted dibit Gray-code value (left-handed folks can use addition if they prefer) to get
the next transmitted dibit Gray code value.  That is, with $\ell$
continuing to stand for Gray code value, we have that
$$a_{\ell}[n] \equiv a_{\ell}[n-1] \mp b_{\ell}[n] \bmod 4. \tag{1}$$
At the demodulator, we have
$$\hat{b}_{\ell}[n]\equiv \hat{a}_{\ell}[n-1] \mp \hat{a}_{\ell}[n] \bmod 4 \tag{2}$$ where the carets denote the demodulated symbols and
it is the job of the project manager to make sure that
the modulator designer and the demodulator designer both use the same
sign ($+$ or $-$) in $(1)$ and $(2)$.  In well-designed DQPSK systems,
there is no explicit (two-bit binary) arithmetic used in $(2)$,
and the DQPSK demodulator determines $b_{\ell}[n]$ from soft-decision
data or more usually, the raw matched-filter/correlator outputs in
the I and Q branches at two successive sampling instants.

Finally, you can apply the same idea to $M$-PSK with $M = 2^n &gt; 4$
with the successive phase shifts being labeled with the Gray-coded representations of the integers $\{0, 1, 2, \ldots, 2^n-1\}$. 
Equations $(1)$ and $(2)$ above get modified to be arithmetic modulo
$M$ but the same ideas work. The receivers are more complicated, though.