How to soft decode DQPSK?
Two successive symbols in the demodulator are $Z_1 = (X_1,Y_1)$ and $Z_2 =(X_2,Y_2)$ 
where $X$ is
the output of the I branch and $Y$ the output of the Q branch of the receiver. The 
hard-decision DBPSK decision device considers the question:


  Is the new symbol $Z_2$ closer to the old symbol $Z_1$ or to the
  negative $-Z_1$ of the old symbol?


and thus compares

$$(X_2-X_1)^2 + (Y_2-Y_1)^2 \gtrless (X_2+X_1)^2 + (Y_2+Y_1)^2$$

which can be simplified to a sign comparison on $\langle Z_1,Z_2\rangle = X_1X_2+Y_1Y_2$.
Note that this is essentially asking 


  Are the two vectors $Z_1$ and $Z_2$
  are pointing in roughly the same direction (in which case the inner product 
  or dot product is positive) or in roughly opposite direction (in which case
  the dot product is negative)?


A third viewpoint thinks of $Z_1$ and $Z_2$ as
complex numbers and asks


  Is $\text{Re}(Z_1Z_2^*) = X_1X_2+Y_1Y_2$ positive or negative?


The soft decision decision device simply
passes on the exact value of the dot product to the soft decision decoder
which may opt to quantize dot products that are very large in magnitude
into hard decisions and continue waffling on the rest.  This is what 
the decision rule stated in the OP's question is, where large is taken as
exceeding $1$ in magnitude.

In DQPSK, the encoding uses one of two conventions:


the signal phase is delayed by $0, \pi/2, \pi, 3\pi/2$ according
as the dibit to be transmitted is $00, 01, 11, 10$
the signal phase is advanced by $0, \pi/2, \pi, 3\pi/2$ according
as the dibit to be transmitted is $00, 01, 11, 10$


Note that a DQPSK signal is not the sum of two DBPSK signals
modulated on phase-orthogonal carriers, but the I and Q bits
jointly affect the net carrier phase.

For demodulating a DQPSK signal, the decision device needs to ask


  Which of the four symbols $Z_1,\quad jZ_1 = (-Y_1,X_1),\quad -Z_1,\quad -jZ_1 = (Y_1,-X_1)$
  is $Z_2$ closest to? 


Thus, in addition to the comparison

$$(X_2-X_1)^2 + (Y_2-Y_1)^2 \gtrless (X_2+X_1)^2 + (Y_2+Y_1)^2$$

it is necessary to compare

$$(X_2+Y_1)^2 + (Y_2-X_1)^2 \gtrless (X_2-Y_1)^2 + (Y_2+X_1)^2$$

which works out to looking at $\text{Im}(Z_1Z_2^*)$ in
addition to $\text{Re}(Z_1Z_2^*)$ and making the decision
according as which quantity has the largest magnitude
and the sign of the largest magnitude.  The details
of how the soft-decision decoder uses the decision
statistic $Z_1Z_2^* = (\text{Re}(Z_1Z_2^*), \text{Im}(Z_1Z_2^*))$
will determine how these numbers are further
massaged.