mathematical expression to detect modulated data within a vector
The same as you'd do for a scalar reception:

Build a stochastic model for your transmit vectors
Build a stochastic model for your noise
from 1. and 2., build a stochastic model for your received vectors
Define a metric on the error statistics that you want to minimize
from 3. and 4., derive a decider

For example:

$x$ are points from an amplitude-phase shift keying modulation $\mathcal X$, with two different amplitudes, and 4 points on the inner &quot;circle&quot;, as well as 6 on the outer, and the inner points being chosen twice as often as the outer points; points on the same amplitude circle are equiprobable. 
From that follows a statistic for $\vec y_1=(y_{11},y_{12})^T$, call it $f_{y_{11},y_{12}}(\xi_1, \xi_2)$.
In many cases this is relatively easy, since the possible points are discrete, in others you'll have to employ more complicated models e.g through random variable pdf transformations using the Jacobi determinant.
Noise $\vec n = (n_1,n_2)^T$ is uniform in phase over $[0,\pi/2]$, and exponentially distributed in amplitude, strongly stationary, white and independent from $\vec y$. That means it can be described by a single $f_n(\chi)$.
$\vec r = (y_{11}+n_1, y_{12}+n_2)^T$. If you modeled your noise to be independent from signal in 2., then the noise pdf is just the convolution of $f_{\vec y}$ and $f_{n}$.
You define that you want to minimize the symbol error probability for every receive point. Other choices might include minimizing bit error rate locally, or symbol error rate globally under constraints (e.g. complexity).
You hence decide to build an MAP decider that gives you an estimate $\hat x$ that gives you $\hat x_{\vec r} = \operatorname*{\arg\max}\limits_x f(\vec r|x)$. So, you write down the 2D density, and find a maximum.