BER performance of a space-time code over Rayleigh channel model
The classic approach is to evaluate the union bound of the pairwise error probability, taken over the code matrices.

If $X_0$ and $X$ are two different code matrices, then their pairwise error probability is the probability that the receiver mistakes $X$ for $X_0$, when $X_0$ is transmitted. It is written $P(X_0 \rightarrow X)$.

Assuming all code matrices are transmitted with equal probability, and that the code is $\mathcal{X}$, then the average error rate is upper bounded by
$$ P_{\text{err}} \leq \frac{1}{|\mathcal{X}|} \sum_{X_n \in \mathcal{X}} \sum_{X_k \in \mathcal{X},k \neq x} P(X_n \rightarrow X_k) $$
which is just an application of the union bound.

The calculation of BER from $P_{\text{err}}$ for specific channels can get complicated and in fact in many cases no closed form formula can be found. I would suggest the book "Space-Time Block Coding for Wireless Communications" by Larsson and Stoica for more details, in particular section 4, "Error Probability Analysis".