Rectangular 8 QAM bit error rate calculation
Neglecting square terms of the Q function, your expression is correct, but it is the symbol error probability, not the bit error probability:

$$P_s=2.5\,Q\left(\frac{d}{2\sigma}\right)\tag{1}$$

With Gray encoding, the bit error probability is closely approximated by

$$P_b\approx\frac13Ps\tag{2}$$

because there are $3$ bits per symbol, and the most likely symbol errors (i.e. between neighboring symbols) cause only one bit error.

Furthermore, if $N_0$ is the one-sided noise density, then $\sigma=\sqrt{N_0/2}$, and the energy per bit is $E_b=E_s/3$, where $E_s$ is the symbol energy. It remains to express the distance $d$ between the symbols in terms of $E_s$. The energy of the four inner symbols is $d^2/2$. The energy of the corner symbols is $5d^2/2$. So for equally probable symbols, the average symbol energy is

$$E_s=\frac12\cdot\frac{d^2}{2}+\frac12\cdot\frac{5d^2}{2}=\frac{3d^2}{2}\tag{3}$$

which gives

$$d=\sqrt{\frac{2E_s}{3}}=\sqrt{2E_b}\tag{4}$$

Plugging (4) into (1) gives for the symbol error probablity

$$P_s=2.5\,Q\left(\sqrt{\frac{E_b}{N_0}}\right)\tag{5}$$

which, combined with (2), gives for the average bit error probability

$$E_b\approx\frac{5}{6}\,Q\left(\sqrt{\frac{E_b}{N_0}}\right)\tag{6}$$


Note that Eq. (6) for the bit error probability was derived under the assumption that the SNR is relatively high. This assumption is implicit in Eq. (1), where quadratic terms of the Q function were neglected. The exact expression for the symbol error probability is

$$P_s=2.5\,Q\left(\frac{d}{2\sigma}\right)-1.5\,Q^2\left(\frac{d}{2\sigma}\right)\tag{a}$$

For $d/2\sigma\rightarrow 0$ the expression (a) approaches $7/8$. This is a consequence of all symbols being detected with equal probability, and since only one of them is the correct symbol, the symbol error probability must be $7/8$.

The expression (2) for the bit error probability assumes that each symbol error causes only one bit error, which is true if Gray encoding is used and if it is assumed that symbol errors only occur between neighboring symbols, the latter being a reasonable assumption at relatively high SNRs. However, for $d/2\sigma\rightarrow 0$ any symbol will be confused with any other with equal probability. There is only one symbol (the correct one) where there are zero bit errors. There are $3$ symbols causing one bit error, $3$ symbols causing $2$ bit errors, and one symbol causing $3$ bit errors. Since for very bad SNRs all symbols are detected with equal probability, the average number of wrong bits per detected symbol is

$$\frac18\cdot 0+\frac{3}{8}\cdot 1+\frac{3}{8}\cdot 2+\frac18\cdot 3=\frac{3}{2}$$

And since there are $3$ bits per symbol, the average bit error rate for $d/2\sigma\rightarrow 0$ is

$$E_b=\frac{3}{2}\cdot\frac13=\frac12$$

as expected.