what is the relationship between the reliability of a monte carlo test and the length of the samples tested and the number of errors found?
For each sample, we note $1$ if error and $0$ otherwise. Then, for bit error rate $q$, the tests are independent Bernoulli random variables $\{X_i\}$ with probability $q$.
The estimate is $\hat{p} = \frac{S}{n}$  where
$$S = \sum^n_i X_i \tag{1}$$
and $\mu = \mathbb{E}[X] = np$.
Chernoff bound (see Corollary 5) tells us that for every $0 &lt; t &lt; \mu$,
$$\mathbb{P}\left( \mid S-np\mid \leq \frac{t}{\mu}\mu\right) \geq 1 - 2 e^{\frac{-t^2}{3\mu}} \tag{2}$$
Choose $t=\epsilon np$ with $0 &lt; \epsilon &lt; 1$,
$$\mathbb{P}\left( \mid S-np\mid \leq \epsilon np\right) \geq 1 - 2 e^{\frac{-\epsilon^2 np}{3}} \tag{3}$$
$$\mathbb{P}\left(p(1-\epsilon) \leq  S/n \leq p(1+\epsilon)\right) \geq 1 - 2 e^{\frac{-\epsilon^2 np}{3}} \tag{4}$$
Therefore, $\epsilon$ is relative estimation error and the right hand side of $(4)$ is confidence level which depends on sample size $n$.
For example, if you want estimate a BER within $10^{-3}\times(1 \pm 10\%)$, using sample size $n=8.99 \times 10^5$ will guarantee a confidence level greater than 90%.
Note that choosing another $t$ and/or using another bound may result in a tighter required sample size.