What is the theoretical probability of error for OOK transmission?
It is not clear whether you want a derivation of the formula for the error probability of a wireless OOK system, or ideas about how the formula might apply to an optical communication system.

For a wireless OOK system, there are two main types of receivers called 
coherent receivers and noncoherent receivers and these have different error probabilities. Neither error probability formula is suitable for application to an optical communication system.

Wireless OOK: The wireless OOK system model is that if a $1$ has to be be transmitted, the transmitted signal is an RF pulse of duration $T$ (the ON part of the keying) while if a $0$ has to be transmitted, then no signal is transmitted (the OFF part of the keying).  Assuming no signal distortion in the channel, then in the interval $0 \leq t \leq T$, the
receiver input can be expressed as
$$r(t) = \begin{cases} A \cos(2\pi f_c t+\theta) + n(t), &amp; 1 ~~\text{transmitted},\\ n(t), &amp; 0 ~~\text{transmitted}, \end{cases}
\tag{1}$$
where $n(t)$ is white Gaussian noise (with two-sided power spectral
density $\frac{N_0}{2}$) that is independent of the signal and
whether $1$ or $0$ is transmitted.  It is generally assumed that $1$ and $0$ are equally likely to be
transmitted.

Coherent receiver: A coherent receiver know the exact value of
the carrier frequency $f_c$, the received phase $\theta$ and the signal
timing, that is, it knows both the duration and location of the pulse
on the time axis. To understand how it knows all this will take us too
far away from this discussion. The receiver constructs the signal
$s(t) = \cos(2\pi f_ct+\theta)$ in a local oscillator and then correlates
this locally generated signal with $r(t)$, that is, it computes
$$X = \int_0^T r(t) s(t)\ \mathrm dt = \begin{cases}\frac{AT}{2} + N,
&amp;1 ~~\text{transmitted},\\ N, &amp; 0 ~~\text{transmitted},\end{cases}
\tag{2}$$
where $N$ is a zero-mean Gaussian random variable with variance
$\frac{N_0T}{4}$. The receiver compares $X$ with the threshold 
$\Theta = \frac{AT}{4}$ and decides
at a $1$ or $0$ is transmitted according $X$ is larger than or smaller
than the threshold. The probability of an erroneous decision when a
$0$ is transmitted is $P\{N &gt; \Theta\}$, 
the probability that $N$ exceeds $\Theta$, while the probability of
an erroneous decision when a $1$ is transmitted is $P\{N &lt; -\Theta\}$
which is the same as $P\{N &gt; \Theta\}$
We readily get that 
$$P_e = P\{N &gt; \Theta\} = Q\left(\frac{AT}{4\sqrt{N_0T/4}}\right)
= Q\left(\sqrt{\frac{A^2T}{4N_0}}\right) \tag{3}$$
where $Q(z)$ is area under the right tail of the standard Gaussian
density to the right of $z$. It is, I believe, built in to MATLAB's
library; no need to fool around with erf or erfc.

It is possible to massage the right side of $(3)$ a little to give
a different perspective on the matter. Note
that $E_b$, the received signal energy per bit is $\frac{A^2T}{4}$
(because for half of the transmitted bits, there is no signal energy received).  Thus, we can re-write the right side of $(3)$ as
$$P_e = Q\left(\sqrt{\frac{E_b}{N_0}}\right)$$
which expression might be familiar to some readers.

Noncoherent receivers: Coherent receivers for OOK are difficult
to implement because the estimation of the RF phase $\theta$ is not
easy when the incoming signal (from which the phase is estimated)
keeps disappearing from time to time.  A noncoherent receiver is easier
to implement, and is sometimes referred to as a square law detector
because the earlier implementations merely measured the passband energy
and decided on $0$ or $1$ according as this energy is small or large.
More modern methods use I-Q signals, but regardless of the method, the
analysis is messy. The error probability expressions involve things like
Marcum's Q function which those knowledgeable about radar will likely
know. It is best to hide this stuff behind a veil.

Optical OOK communication:  This is completely different and the
system model is quite different. In particular, in contrast to 
coherent wireless OOK communications, the probability of error
when a $1$ is transmitted is quite different from the error
probability when a $0$ is transmitted. Indeed, most of the errors are
mistaking $1$'s for $0$'s, and it is hardly ever the case that a
$0$ is mistaken for $1$. The reason is that the detector is essentially
a photon counter which can fail to detect a photon sometimes, but hardly
ever sees photons where none exist. As such, for the OP's simulation
of an optical communication system, the results from the analysis of
wireless RF communication systems are totally inapplicable.