What are overlap integrals?
Reduce the problem to one dimension first. The overlap integral between two functions, $f = f(x)$ and $g = g(x)$, is defined by

$$S_{fg} = \int_{-\infty}^\infty [f(x)]^*g(x)\,\mathrm{d}x$$

where $^*$ denotes complex conjugation. $S$ is the common symbol for the overlap integral; the subscript $fg$ indicates that it is the overlap between $f$ and $g$.

So, for example, let's try a simple case: let

$$f(x) = g(x) = \left(\frac{m\omega}{\pi \hbar}\right)^{1/4} \exp\left(-\frac{m\omega x^2}{2\hbar}\right)$$

(the ground state of the quantum harmonic oscillator). This function is real, so the complex conjugation doesn't do anything. We have

$$\begin{align}
S_{fg} &amp;= \int_{-\infty}^\infty \left(\frac{m\omega}{\pi \hbar}\right)^{1/2} \exp\left(-\frac{m\omega x^2}{\hbar}\right)\,\mathrm{d}x \\
&amp;= \sqrt{\frac{m\omega}{\pi\hbar}} \int_{-\infty}^\infty  \exp\left(-\frac{m\omega x^2}{\hbar}\right)\,\mathrm{d}x \\
&amp;= \sqrt{\frac{m\omega}{\pi\hbar}} \sqrt{\frac{\pi\hbar}{m\omega}} \\
&amp;= 1
\end{align}$$

(I used a standard integral, $\int_{-\infty}^\infty \exp{(-ax^2)}\,\mathrm{d}x = \sqrt{\pi/a}$). So this tells you that the overlap of this function with itself is equal to 1. That's not very interesting, because if you substitute $g(x) = f(x)$ into the original definition, you'll find that that is simply the normalisation condition for a wavefunction. I just wanted to use it as an illustration.

In general, the overlap between two normalised states ranges from 0 (when they are orthogonal) to 1 (when they are identical). This applies to three dimensions as well.



Of course, the notion of overlap is usually talked about in a three-dimensional context. So, all you have to do is to generalise the definition to three dimensions. Let's say we have two functions $\psi_1(x,y,z)$ and $\psi_2(x,y,z)$. Then we simply have

$$S_{12} = \int\limits_{-\infty}^\infty \int\limits_{-\infty}^\infty \int\limits_{-\infty}^\infty \psi_1^*\psi_2 \,\mathrm{d}x\,\mathrm{d}y\,\mathrm{d}z$$

or in more concise notation

$$S_{12} = \int \psi_1^*\psi_2 \,\mathrm{d}\tau$$

where $\mathrm{d}\tau$ is the volume element (note that Wikipedia denotes it as $\mathrm{d}V$) and the limits of integration are assumed to be over all space.

This won't get you anywhere with atomic orbitals, though. Orbitals are frequently expressed in terms of spherical coordinates. So, you need to know what the volume element in spherical coordinates is, and that is:

$$\mathrm{d}\tau = r^2\sin\theta\,\mathrm{d}r\,\mathrm{d}\theta\,\mathrm{d}\phi$$

(note again that Wikipedia can be confusing sometimes, because the symbols used for spherical coordinates are different: $\theta$ and $\phi$ in chemistry have exactly the opposite meaning from $\theta$ and $\phi$ in mathematics.)

The limits of the integral are still "over all space", but in spherical coordinates this is expressed differently. $r$ takes values from zero to infinity, $\theta$ from $0$ to $\pi$, and $\phi$ from $0$ to $2\pi$. So:

$$S_{12} = \int\limits_0^{2\pi} \int\limits_0^\pi \int\limits_0^\infty \psi_1^*\psi_2 r^2 \sin\theta \,\mathrm{d}r\,\mathrm{d}\theta\,\mathrm{d}\phi$$

All you need to do now, is to find the form of the function, plug it into the integral, and evaluate the integral. Warning: That's best done by a computer.