Deflection of Alpha Particles in Rutherford's model of atom
See, what the Geiger-Marsden-Rutherford experiment achieved was the following: by bombarding (with alpha particles) a one-atom thick gold sheet and counting how many alpha particles passed through, they were able to relate the already known atomic radius with the actual area that could get collided by alpha particles.
Figure 1 shows a sketch of the apparatus.

Figure 1: depiction of the apparatus (courtesy of Wikipedia).
Now follows a simplified rationalisation.
Assume you have an atom-thick sheet of some material.
By knowing the element's molar mass and the mass of the sheet you can calculate how many atoms $n$ are there, and knowing the sheet's surface area $A$ allows you to calculate the atomic surface density $\sigma = n/A$.
Now if the atoms have radius $\rho$, an area $A_\rho = n \pi \rho^2$ of the sheet should be covered by atoms.
This means a ratio
$$r = \frac{A_\rho}{A} = \pi \rho^2 \sigma$$
is actually covered by atoms and $f = 1 - r$ is the ratio of &quot;free space&quot;.
Rutherford actually tested if this made sense by throwing particles at the sheet: those scattered simply hit something, the rest passed through.
Imagine $M$ particles were thrown and $m$ of those passed through.
If $M$ is big enough we can set $f_R = m/M$ as Rutherford's &quot;free space&quot; estimate.
Going backwards we may set the coverage ratio
$$r_R = 1 - f_R = 1 - \frac{m}{M} = \frac{A_R}{A} = \pi \rho_R^2 \sigma$$
were $\rho_R$ above is the measured radius of the collision cross section of an atom.
The experiment has shown that $r_R \ll r$ and thus $\rho_R \ll \rho$, which implies that most of the atomic mass is buried inside the atom.
In fact (Wikipedia),

The diameter of the nucleus is in the range of $1.75$ fm ($1.75 \times 10^{-15}$ m) for hydrogen (the diameter of a single proton) to about $15$ fm for the heaviest atoms, such as uranium. These dimensions are much smaller than the diameter of the atom itself (nucleus + electron cloud), by a factor of about $23,000$ (uranium) to about $145,000$ (hydrogen).

In Rutherford's own words (p. 68),

It was quite the most incredible event that has ever happened to me in my life. It was almost as incredible as if you fired a 15-inch shell at a piece of tissue paper and it came back and hit you. On consideration, I realized that this scattering backward must be the result of a single collision, and when I made calculations I saw that it was impossible to get anything of that order of magnitude unless you took a system in which the greater part of the mass of the atom was concentrated in a minute nucleus. It was then that I had the idea of an atom with a minute massive centre, carrying a charge.
--- Ernest Rutherford

Thus, nuclei are simply very small, but very dense, as stated by Enigma and Ilmari Karonen.