What causes waves to form the characteristic "breaking" shape as they approach the shoreline?
The physical process you describe is known as wave shoaling. 

At the basic level, waves propagating into shallow water become shorter and higher, and consequently, steeper. In shallow water, the water particles near the crest move forward faster than those below them. Similarly, the particles near the trough move backward faster than those above them. This causes strong shearing of the near-surface body of water, eventually forming a plunging breaker, or a surf wave. For small-slope (linear) and inviscid (no friction) waves, the above is a consequence of the bottom boundary condition for the water velocity to be zero at the sea floor. 

There are two fundamental and related properties of water waves that contribute to shoaling. One is the wave frequency $\omega$ remaining constant as the depth $d$ decreases. Think of this as the conservation of wave crests at any fixed point. However, the wavenumber $k$ (wavelength $\lambda$) must increase (decrease) with decreasing depth, as per the dispersion relationship of water waves (neglecting the effects of viscosity):

$$\omega^{2} = gk\tanh{(kd)}$$

where $g$ is gravitational acceleration. In shallow water, the dispersion relationship reduces to:

$$\omega = \sqrt{gd}k $$

and phase speed $C_p$ and group velocity $C_g$ are both proportional to the square root of the water depth:

$$ C_p = \dfrac{\omega}{k} = \sqrt{gd} $$

$$ C_g = \dfrac{\partial\omega}{\partial k} = \sqrt{gd} $$

Individual wave crests propagate with phase speed $C_p$. Wave groups and wave energy propagate with group velocity $C_g$. Thus, waves entering shallow water become shorter and slower. The second important property leading to shoaling is the conservation of wave energy flux, which is proportional to the group velocity and wave energy: 

$$\dfrac{\partial(C_{g}E)}{\partial x}=0$$

Because the group velocity decreases, the wave energy (read: height) must increase (locally). This causes the waves to grow in height as they enter shallow water. As pointed out by @IsopycnalOscillation in a comment above, the separation of water particles from the wave crests happen because the individual orbital velocities at the top of the crest exceed by far the phase speed of the wave.

Although bottom friction is significant and non-negligible in shallow water, it does not cause shoaling. Mathematically, wave shoaling can occur for completely inviscid and small-slope (linear) waves that propagate into water of decreasing depth.