Understanding Montgomery's parameterization of elliptic curves
Write $(x_i, y_i) = (X_i : Y_i : Z_i)$, so that $x_i = X_i/Z_i$ and $y_i = X_i/Z_i$, where $Z_i \ne 0$ is arbitrary.  (If you are not familiar with projective coordinates or you like visuals, see an illustration of projective coordinates on a real elliptic curve.)

Let's take the doubling formula for example, where $(x_3, y_3) = (x_1, y_1) + (x_1, y_1) = [2](x_1, y_1)$:

\begin{equation*}
  4 x_1 x_3 ({x_1}^2 + A x_1 + 1) = ({x_1}^2 - 1)^2 \tag{p. 261, second display}
\end{equation*}

so that

\begin{equation*}
  x_3 = \frac{({x_1}^2 - 1)^2}{4 x_1 ({x_1}^2 + A x_1 + 1)},
\end{equation*}

which in projective coordinates is

\begin{align*}
  \frac{X_3}{Z_3}
   &amp;= \frac{\Bigl(\bigl(\frac{X_1}{Z_1}\bigr)^2 - 1\Bigr)^2}
           {4 \frac{X_1}{Z_1}
              \Bigl(\bigl(\frac{X_1}{Z_1}\bigr)^2 + A \frac{X_1}{Z_1} + 1\Bigr)} \\
   &amp;= \frac{{Z_1}^4}{{Z_1}^4}
      \cdot
      \frac{\Bigl(\bigl(\frac{X_1}{Z_1}\bigr)^2 - 1\Bigr)^2}
           {4 \frac{X_1}{Z_1}
              \Bigl(\bigl(\frac{X_1}{Z_1}\bigr)^2 + A \frac{X_1}{Z_1} + 1\Bigr)} \\
   &amp;= \frac{\bigl({Z_1}^2\bigr)^2 \Bigl(\bigl(\frac{X_1}{Z_1}\bigr)^2 - 1\Bigr)^2}
           {4 X_1 Z_1 {Z_1}^2
            \Bigl(\bigl(\frac{X_1}{Z_1}\bigr)^2 + A \frac{X_1}{Z_1} + 1\Bigr)} \\
   &amp;= \frac{\bigl({X_1}^2 - {Z_1}^2\bigr)^2}
           {4 X_1 Z_1 \bigl({X_1}^2 + A X_1 Z_1 + {Z_1}^2\bigr)}.
\end{align*}

From this, we can read off the numerator and denominator:

\begin{equation*}
  X_3 = \bigl({X_1}^2 - {Z_1}^2\bigr)^2
  \qquad\text{and}\qquad
  Z_3 = 4 X_1 Z_1 \bigl({X_1}^2 + A X_1 Z_1 + {Z_1}^2\bigr).
\end{equation*}

Obviously, we can multiply both of them by the same arbitrary nonzero factor too, but there's no need here.

For the addition formula that is not doubling, you're trying to write $[m + n]P = [m]P + [n]P$ in terms of $X$ and $Z$ coordinates for $[m]P$, $[n]P$, and $[m - n]P = [m]P - [n]P$.  That is, you have $x_1 = x([m]P)$, $x_2 = x([n]P)$, and $x_4 = x([m - n]P)$, and you're trying to find $x_3 = x([m + n]P)$.  Use the top equation on p. 261, $$x_3 x_4 (x_1 - x_2)^2 = (x_1 x_2 - 1)^2,$$ and again write it out in terms of $x_i = X_i/Z_i$.

The last part is a matter of tidying to reduce the number of distinct intermediate quantities, e.g. using the observation that

\begin{multline*}
  {X_1}^2 + A X_1 Z_1 + {Z_1}^2
   = {X_1}^2 - 2 X_1 Z_1 + {Z_1}^2 + 2 X_1 Z_1 + (A/4) 4 X_1 Z_1 \\
   = (X_1 - Z_1)^2 + \frac{A + 2}{4} 4 X_1 Z_1
\end{multline*}

to write the whole doubling formula in terms of $X_1 \pm Z_1$ for a total cost of 4A + 2S + 3M.