Inverse-Transpose of Vertex Normals
You are correct, when they are talking about taking the inverse-transpose they are talking about the 3x3 part of the affine transformation leaving the translation since it doesn't affect the normal vector.

Check : Normal / Transforming Normals | Wikipedia

About how or why is it the inverse transpose, a simple explanation, that is also given in the book "Essential Mathematics for Games" is

Suppose we have a Point $P$ and a normal vector $n$ both represented by column matrix. Then we know that
$$
n^T P = 0
$$
This is the same as the dot product.
Now if we apply a transformation matrix $M$ to point $P$, we'll have to apply some sort of transformation matrix $Q$ to $n$ also.
$$
\begin{eqnarray}
(Q n)^T M P &amp; = &amp; 0 \\
n^T Q^T M P &amp; = &amp; 0
\end{eqnarray}
$$

One solution is
$$
\begin{eqnarray}
Q^T M &amp; = &amp; I \\
Q &amp; = &amp; {M^{-1}}^T
\end{eqnarray}
$$
Hence the inverse transpose. This stands for any sort of transformation matrix $M$ be it scaling, rotation etc. Also for pure rotation matrix, since they are orthogonal, the matrix transforming the normals is the same as that of the vertices. as ${M^T}^T = M$