Group delay of $H(\omega)= 1- re^{j \theta}e^{ - j \omega} $
The book's formula is right.

Let $$H(w) = 1 - r e^{j(\theta - w)} = [1-r \cos(\theta - w)] + j [-r \sin(\theta - w)]$$ Since the group delay $\tau$ is the negative of the derivative of the phase of $H(w)$, we first define the phase as:
$$\phi(w) = \tan^{-1}\left( \frac{-r \sin(\theta - w)}{1-r \cos(\theta - w)} \right)$$ 

Using the derivative rule for the inverse tangent as: 
$$\frac {d}{dw} \left( \tan^{-1}(u)\right) = \frac{u'}{1+u^2}$$

We shall compute the derivative of the phase as;
$$ \phi^{'}(w) = \frac{ \left( \frac{-r \sin(\theta - w)}{1-r \cos(\theta - w)} \right)^{'} }{ 1 + \left( \frac{-r \sin(\theta - w)}{1-r \cos(\theta - w)} \right)^2 }$$

which continues as:

$$\begin{align}
\phi^{'}(w) &amp;= \frac{ r \cos(\theta-w) \cdot[1 -r \cos(\theta-w)] + r \sin(\theta-w) \cdot [-r \sin(\theta-w)]}{(1-r \cos(\theta-w))^2 \cdot \left( 1 + \left( \frac{-r \sin(\theta - w)}{1-r \cos(\theta - w)} \right)^2 \right) }\\
\phi^{'}(w) &amp;= \frac{ r \cos(\theta-w) -r^2 \cos^2(\theta-w) - r^2 \sin^2(\theta-w) }{(1-r \cos(\theta-w))^2 \cdot \left( 1 + \left( \frac{-r \sin(\theta - w)}{1-r \cos(\theta - w)} \right)^2 \right) }\\
\phi^{'}(w) &amp;= \frac{ r \cos(\theta-w) -r^2 } { (1-r \cos(\theta-w))^2 \cdot \left( \frac{ (1-r \cos(\theta-w))^2 + r^2 \sin^2(\theta - w)} { (1-r \cos(\theta-w))^2} \right) }\\
\phi^{'}(w) &amp;= \frac{ r \cos(\theta-w) -r^2 } { (1-r \cos(\theta-w))^2 + r^2 \sin^2(\theta - w)}\\
\phi^{'}(w) &amp;= \frac{ r \cos(\theta-w) -r^2 } { 1 - 2r\cos(\theta-w) + r^2 \cos^2(\theta - w) + r^2 \sin^2(\theta - w)}\\
\end{align}
$$

Which finally simplifes to
$$ \phi^{'}(w) = \frac{ r \cos(\theta-w) -r^2 } { 1-2r \cos(\theta-w) + r^2}$$

Then since the group delay is the negative of this we finally have:
$$ \tau(w) = -\phi^{'}(w) = \frac{ r^2 - r \cos(\theta-w) } { 1 + r^2 -2r \cos(\theta-w) }$$

$$ \tau(w) = -\phi^{'}(w) = \frac{ r^2 - r \cos(w-\theta) } { 1 + r^2 -2r \cos(w -\theta) }$$

Which is the same as the book's formula.

Coming to Matlab's output, you may try simplifications. But they are known to produce more verbose outputs than possible with simplifications.