Evauating response to $x(n) = \sin{2\pi\frac{1}{4}n}$ given the system function
You can start from z-transform of sine:

$ Z[sin(2\pi f n)\cdot u(n)](z)= \frac{z \cdot sin(2 \pi f)}{z^2-2z \cdot cos(2 \pi f)+1}$

For your signal you have $f=1/4$ and substituting you obtain:

$ X(z)=\frac{z}{z^2+1} $

The output of the system will be:

$Y(Z)=X(Z) \cdot H(Z) = \frac{z}{z^2+1} \cdot \frac{1}{1-z^{-1}+\frac{3}{16}z^{-2}} $

Computing the inverse transform you will obtain:

$ y(n)=Z^{-1}[Y(z)](n)=\frac{2}{425} ((-64 + 52 i) (-i)^n - (64 + 52 i) i^n - 25 4^{(-n)} + 17 \cdot 3^{(n + 2)} 4^{(-n)})$

That I computed with wolfram alpha but you can solve it from yourself with partial fraction decomposition. Also if appears the $i$ coefficient the succession is not complex because is real for each integer $n \geq 0 $ 

The method used by your book uses the conversion from z-transform to dft ($ z=exp(j 2 \pi f)$ ):

$ X(z) \longrightarrow X(exp(j 2 \pi f)) $

The dft of digital sine is a train of Kronecker  Delta so the only part that remains doing the multiplication $ X(f) \cdot H(f) $ is $H(f)$ calcutated in $f=1/4+k$ and you can compute the idft of $H(exp(j 2 \pi (1/4+k))) $ to obtain the result.

But I think that its solution misses the $k$ that should be $\{0;1/2\}$ because the dft should be symmetric respect $f=0.5$ Nyquist frequency otherwise the idft begin complex. I don't know if the book defines an half dft for real signals. Also phase misses because sin has a phase of $\pi /2$ that it doesn't consider. I don't know exactly what is it doing, maybe it wants work with overlapping effects and the question misses other effects that need to be summed at this to perform the correct result.