why difference between this example and matlab result
When you use Matlab's function butter, all frequencies need to be normalized by $f_s/2$, so your second call to the function butter is the correct one, the first one is wrong. And, as usually, Matlab is correct. The difference between the solution in your book and Matlab's solution is that the function butter uses prewarping to guarantee that after the bilinear transformation, the resulting cut-off frequency of the digital filters equals the specified cut-off frequency. For whatever reason, the solution in your book doesn't take the frequency warping of the bilinear transform into account, so the resulting cut-off frequency of the digital filter does not exactly equal the desired $\omega_c$.

The frequency warping caused by the bilinear transform used in your book maps the cut-off frequency $\omega_c$ of the analog filter to the following cut-off frequency of the resulting digital filter:

$$\tilde{\omega}_c=\frac{2}{T}\arctan\left(\frac{\omega_cT}{2}\right)=2\pi\cdot 2906.8\tag{1}$$

instead of $\omega_c=2\pi\cdot 3000$.

You can check the actual cut-off frequencies of the digital filters using the following Matlab commands:


wc = pi/5;                          % normalized cut-off frequency
wc2 = 2*atan((2*pi*3000)/30000/2);  % normalized warped cut-off frequency
[b,a] = butter(2,wc/pi,'low');      % Matlab's solution
b0 = 0.063964*[1,2,1];              % solution from book
a0 = [1,-1.168261,0.424118]
abs(freqz(b,a,[0,wc])).^2           % check |H|^2 at DC and at wc;
                                    % (should be [1,0.5])
abs(freqz(b0,a0,[0,wc])).^2         % [0.99999   0.46637]
abs(freqz(b0,a0,[0,wc2])).^2        % [0.99999   0.49999]


This shows that the filter in the book indeed has cut-off frequency $\tilde{\omega}_c$ given in $(1)$ instead of $\omega_c$.

If you don't understand the frequency warping of the bilinear transform, I'd suggest you study chapter 11 of this excellent and free book by S.J. Orfanidis.