Plot the frequency response of MTI delay line canceller using Octave
Magnitude |H(z)| is always defined as an absolute value of your transfer function H(z) values which are complex. What you are plotting is the vector that contains complex numbers. You need to take their absolute value and that will give you magnitude. 

If you call the angle function then you are going to get phase. Transfer function H(z) is always complex. Try to print values of real(H_z) and imag(H_z), then you will see real and imaginary values of $H(z)$. Now magnitude |H(z)| is sqrt(real * real + imag * imag) for every sample...