Cepstrum Calculation of Rational Function H(z)
We're looking for a series representation

$$\log H(z)=C(z)=\sum_{n=-\infty}^{\infty}c[n]z^{-n}\tag{1}$$

which converges in an annular region $r_1&lt;|z|&lt;r_2$ with $0&lt;r_1&lt;1&lt;r_2$.

Note that the poles as well as the zeros of $H(z)$ lead to singularities of $C(z)$. All poles and zeros of $H(z)$ inside the unit circle contribute to the right-sided part of $c[n]$, whereas the poles and zeros of $H(z)$ outside the unit circle contribute to the left-sided part of $c[n]$. Consequently, a minimum-phase transfer function $H(z)$ has a right-sided cepstrum.

The given function $H(z)$ has all its poles inside the unit circle ($z^{\infty}_{1,2}=\pm 0.8$, $z^{\infty}_{3}=0$), one of its zeros is inside the unit circle ($z^{0}_{1}=0.5$), and two zeros are outside the unit circle ($z^{0}_{2,3}=\pm 2j$). Consequently, its complex cepstrum $c[n]$ is a two-sided sequence.

In order to find the sequence $c[n]$ we make use of the well-known Mercator series:

$$\log(1-z)=-\sum_{n=1}^{\infty}\frac{z^n}{n},\qquad |z|&lt;1\tag{2}$$

From $(2)$ it is clear that terms of the form

$$\log(1-az^{-1})=-\sum_{n=1}^{\infty}\frac{a^n}{n}z^{-n},\qquad |z|&gt;|a|\tag{3}$$

contribute to the right-sided part of $c[n]$, whereas terms of the form

$$\log(1-bz)=-\sum_{n=1}^{\infty}\frac{b^n}{n}z^{n}=\sum_{n=-\infty}^{-1}\frac{b^{-n}}{n}z^{-n},\qquad |z|&lt;\frac{1}{|b|}\tag{4}$$

contribute to the left-sided part of $c[n]$. Note the argument can be easily generalized to higher powers of $z$ on the left-hand sides of $(3)$ and $(4)$.

The term $(1+4z^{-2})$ in the numerator of the given function $H(z)$ corresponds to two zeros outside the unit circle, and, consequently, it contributes to the left-sided part of $c[n]$. Hence, according to $(4)$, it must be rewritten in terms of positive powers of $z$:

$$(1+4z^{-2})=4z^{-2}(1+0.25z^2)\tag{5}$$

We have

$$H(z)=z^{-2}\frac{4(1-0.5z^{-1})(1+0.25z^2)}{(1-0.64z^{-2})}=z^{-2}\tilde{H}(z)\tag{6}$$

I think that due to the linear-phase term $z^{-2}$ in $(6)$ we actually cannot compute the complex cepstrum of $H(z)$, but we can compute the complex cepstrum of $\tilde{H}(z)$, which is just a shifted version of the original function $H(z)$.

From $(6)$ we obtain

$$\log \tilde{H}(z)=\log(4)+\log(1-0.5z^{-1})+\log(1+0.25z^{2})-\log(1-0.64z^{-2})\tag{7}$$

Using $(3)$ and $(4)$ we get

$$\log \tilde{H}(z)=\log(4)-\sum_{n=1}^{\infty}\frac{(0.5)^n}{n}z^{-n}+\sum_{n=-\infty}^{-1}\frac{(-0.25)^{-n}}{n}z^{-2n}+\sum_{n=1}^{\infty}\frac{(0.64)^n}{n}z^{-2n}\tag{8}$$

which is equivalent to

$$\log \tilde{H}(z)=\log(4)-\sum_{n=1}^{\infty}\frac{(0.5)^n}{n}z^{-n}+\sum_{n=-\infty\\n\textrm{ even}}^{-1}\frac{2(-0.5)^{-n}}{n}z^{-n}+\sum_{n=1\\n\textrm{ even}}^{\infty}\frac{2(0.8)^n}{n}z^{-n}\tag{9}$$

The region of convergence of the series representation $(9)$ is $0.8&lt;|z|&lt;2$, which includes the unit circle, as required by the definition of the complex cepstrum.

From $(9)$ we can directly write down the complex cepstrum of $\tilde{H}(z)$:

$$\tilde{c}[n]=\begin{cases}\log(4),&amp;n=0\\
\frac{1}{n}\left[2(0.8)^n-(0.5)^n\right],&amp;n&gt;0 \land n\textrm{ even}\\-\frac{1}{n}(0.5)^n,&amp;n&gt;0 \land n\textrm{ odd}\\\frac{2}{n}(-0.5)^{-n},&amp;n&lt;0 \land n\textrm{ even}\\0,&amp;n&lt;0\land n \textrm{ odd}\end{cases}\tag{10}$$