Non-polynomial Z-transform
First of all - I myself am not a pro in Control theory, but a mathematician - so I write what I think might be what your prof means.

Part where i am quite certain:

Not to be polynomial means, that there exists NO polynomial which describes the transfer function. E.g. let $H$ be your transfer function, then there exists no $n\in\mathbb{N}$ and $p\in P^n[X]$ such that $$H(s) = \sum_{i = 0}^{n}p_is^i$$ 
holds.
Examples for functions which are not polynomial are: exponential functions (e.g $e^s$) or trigonometric functions (e.g. $sin(s)$), but there are many more.



The next part is where I am not 100% sure, because it is not my main subject.

I assume you mean the inverse Z-Transform when saying anti transform. I do not know too much about the z-transform, but i think it is the discrete-time equivalent to the laplace-transform and therefore:

Let $H$ be a transfer function. Then $H$ is the Laplace-transform of an impulse response. Thus the inverse Laplace transform (i.e. $h$) exists. (The stability however is something different.)

But if you want to check for a given "transfer function" (an arbitrary function in the "s"-Domain) if there exists an impulse response corresponding to this "transfer function", then I think you need the inverse Laplace transform to converge, i.e.
$$ \frac{1}{2\pi i} \lim_{y\rightarrow\infty}\int_{\gamma-iy}^{\gamma+iy}e^{st}H(s)\,ds$$ to exist for some $\gamma\in\mathbb{R}$.
The integral then is your impulse answer. There exist non-polynomial functions for which this integral exists. Thus, there are non-polynomial transfer functions which correspond to a system. Many many authors in literature however assume only rational valued transfer functions... i really do not know why.

If this answer is NOT correct (because for the discrete-time case there is something mayor different), or does not help, please say so and apology. As I said, this is not my main subject.