Fully homomorphic encryption with polynomials
I missed some pitfalls?


The most obvious one is that the secret key $x_0$ is recoverable with a single known plaintext/ciphertext pair.  That is, if we know both $m$ and $Enc(m) = a_{p-2}x^{p-2} +\  ... \ + a_1 x + a_0$, then we know that $x_0$ is a root of the polynomial $$a_{p-2}x^{p-2} +\  ...\  +\  a_1 x + (a_0 - m)$$
Roots of polynomials modulo $p$ can be efficiently recovered; this will give us the value $x_0$ (and possibly a few other false hits).

The other practical issue is that your ciphertexts consists of $p-1$ values; unless $p$ is tiny (e.g. no more than $2^{40}$), then you have impractically large ciphertexts.