Hill cipher is not perfectly secure
One possibility for what you might be missing: normally the same key (the same matrix) is re-used to encrypt many messages.  So now try counting the total entropy in $M$ length-$N$ messages, and the entropy in a $N\times N$ matrix, and compare what happens when $M$ gets large....

Another possibility you might be missing is the consequences of the fact that you are working modulo $26$.  I think you need to calculate out $p(y|x)$ carefully.  You might start by considering what happens when $N=1$.  Can you calculate the value of $p(y|x)$ and $p(x|y)$, for all possible values of $x,y$?  Maybe write a program to do it, or do the case analysis so you don't have to consider $26\times 26$ cases exhaustively.  I think you'll find that is enough to let you figure out what's going on.

I don't want to say more, because this is an exercise, and should solve your own exercise.