Permanents - Approximation and connection to integer factorization
Thanks for clarifying your earlier question. 

In this case, we would indeed say the Permanent is in $P$. First, since addition and multiplication on $O(n^k)$ size numbers can be done in $n^k \log^{O(1)} n$ time, the $O(n^2)$ arithmetic operations would take less than $O(n^{k+3})$ time to execute. If $k$ is independent of $n$ and the numbers themselves, this is polynomial time.

Secondly, if you are computing the Permanent over the integers, then in the "hard case" you may not only assume that the integers are non-negative, but also you may assume that all integers are just $0$ and $1$! This is commonly known as Valiant's theorem. For information on how to prove this, see:

http://en.wikipedia.org/wiki/Permanent_is_sharp-P-complete#01-Matrix 

This means that restricting to a non-negative matrix with bounded entries is not really a restriction of the Permanent problem. A polynomial time algorithm for this version of the problem would result in a polynomial time algorithm for the Permanent over any integer matrix.