Smooth Complexity of the Nonnegative Permanent
Lipton (New directions in testing, 1991) showed that if the permanent is easy 
for most matrices, then it is easy for all matrices. I do not know an online version but you can find the result in many lecture notes, for instance here:
http://www.cse.cuhk.edu.hk/~andrejb/courses/f07-80240233/notes/lec16.pdf
There are improvements by Gemmel and Sudan (IPL 43(4): 169-174. 1992).
So the permanent is hard on the average for the uniform distribution.
For a smoothed polynomial time algorithm you have to choose the distribution in such a way that this average-case hardness is circumvented.