Too Many Samples = Unrealistic?
No. Larger sample numbers make the image more accurate. If images appear to lack variation and imperfections, (I assume that's what you mean by clean) you can manually add them. This may include 


Particle systems in the air for dust
Manually added light 'shafts'
scratches on surfaces
fading, or slightly different shades of colors
high amounts of randomness (not weirdness, RANDOMNESS)


All of these can go to make sure your picture look less 'perfect' although most of the time, you don't need to add them.
It all depends on how closely the final will be observed, and how critical your audience is. (or how dirty, being the opposite of clean, you feel comfortable with.