Are there downsides to increasing viewport samples?
The only disadvantage I see is that it will take longer (as you said) and your machine might run hotter due to the increased time taken rendering, which could after a while decrease the life of your battery, fan, GPU etc. The rate at which your cooling system(s) is running is a good indicator of how much the rendering may be slowing down and/or decreasing the life of your components. Just make sure the rendering not causing the cooling system to run excessively high for long periods of time.