Is there a reason I need to unset CSS files provided by contrib modules if I have CSS aggregation enabled?
As I understand it unsetting CSS stylesheets is to:


reduce the final size of the aggregrated css file(s)
more easily override / implement custom CSS rules


Beyond these 2 reasons, your site probably "uses" all css files defined in the modules you have activated. Unless you need the performance boost from cutting out the extra bloat of X css files, or want to hand-theme certain things completely .... I see little need to unset style sheets.

I've only unset style sheets once .... and it's probably because i'm not making use of drupals multisite capabability where i have 1 site resolving from 2 url domains: http://www.foo.com and https://secure.webhost.com/foo.com/page depending on the domain I use a hook to unset/set a CSS files that fixes image paths in CSS when i'm on a secure url.