Is there a reason I need to unset CSS files provided by contrib modules if I have CSS aggregation enabled? As this question makes clear, stylesheets provided by modules can be unset.  Some themes, such as Adaptivetheme 3.x, also provide a GUI for unsetting the stylesheets provided by modules.  My question is this: if I have CSS aggregation enabled, is there any benefit to unsetting CSS files of modules I do not use?  Or is Drupal's CSS aggregation sufficiently efficient that I do not need to bother with unsetting any files?
As I understand it unsetting CSS stylesheets is to:


reduce the final size of the aggregrated css file(s)
more easily override / implement custom CSS rules


Beyond these 2 reasons, your site probably "uses" all css files defined in the modules you have activated. Unless you need the performance boost from cutting out the extra bloat of X css files, or want to hand-theme certain things completely .... I see little need to unset style sheets.

I've only unset style sheets once .... and it's probably because i'm not making use of drupals multisite capabability where i have 1 site resolving from 2 url domains: http://www.foo.com and https://secure.webhost.com/foo.com/page depending on the domain I use a hook to unset/set a CSS files that fixes image paths in CSS when i'm on a secure url.