Does inactive modules can make my drupal site slower?
I don't understand Jance's response.  He says "it won't hurt", but then says these disabled modules are sucking up resources?  

Saying inactive modules don't impact performance is not completely accurate... but it certainly is less important than the number of enabled modules.

While the hooks of an inactive module with code left in sites/all/modules aren't loaded when a specific form or page is loaded, all .info files from any code active or inactive are scanned when loading the admin/modules and admin/themes.  Drush also scans all directories for Drush related includes in profiles and sites... regardless of what's active.  Inactive modules aren't going to cause as much of a hit to CPU and memory as an enabled module, but having unused code in sites/all/modules isn't a good thing. Best practice would be to only install modules on a test site.  Never install a module on a live site just to test it.  Even when uninstalled, modules can leave a lot of cruft behind that can impact performance.

Modules that were once installed, but are now inactive could create tables and define variables add to the cruft at the database layer.  Even if you uninstall the module, the uninstall process relies on the developers who wrote the module to clean up after themselves.   Modules that use Features often leave even more cruft behind.  Disabling a Feature "module" does not remove the content types, taxonomies, fields, views, blocks, etc that it created.  Even uninstalling and deleting the code doesn't always clean up after a Feature.    

I realize that not everyone can afford the type of dev -> test -> live workflow you get with a Drupal specific host like Acquia or Pantheon or has the technical skill to run a local version of their site, but if you don't have that type of testing environment be prudent about the modules you enable then uninstall.

Alas, hindsite is 20/20... for modules you've already enabled in the past that you've decided not to use, uninstall them and remove the code from sites/all any module.  This won't improve performance as much as reducing the number of modules enabled, but leaving this code certainly won't help performance or your ability to maintain the site.