How long should a CiviCRM / Drupal upgrade take?
Based only what you have said so far 40 hours is excessive and 4 hours is more reasonable. But some further discovery or prior knowledge of the kind of customizations you have in your site might make 40 hours a reasonable quote. For example, needing to upgrade a few extensions so they are available for 4.6 and do some advanced theming might require 40 hours.

Our upgrade process involves:


Reviewing the standard native CiviCRM extensions and CMS modules/plugins used and ensuring they are available for the new CiviCRM version,
Finding and analyzing the native CiviCRM extension built just for the site as well as the custom CMS modules/plugins developed just for the site - they may need adjustment, perhaps significant, 
Reviewing whether there is custom code and / or templates overriding CiviCRM or even using jQuery to adjust the DOM - this is often where some developer time is required during an upgrade,
Reviewing the theme for CiviCRM customizations and noting they will need to be checked post-upgrade - the more extensive the more risk of breakage during the upgrade,
On Drupal sites, checking for tweaks and customizations within the views using CiviCRM, or any non-views blocks etc that pull in CiviCRM data,
On multilingual or non-English sites, checking for the availability of translations for CiviCRM, native extensions, and CMs modules/plugins,
On a complex upgrade this is where we would provide the scope and estimate to the client.


In terms of the actual upgrade process, it involves:


Backing up the site and restoring it locally, 
Running the upgrade process,
Addressing issues identified in 1 to 6 above, 
QA of result,
Usually deploying site to staging for signoff by other JMA staff and client (skipped for simple upgrades)
After signoff, upgrading production site, usually by taking it offline during night in primary site timezones, doing the upgrade, deploying the changes from 9 to 11 above, and putting it back online.


To summarize the factors affecting the cost of upgrades:


In browser CiviCRM configuration changes like creating custom fields and profiles don't require extra work to upgrade.
More references to pages in CiviCRM from the CMS don't create extra work on upgrades.
The use of each additional native extension from the CiviCRM extensions directory and standard downloaded CMS module/plugin that interacts with CiviCRM slightly increases the risk of extra work.
The more code in extensions, modules/plugins, themes and especially custom PHP and Smarty template override files developed just for this site the more likely that developer time will be needed, and that more of it will be needed.