Huge variations on page load times
Seems to be too broad to answer, few pointers that may help you.

Root Cause of problem might be at 


Custom module level which can be drill down with profiling tools as devel profiling which you have already used, another tool can be XHPROF.
Ensure all your custom queries are optmised.
Uninstall all modules which you are not using.
Try to have a comparison of site performance on your local and server, your server might not be allocating required resources.


As per my experience when you have long list of modules installed it causes performance issue especially panels, to overcome you can use caching provided by views module and for panels check if this module does the trick for you.

Having said that Authenticated User Page Caching mdoule will be worth looking at.

Views and panels provide caching