Reporting Services service restart fixes slow reports
I believe problem was that the report definitions no longer matched the stored procedures that were called. After I refreshed all of my reports the problem seems to be resolved.

Unfortunately I can not say for sure that is the final answer as it could be up to 90 days between the last restart and as little as 4 days, so I am still waiting to remove this from my task list. 

These are old reports that were originally in version 2005.