What causes very long initial visit load times intermittently?
If users experience very long response time for initial visit, but then next visits are fast - then it definitely could be related to the cron (as suggested by Geoff).

Here is a quote from official cron documentation:


  It [internal cron] works by checking at the end of each Drupal request to see when cron last ran and, if it has been too long, processing the cron tasks as part of that request. The two down-sides are (1) cron tasks will only run when Drupal is processing requests; and (2) the 'weight' (processing and memory) of running the cron tasks will be added to some arbitrary unknown page request, which may slow down those requests, and has the potential to exceed memory limits on a complex site.


To prevent this, configure external cron job that will trigger Drupal cron jobs more often that 3 hours (my suggestion is to run external cron every hour).

In addition you can configure internal cron to run less often by configuring it here /admin/config/system/cron.