Craft sometimes takes several minutes to respond to a request
I have had this same experience many times, but only with certain hosting providers, so I suspect it has something to do with server configuration.  One thing that I noticed was that saving an entry, for example, would hang, but the entry had indeed been saved pretty much immediately.  

I know it is not my local setup, because my co worked, who works from a different location, had the same issues at the same time.  Next time it happens I will try accessing via IP.