How can I improve running repeating background tasks? I ran into an issue recently, where I have a piece of code that accesses content from an external source. The content generation from this source is quite slow (3-5 seconds), but it only needs to be refreshed every 30-60 seconds.

Currently it's a bit of a Russian roulette. Every minute a user will receive a generally slower page load, while the content is refreshed and cached, and then other people will receive the cached version. Naturally, this is a headache for the users and for our apdex score.

I was trying to find a way to refresh the content in a non-obtrusive way, either automatically via scheduled action sets, or in background process when the user load the page, but I've failed with both. Normally it would not be an issue (simple enough using Elysiacron), but on Pantheon I don't think it's possible to manipulate the crontab in the fashion I require it.

Did anyone face a similar issue on Pantheon so far? Is there any solution that would solve my problem within Pantheon? Can something like this be achieved with AJAX?

I have an idea for a workaround, which would involve a third party machine requesting the part of the module that generates and caches content every 30 seconds, but that is a really ugly and inelegant way to do this.

Thanks for any input regarding this!
Instead of updating a global cache of remote objects (which a visitor might see a stall on while data is being fetched ...)

Simply use 2 caches. 1 cache is what visitors see data from from say a Drupal cache bin. The second cache bin is another private drupal cache bin updated by your module from the remote host. You could make custom cache tables in Drupal or simply use raw MySQL tables without the Drupal API if you're not familiar with it.

Then on occasion do a database table flip (eg, truncate cache2; select cache1.* INSERT INTO cache2;, or sync from 1 Drupal cache bin to the other. This local cache store updating should be faster than the remote server.