Scaling thousands of automated forecasts in R Hello and thanks in advance!  I'd like some advice on a scalability issue and the best way to resolve.  I'm writing an algorithm in R to produce forecasts for several thousand entities.  One entity takes about 43 seconds to generate a forecast and upload the data to my database.  That equates to about 80+ hours for the entire set of entities and that's much too long.

I thought about running several R processes in parallel, possibly many on a few different servers, each performing forecasts for a portion of total entities.  Though that would work, is there a better way?  Can Hadoop help at all?  I have little experience with Hadoop so don't really know if it can apply.  Thanks again!
If you're working with R language, I would suggest first to try use R ecosystem's abilities to parallelize the processing, if possible. For example, take a look at packages, mentioned in this CRAN Task View.

Alternatively, if you're not comfortable or satisfied with the approaches, implemented by the above-referred packages, you can try some other approaches, such as Hadoop or something else. I think that a Hadoop solution would be an overkill for such problem, considering the learning curve, associated with it, as well as the fact that, as far as I understand, Hadoop or other MapReduce frameworks/architectures target long-running processes (an average task is ~ 2 hours, I read somewhere recently). Hope this helps.