Scala - best API for doing work inside multiple threads
Question: In Python, I am using a library called `futures`, which allows me to do my
processing work with a pool of N worker processes, in a succinct and crystal-
clear way:

    
    
    schedulerQ = []
    for ... in ...:
        workParam = ...  # arguments for call to processingFunction(workParam)
        schedulerQ.append(workParam)
    with futures.ProcessPoolExecutor(max_workers=5) as executor:  # 5 CPUs
        for retValue in executor.map(processingFunction, schedulerQ):
            print "Received result", retValue
    

_(The`processingFunction` is CPU bound, so there is no point for async
machinery here - this is about plain old arithmetic calculations)_

I am now looking for the closest possible way to do the same thing in Scala.
Notice that in Python, to avoid the GIL issues, I was using processes (hence
the use of `ProcessPoolExecutor` instead of `ThreadPoolExecutor`) - and the
library automagically marshals the `workParam` argument to each process
instance executing `processingFunction(workParam)` \- and it marshals the
result back to the main process, for the executor's `map` loop to consume.

Does this apply to Scala and the JVM? My processingFunction can, in principle,
be executed from threads too (there's no global state at all) - but I'd be
interested to see solutions for both multiprocessing and multithreading.

The key part of the question is whether there is anything in the world of the
JVM with as clear an API as the Python `futures` you see above... I think this
is one of the best SMP APIs I've ever seen - prepare a list with the function
arguments of all invocations, and then just two lines: create the
poolExecutor, and `map` the processing function, getting back your results as
soon as they are produced by the workers. Results start coming in as soon as
the first invocation of `processingFunction` returns and keep coming until
they are all done - at which point the for loop ends.


Answer: You have way less boilerplate than that using parallel collections in Scala.

    
    
    myParameters.par.map(x => f(x))
    

will do the trick if you want the default number of threads (same as number of
cores).

If you insist on setting the number of workers, you can like so:

    
    
    import scala.collection.parallel._
    import scala.concurrent.forkjoin._
    
    val temp = myParameters.par
    temp.tasksupport = new ForkJoinTaskSupport(new ForkJoinPool(5))
    temp.map(x => f(x))
    

The exact details of return timing are different, but you can put as much
machinery as you want into `f(x)` (i.e. both compute and do something with the
result), so this may satisfy your needs.

In general, simply having the results appear as completed is not enough; you
then need to process them, maybe fork them, collect them, etc.. If you want to
do this in general, Akka Streams (follow links from
[here](http://akka.io/docs/)) are nearing 1.0 and will facilitate the
production of complex graphs of parallel processing.

