title: Using Spark to call out-of-process dll functions
body: <p>We're evaluating wether to use Spark to run our grid calculations and we're having some trouble with a particular use case. Wondering if the community has any good ideas.</p>
  
  <p>We have an extensive library of C++ functions in a dll that we need to use all over the grid. The C++ code base is large, not particularly stable (it falls over frequently) and holds state (thread un-safe). For this reason the dll needs to live out of process. </p>
  
  <p>To handle this we've built a thin zero mq wrapper around the dll and a thin scala zero mq client to handle requests to the dll. We can distribute the zeromq wrapped dll to all the nodes in the grid.</p>
  
  <p>My question is: Is it possible to use Spark to handle this situation? How can I make calls to the zeromq scala client inside an <code>RDD.map()</code> call? How can this get handled on the grid? Do we need to serialise the zeromq client? </p>
  
  <p>Any ideas much appreciated :)</p>