title: Is Apache Spark good for lots of small, fast computations and a few big, non-interactive ones?
body: <p>I'm evaluating Apache Spark to see if it's good platform for the following <strong>requirements</strong>:</p>
  
  <ul>
  <li> Cloud computing environment.
  <li> Commodity hardware.
  <li> Distributed DB (e.g. HBase) with possibly a few petabytes of data.
  <li> Lots of simultaneous small computations that need to complete fast (within seconds). Small means 1-100 MBs of data.
  <li> A few large computations that don't need to complete fast (hours is fine). Large means 10-1000 GBs of data.
  <li> Very rarely, very large computations that don't need to complete fast (days is fine). Very large means 10-100 TBs of data.
  <li> All computations are mutually independent.
  <li> Real-time data stream incoming for some of the computations.
  <li> Machine learning involved.
  </ul>
  
  <p>Having read a bit about Spark, I see the following <strong>advantages</strong>:</p>
  
  <ul>
  <li> Runs well on commodity hardware and with HBase/Cassandra.
  <li> MLlib for machine learning.
  <li> Spark Streaming for real-time data.
  <li> While MapReduce doesn't seem strictly necessary, maybe it could speed things up, and would let us adapt if the requirements became tighter in the future.
  </ul>
  
  <p>These are the main <strong>questions</strong> I still have:</p>
  
  <ul>
  <li> Can it do small computations very fast?
  <li> Will it load-balance a large number of simultaneous small computations?
  </ul>
  
  <p>I also wonder if I'm generally not trying to use Spark for a purpose it wasn't designed for, not using the main advantages: MapReduce and in-memory RDDs. If so, I'd also welcome a suggestion for an alternative. Many thanks!</p>