title: Spark streaming as event processing/handling solution (micro-services)
body: <p>Spark batch processing is bringing a lot of value to our business since it's very easy to scale horizontally (we use AWS EMR with YARN).</p>
  
  <p>However, a new challenge has arisen as our newest proprietary solution follows a micro-service architecture. So far, there are <strong>~230 micro-services acting as a producer</strong> where the events are stored in Kafka (that means ~230 Kafka topics).</p>
  
  <p>Although we've managed to validate the use of Spark Streaming as event processing to build the lastest state of the objects, am I right saying that I'll need one spark streaming app per Kafka topic (so, ~230 apps)?</p>
  
  <p>If so, our cluster with <strong>48 vCPU</strong> and <strong>192GiB of Memory</strong> can only handle <strong>52 stream processing apps concurrently</strong>. That sounds way too little since those apps (which need to run 24 hours) don't do much as they simply pull events every 5 seconds and perform CRUD operations against our Data Store.</p>
  
  <p>Am I miss using Spark streaming? What other approach or framework would you take/use?</p>