title: Spark-Streaming Kafka Direct Streaming API & Parallelism
body: <p>I understood the automated mapping that exists between a Kafka Partition and a Spark RDD partition and ultimately Spark Task. However in order to properly Size My Executor (in number of Core) and therefore ultimately my node and cluster, I need to understand something that seems to be glossed over in the documentations. </p>
  
  <p>In Spark-Streaming how does exactly work the <strong>data consumption vs data processing vs task allocation</strong>, in other words: </p>
  
  <blockquote>
    <ol>
    <li>Does a corresponding <strong>Spark task</strong> to a <strong>Kafka partition</strong> both read
    and process the data altogether ?</li>
    </ol>
  </blockquote>
  
  <ul>
  <li><p>The rational behind this question is that in the previous API, that
  is, the receiver based, a TASK was dedicated for receiving the data,
  meaning a number tasks slot of your executors were reserved for data
  ingestion and the other were there for processing.  This had an
  impact on how you size your executor in term of cores. </p></li>
  <li><p>Take for example the advise on how to launch spark-streaming with<br>
  <strong><em>--master local</em></strong>. Everyone would tell that in the case of spark streaming, 
  one should put <strong><em>local[2]</em></strong> minimum, because one of the 
  core, will be dedicated to running the long receiving task that never
  ends, and the other core will do the data processing.</p></li>
  <li><p>So if the answer is that in this case, the task does both the reading
  and the processing at once, then the question that follows, is that<br>
  really smart, i mean, this sounds like asynchronous. We want to be<br>
  able to fetch while we process so on the next processing the data is 
  already there. However if there only one core or more precisely to<br>
  both read the data and process them, how can both be done in<br>
  parallel, and how does that make things faster in general.</p></li>
  <li><p>My original understand was that, things would have remain somehow the
  same in the sense that, a task would be launch to read but that the<br>
  processing would be done in another task. That would mean that, if<br>
  the processing task is not done yet, we can still keep reading, until
  a certain memory limit.</p></li>
  </ul>
  
  <p>Can someone outline with clarity what is exactly going on here ?</p>
  
  <p><strong>EDIT1</strong></p>
  
  <p>We don't even have to have this memory limit control. Just the mere fact of being able to fetch while the processing is going on and stopping right there. In other words, the two process should be asynchronous and the limit is simply to be one step ahead. To me if somehow this is not happening, i find it extremely strange that Spark would implement something that break performance as such.</p>