title: How to fix data locality in Spark Streaming?
body: <p><strong>Spark version:</strong> 1.6.2</p>
  
  <p><strong>Spark configuration:</strong></p>
  
  <ul>
  <li><p>executors: 18</p></li>
  <li><p>memory per executor: 30g</p></li>
  <li><p>cores x executor: 2</p></li>
  </ul>
  
  <p><strong>Kafka configuration:</strong></p>
  
  <ul>
  <li>partitions: 18</li>
  </ul>
  
  <p><strong>Context:</strong></p>
  
  <p>I have a Spark-streaming pipeline that is using the new Direct approach from Kafka instead of receivers. This pipeline receives data from kafka, transforms the data, and inserts to Hive. We release that the trigger used as action to insert the data has a locality level equal to <code>RACK_LOCAL</code>, instead of <code>PROCESS_LOCAL</code> or <code>NODE_LOCAL</code>. Moreover, the input size column shows network information instead of memory. I have set the config variable <code>spark.locality.wait=30s</code> to test if spark waits enough to select the best locality mode, but none improvements has been shown. </p>
  
  <p>Considering that the <em>kafka partitions = num partitions in spark</em> are equal, I thought that data was processed in the same executor which execute the action at the end, but I am not sure about that watching the results.</p>
  
  <p><img src="https://i.stack.imgur.com/edryI.png" alt="Spark task screenshot">
  <img src="https://i.stack.imgur.com/vg1lq.png" alt="DAG"></p>
  
  <p>Does someone the idea how to force the task to <code>NODE_LOCAL</code> or <code>PROCESS_LOCAL</code>??</p>