title: Spark 2.X: number of tasks set by a Spark Job when querying a Hive Table with Spark SQL
body: <p>Plenty of questions have been already posed about the number of Spark tasks and how this relates to the number of partitions. But somehow I cannot understand the following case. </p>
  
  <p>I have a Hive table (an HDFS folder) which contains 160 Parquet-compressed files.
  The files are mostly well balanced: smallest is 7.5MB, largest 49.2MB. 
  In the HDFS browser I see each file is within 1 (non-full) HDFS block (128MB).</p>
  
  <p>The cluster has the following properties: 10 machines, 1 master and 9 workers. Each machine has 6 Cores (12 virtual cores). I am using Yarn. Moreover:</p>
  
  <pre><code>spark.executor.cores = 6
  </code></pre>
  
  <p>Now, I create the following dataframe:</p>
  
  <pre><code>val myDF = spark.sql("SELECT * FROM myHiveTable WHERE myCol='someValue')
  </code></pre>
  
  <p>Even before the job is triggered, it is possible to know in advance that:</p>
  
  <pre><code>myDF.rdd.partitions.size
  </code></pre>
  
  <p>returns 60.</p>
  
  <p>To trigger the job one needs an action, so I write "myDF" to HDFS. The job indeed results in 42 Executors and <strong>60 tasks</strong>.</p>
  
  <p>My questions: </p>
  
  <ol>
  <li><p>If I started with 160 partitions, how come I ended up having 60?</p></li>
  <li><p>If I have 60 tasks and 10 machines, then I would optimally need only 10 executors (somewhere I read that each executor can run as many tasks in parallel as the number of cores, which in my case is 6). I know that this would happen only if the dataset is perfectly balanced among the datanodes, but 42 Executors seems to me far away from 10. Or is my reasoning wrong?</p></li>
  <li><p>How can Spark know in advance, even before running the query, that this will result in 60 partitions.</p></li>
  </ol>
  
  <p>Thank you!</p>