title: Does Spark know the partitioning key of a DataFrame?
body: <p>I want to know if Spark knows the partitioning key of the parquet file and uses this information to avoid shuffles.</p>
  
  <p><strong>Context:</strong></p>
  
  <p>Running Spark 2.0.1 running local SparkSession. I have a csv dataset that I am saving as parquet file on my disk like so:</p>
  
  <pre><code>val df0 = spark
    .read
    .format("csv")
    .option("header", true)
    .option("delimiter", ";")
    .option("inferSchema", false)
    .load("SomeFile.csv"))
  
  
  val df = df0.repartition(partitionExprs = col("numerocarte"), numPartitions = 42)
  
  df.write
    .mode(SaveMode.Overwrite)
    .format("parquet")
    .option("inferSchema", false)
    .save("SomeFile.parquet")
  </code></pre>
  
  <p>I am creating 42 partitions by column <code>numerocarte</code>. This should group multiple <code>numerocarte</code> to same partition. I don't want to do partitionBy("numerocarte") at the <code>write</code> time because I don't want one partition per card. It would be millions of them. </p>
  
  <p>After that in another script I read this <code>SomeFile.parquet</code> parquet file and do some operations on it. In particular I am running a <code>window function</code> on it where the partitioning is done on the same column that the parquet file was repartitioned by.</p>
  
  <pre><code>import org.apache.spark.sql.expressions.Window
  import org.apache.spark.sql.functions._
  
  val df2 = spark.read
    .format("parquet")
    .option("header", true)
    .option("inferSchema", false)
    .load("SomeFile.parquet")
  
  val w = Window.partitionBy(col("numerocarte"))
  .orderBy(col("SomeColumn"))
  
  df2.withColumn("NewColumnName",
        sum(col("dollars").over(w))
  </code></pre>
  
  <p>After <code>read</code> I can see that the <code>repartition</code> worked as expected and DataFrame <code>df2</code> has 42 partitions and in each of them are different cards.</p>
  
  <p><strong>Questions:</strong></p>
  
  <ol>
  <li>Does Spark know that the dataframe <code>df2</code> is partitioned by column <code>numerocarte</code>?</li>
  <li>If it knows, then there will be no shuffle in the window function. True?</li>
  <li>If it does not know, It will do a shuffle in the window function. True?</li>
  <li>If it does not know, how do I tell Spark the data is already partitioned by the right column?</li>
  <li>How can I check a partitioning key of <code>DataFrame</code>? Is there a command for this? I know how to check number of partitions but how to see partitioning key? </li>
  <li>When I print number of partitions in a file after each step, I have 42 partitions after <code>read</code> and 200 partitions after <code>withColumn</code> which suggests that Spark repartitioned my <code>DataFrame</code>.</li>
  <li>If I have two different tables repartitioned with the same column, would the join use that information?</li>
  </ol>