title: How to set the number of partitions/nodes when importing data into Spark
body: <p><strong>Problem:</strong> I want to import data into Spark EMR from S3 using:</p>
  
  <pre><code>data = sqlContext.read.json("s3n://.....")
  </code></pre>
  
  <p>Is there a way I can set the number of nodes that Spark uses to <em>load</em> and <em>process</em> the data? This is an example of how I process the data:</p>
  
  <pre><code>data.registerTempTable("table")
  SqlData = sqlContext.sql("SELECT * FROM table")
  </code></pre>
  
  <p><strong>Context</strong>: The data is not too big, takes a long time to load into Spark and also to query from. I think Spark partitions the data into too many nodes. I want to be able to set that manually. I know when dealing with RDDs and <code>sc.parallelize</code> I can pass the number of partitions as an input. Also, I have seen <code>repartition()</code>, but I am not sure if it can solve my problem. The variable <code>data</code> is a <code>DataFrame</code> in my example.</p>
  
  <p>Let me define partition more precisely. Definition one: commonly referred to as "partition key" , where a column is selected and indexed to speed up query (that is not what i want). Definition two: (this is where my concern is) suppose you have a data set, Spark decides it is going to distribute it across many nodes so it can run operations on the data in parallel. If the data size is too small, this may further slow down the process. How can i set that value </p>