title: Partitioning by column in Apache Spark to S3
body: <p>have use-case where we want to read files from S3 which has JSON. Then, based on a particular JSON node value we want to group the data and write it to S3. </p>
  
  <p>I am able to read the data but not able to find good example on how partition the data based on JSON key and then upload to S3. Can anyone provide any example or point me to a tutorial which can help me with this use-case?</p>
  
  <p>I have got the schema of my data after creating the dataframe:</p>
  
  <pre><code>root
   |-- customer: struct (nullable = true)
   |    |-- customerId: string (nullable = true)
   |-- experiment: string (nullable = true)
   |-- expiryTime: long (nullable = true)
   |-- partitionKey: string (nullable = true)
   |-- programId: string (nullable = true)
   |-- score: double (nullable = true)
   |-- startTime: long (nullable = true)
   |-- targetSets: array (nullable = true)
   |    |-- element: struct (containsNull = true)
   |    |    |-- featured: array (nullable = true)
   |    |    |    |-- element: struct (containsNull = true)
   |    |    |    |    |-- data: struct (nullable = true)
   |    |    |    |    |    |-- asinId: string (nullable = true)
   |    |    |    |    |-- pk: string (nullable = true)
   |    |    |    |    |-- type: string (nullable = true)
   |    |    |-- reason: array (nullable = true)
   |    |    |    |-- element: string (containsNull = true)
   |    |    |-- recommended: array (nullable = true)
   |    |    |    |-- element: string (containsNull = true)
  </code></pre>
  
  <p>I want to partition the data based on the random hash on the customerId column. But when i do this:</p>
  
  <pre><code>df.write.partitionBy("customerId").save("s3/bucket/location/to/save");
  </code></pre>
  
  <p>It give error:</p>
  
  <pre><code>org.apache.spark.sql.AnalysisException: Partition column customerId not found in schema StructType(StructField(customer,StructType(StructField(customerId,StringType,true)),true), StructField(experiment,StringType,true), StructField(expiryTime,LongType,true), StructField(partitionKey,StringType,true), StructField(programId,StringType,true), StructField(score,DoubleType,true), StructField(startTime,LongType,true), StructField(targetSets,ArrayType(StructType(StructField(featured,ArrayType(StructType(StructField(data,StructType(StructField(asinId,StringType,true)),true), StructField(pk,StringType,true), StructField(type,StringType,true)),true),true), StructField(reason,ArrayType(StringType,true),true), StructField(recommended,ArrayType(StringType,true),true)),true),true));
  </code></pre>
  
  <p>Please let me know i can access customerId column.</p>