title: Upload data to Redshift with PySpark
body: <p>I have a script written on pyspark. What I try to do is read *.csv file from S3 bucket in AWS using pyspark.</p>
  <p>I create a DataFrame with all data, select all the columns I need and cast them types my Redshift table expects:</p>
  <pre><code>    mapping = [('id', StringType), ('session', StringType), ('ip', StringType)]
  
      df = spark.read.\
          format(&quot;csv&quot;).\
          option(&quot;header&quot;, True).\
          load(f&quot;...&quot;)
      
      rows_to_map = [field[0] for field in columns_mapping]
      # We need to select only specific columns
      mapped_df = df.select(*rows_to_map)
      # Now need to cast types
      for mapping in columns_mapping:
          mapped_df = mapped_df.withColumn(mapping[0], mapped_df[mapping[0]].cast(mapping[1]()))
      
      mapped_df.printSchema()
      
      mapped_df.write.format(&quot;com.databricks.spark.redshift&quot;).\
          option(&quot;url&quot;, &quot;...&quot;).\
          option(&quot;dbtable&quot;, &quot;...&quot;).\
          option(&quot;tempdir&quot;, &quot;...&quot;).\
          option(&quot;user&quot;, &quot;...&quot;).\
          option(&quot;password&quot;, &quot;...&quot;).\
          option(&quot;aws_iam_role&quot;, &quot;...&quot;).\
          mode(&quot;append&quot;).\
          save()
  </code></pre>
  <p>And I receive an error during inserting data into redshift: Check 'stl_load_errors' system table for details.</p>
  <p>There I see that it tries to read columns from csv randomly(almost).</p>
  <p>SCHEMA of my dataframe:</p>
  <pre><code>|-- id: string (nullable = true) 
  |-- session: string (nullable = true) 
  |-- ip: string (nullable = true)
  ...
  </code></pre>
  <p>As you can see by first rows it goes like that id -&gt; session -&gt; ip ...
  But my Redshift table shows schema with the same fields but in different order. First 3 rows:</p>
  <pre><code>|-- id: string (nullable = true) 
  |-- created_at: long (nullable = true) 
  |-- session: string (nullable = true)
  </code></pre>
  <p>As a result on the second column he cries that I'm trying to write STRING into LONG column. Instead of created_at he read from the file session.</p>
  <p>Question: does order of columns in my DataFrame(tmp_file) is critical?
  Any solution for that? It will take too much time process each file.</p>
  <p>Thanks for any help.</p>