title: Efficient way of appending Spark DataFrames in a loop using pyspark
body: <p>
  I have '|' delimited huge text files, I want to merge all the text files and create one huge spark dataframe, it will be later used for ETL process, using pyspark.</p>
  
  <p><strong>Inefficient way</strong></p>
  
  <p>1) Create an empty spark dataframe, df</p>
  
  <p>2) In a loop,read the text file as to spark dataframe df1 and appending it to empty spark dataframe df</p>
  
  <pre><code>df = spark.createDataFrame([],schema)
  for x in os.listdir(textfiles_dir):
      filepath = '{}/{}'.format(textfiles_dir,x)
      df1 = spark.read.format("csv") \
                      .option("header", "true") \
                      .option("delimiter", "|") \
                      .option("inferSchema","true") \
                      .load(filepath)
      df = df.union(df1)
  </code></pre>
  
  <p>This is not an efficient spark way.</p>
  
  <p>Can anyone <strong>suggest an efficient way</strong> of doing this?
  That would be great if explained with sample code.</p>
  
  <p>Thanks :)
  </p>