title: pyspark - iterating through the file and substituting the value in dataframe
body: <p>I need to read through the file and substitute the value while the s3 path.I am able to loop through the file but I am not able to substitute the value.</p>
  
  <pre><code>File ending with \n
  /MTD_avg_cust_bal1
  /MTDSumOfCustomerInitiatedTrxns1
  /MTDCountOfCustomerInitiatedTrxns1
  </code></pre>
  
  <p><strong>Code:</strong></p>
  
  <pre><code>metrics = open("Output.txt","r")
  line = metrics.readline()
  
  while line:
      print line
      line = metrics.readline()
      s3path = ("SELECT * FROM parquet.`s3n://bucket{}/loaddate=20170406/part-r-00000-d60b633d-ff49-4515-8cff-ace9faf1b267.csv`") .format(line).strip('\n')
  
      print s3path
      df1 = sqlContext.sql(s3path)
  </code></pre>
  
  <p><strong>Error:</strong></p>
  
  <pre><code>pyspark.sql.utils.AnalysisException: u'Path does not exist: s3n://omniscience1/MTDSumOfCustomerInitiatedTrxns1\n/loaddate=20170406/part-r-00000-d60b633d-ff49-4515-8cff-ace9faf1b267.csv;; line 1 pos 14'
  </code></pre>
  
  <p>The problem is it's including <code>\n</code> when the value is substituted and also I need to have separate data frame for each line.</p>