title: Convert csv to parquet file using python
body: <p>I am trying to convert a .csv file to a .parquet file.<br>
  The csv file (<code>Temp.csv</code>) has the following format    </p>
  
  <pre><code>1,Jon,Doe,Denver
  </code></pre>
  
  <p>I am using the following python code to convert it into parquet</p>
  
  <pre><code>from pyspark import SparkContext
  from pyspark.sql import SQLContext
  from pyspark.sql.types import *
  import os
  
  if __name__ == "__main__":
      sc = SparkContext(appName="CSV2Parquet")
      sqlContext = SQLContext(sc)
  
      schema = StructType([
              StructField("col1", IntegerType(), True),
              StructField("col2", StringType(), True),
              StructField("col3", StringType(), True),
              StructField("col4", StringType(), True)])
      dirname = os.path.dirname(os.path.abspath(__file__))
      csvfilename = os.path.join(dirname,'Temp.csv')    
      rdd = sc.textFile(csvfilename).map(lambda line: line.split(","))
      df = sqlContext.createDataFrame(rdd, schema)
      parquetfilename = os.path.join(dirname,'output.parquet')    
      df.write.mode('overwrite').parquet(parquetfilename)
  </code></pre>
  
  <p>The result is only a folder named, <code>output.parquet</code> and not a parquet file that I'm looking for, followed by the following error on the console.  </p>
  
  <p><a href="https://i.stack.imgur.com/WWDID.png" rel="noreferrer"><img src="https://i.stack.imgur.com/WWDID.png" alt="CSV to Parquet Error"></a></p>
  
  <p>I have also tried running the following code to face a similar issue.  </p>
  
  <pre><code>from pyspark.sql import SparkSession
  import os
  
  spark = SparkSession \
      .builder \
      .appName("Protob Conversion to Parquet") \
      .config("spark.some.config.option", "some-value") \
      .getOrCreate()
  
  # read csv
  dirname = os.path.dirname(os.path.abspath(__file__))
  csvfilename = os.path.join(dirname,'Temp.csv')    
  df = spark.read.csv(csvfilename)
  
  # Displays the content of the DataFrame to stdout
  df.show()
  parquetfilename = os.path.join(dirname,'output.parquet')    
  df.write.mode('overwrite').parquet(parquetfilename)
  </code></pre>
  
  <p>How to best do it? Using windows, python 2.7.</p>