title: Open parquet from GCS using local Pyspark
body: <p>i have a folder on Google Cloud Storage with several parquet files. I installed in my VM pyspark and now i want to read the parquet files. Here's my code:</p>
  
  <pre><code>from pyspark.sql import SparkSession
  
  spark = SparkSession\
          .builder\
          .config("spark.driver.maxResultSize", "40g") \
          .config('spark.sql.shuffle.partitions', '2001') \
          .config("spark.jars", "~/spark/spark-2.4.4-bin-hadoop2.7/jars/gcs-connector-hadoop2-latest.jar")\
          .getOrCreate()
  sc = spark.sparkContext
  # using SQLContext to read parquet file
  from pyspark.sql import SQLContext
  sqlContext = SQLContext(sc)
  # to read parquet file
  filename = "gs://path/to/parquet"
  df = sqlContext.read.parquet(filename)
  print(df.head())
  </code></pre>
  
  <p>When i run it, it gives me the following error:</p>
  
  <pre><code>WARN FileStreamSink: Error while looking for metadata directory.
  </code></pre>
  
  <p>To install pyspark i followed this tutorial: <a href="https://towardsdatascience.com/how-to-get-started-with-pyspark-1adc142456ec" rel="nofollow noreferrer">https://towardsdatascience.com/how-to-get-started-with-pyspark-1adc142456ec</a></p>