title: save Spark dataframe to Hive: table not readable because "parquet not a SequenceFile"
body: <p>I'd like to save data in a Spark (v 1.3.0) dataframe to a Hive table using PySpark.</p>
  
  <p>The <a href="https://spark.apache.org/docs/1.3.0/sql-programming-guide.html" rel="noreferrer" title="documentation">documentation</a> states:</p>
  
  <blockquote>
    <p>"spark.sql.hive.convertMetastoreParquet: When set to false, Spark SQL will use the Hive SerDe for parquet tables instead of the built in support."</p>
  </blockquote>
  
  <p>Looking at the <a href="http://ampcamp.berkeley.edu/5/exercises/data-exploration-using-spark-sql.html" rel="noreferrer">Spark tutorial</a>, is seems that this property can be set:</p>
  
  <pre class="lang-py prettyprint-override"><code>from pyspark.sql import HiveContext
  
  sqlContext = HiveContext(sc)
  sqlContext.sql("SET spark.sql.hive.convertMetastoreParquet=false")
  
  # code to create dataframe
  
  my_dataframe.saveAsTable("my_dataframe")
  </code></pre>
  
  <p>However, when I try to query the saved table in Hive it returns:</p>
  
  <pre><code>hive&gt; select * from my_dataframe;
  OK
  Failed with exception java.io.IOException:java.io.IOException: 
  hdfs://hadoop01.woolford.io:8020/user/hive/warehouse/my_dataframe/part-r-00001.parquet
  not a SequenceFile
  </code></pre>
  
  <p>How do I save the table so that it's immediately readable in Hive?</p>