title: Spark: writing BZip2 compressed parquet files
body: <p>I am wanting to write parquet files from a DataFrame in spark-sql with BZip2 codec compression so that they are splittable. With the following code, I'm able to use codecs such as snappy and gzip:</p>
  
  <pre><code>sqlContext.setConf("spark.sql.parquet.compression.codec", "snappy")
  sqlContext.read.parquet(stagingDir)
          .dropDuplicates()
          .write
          .mode(SaveMode.Append)
          .parquet(outputDir)
  </code></pre>
  
  <p>However, when I try BZip2 it seems it isn't available as I get this exception, even though I was able to write BZip2 compressed text files from an RDD</p>
  
  <pre><code>java.lang.IllegalArgumentException: The value of spark.sql.parquet.compression.codec should be one of uncompressed, snappy, gzip, lzo, but was bzip2
  </code></pre>
  
  <p>Is there a way to write BZip2 compressed parquet files from Spark SQL?</p>