title: How to decompress LZ4 JSON in Spark 2
body: <p>I have downloaded a <code>xxxx.json.lz4</code> file from <code>https://censys.io/</code> however when I try to read the file using the following line I get no data out/count of 0. </p>
  
  <pre><code>metadata_lz4 = spark.read.json("s3n://file.json.lz4")
  </code></pre>
  
  <p>it returns no results although decompressing manually works fine and can be imported into Spark.</p>
  
  <p>I have also tried </p>
  
  <pre><code>val metadata_lz4_2 = spark.sparkContext.newAPIHadoopFile("s3n://file.json.lz4", classOf[TextInputFormat], classOf[LongWritable], classOf[Text])
  </code></pre>
  
  <p>Which also returns no results. </p>
  
  <p>I have multiple of these files which are <code>100GBs</code> each so really keen on not having to decompress each one manually.</p>
  
  <p>Any ideas?</p>