title: Reading Avro container files in Spark
body: <p>I am working on a scenario where I need to read Avro container files from HDFS and do analysis using Spark.</p>
  
  <blockquote>
    <p><strong>Input Files Directory:</strong> <em>hdfs:///user/learner/20151223/</em>.lzo*</p>
  </blockquote>
  
  <p>Note : The Input Avro Files are lzo compressed. </p>
  
  <pre><code>val df = sqlContext.read.avro("/user/learner/20151223/*.lzo");
  </code></pre>
  
  <p>When I run the above command.It throws an error :</p>
  
  <pre><code>java.io.FileNotFoundException: No avro files present at file:/user/learner/20151223/*.lzo
  at com.databricks.spark.avro.AvroRelation$$anonfun$11.apply(AvroRelation.scala:225)
  at com.databricks.spark.avro.AvroRelation$$anonfun$11.apply(AvroRelation.scala:225)
  at scala.Option.getOrElse(Option.scala:120)
  at com.databricks.spark.avro.AvroRelation.newReader(AvroRelation.scala:225)
  </code></pre>
  
  <p>This make sense,because the method read.avro() is expecting .avro  extension files as input.So I extract and rename the input .lzo file to .avro.I am able to read the data in avro file properly. </p>
  
  <p>Is there any way to read lzo compressed Avro files in spark ? </p>
  
  <p><strong>Solution worked, But !</strong> </p>
  
  <p>I have found a way to solve this issue. I created a shell wrapper in which I have decompressed the .lzo into .avro file format using following way:</p>
  
  <pre><code>hadoop fs -text &lt;file_path&gt;*.lzo | hadoop fs - put - &lt;file_path&gt;.avro
  </code></pre>
  
  <p>I am successfull in decompressing lzo files but the problem is I am having atleast 5000 files in compressed format.Uncompressing and Converting one by one is taking nearly 1+ hours to run this Job. </p>
  
  <p>Is there any way to do this Decompression in bulk way ?</p>
  
  <p>Thanks again !</p>