title: Writing to HDFS in Spark/Scala reading the zip files
body: <p>I am writing a spark/scala program to read in ZIP files, unzip them and write the contents to a set of new files. I can get this to work for writing to the local file system but wondered if there was a way to to write the output files to a distributed file system such as HDFS. Code is shown below`</p>
  
  <pre><code>import java.util.zip.ZipInputStream
  import org.apache.spark.input.PortableDataStream
  import java.io._
  
  var i =1
  sc.binaryFiles("file:///d/tmp/zips/").flatMap((file:(String, PortableDataStream)) =&gt; {   
     val zipStream = new ZipInputStream(file._2.open)            
     val entry = zipStream.getNextEntry                            
     val iter = scala.io.Source.fromInputStream(zipStream).getLines          
     val fname = f"/d/tmp/myfile$i.txt" 
  
     i = i + 1
  
     val xx = iter.mkString
     val writer = new PrintWriter(new File(fname))
     writer.write(xx)
     writer.close()
  
     iter                                                       
  }).collect()
  </code></pre>
  
  <p>`</p>