title: Flume to stream gz files
body: <p>I have a folder contains a lot of gzip files. Each gzip file contains xml file. I had used flume to stream the files into HDFS. Below is my configuration file:</p>
  
  <pre><code>agent1.sources = src
  agent1.channels = ch
  agent1.sinks = sink
  
  agent1.sources.src.type = spooldir
  agent1.sources.src.spoolDir = /home/tester/datafiles
  agent1.sources.src.channels = ch
  agent1.sources.src.deserializer = org.apache.flume.sink.solr.morphline.BlobDeserializer$Builder
  
  agent1.channels.ch.type = memory
  agent1.channels.ch.capacity = 1000
  agent1.channels.ch.transactionCapacity = 1000
  
  agent1.sinks.sink.type = hdfs
  agent1.sinks.sink.channel = ch
  agent1.sinks.sink.hdfs.path = /user/tester/datafiles
  agent1.sinks.sink.hdfs.fileType = CompressedStream
  agent1.sinks.sink.hdfs.codeC = gzip
  agent1.sinks.sink.hdfs.fileSuffix = .gz
  agent1.sinks.sink.hdfs.rollInterval = 0
  agent1.sinks.sink.hdfs.rollSize = 122000000
  agent1.sinks.sink.hdfs.rollCount = 0
  agent1.sinks.sink.hdfs.idleTimeout = 1
  agent1.sinks.sink.hdfs.batchSize = 1000
  </code></pre>
  
  <p>After I stream the files into HDFS, i use Spark to read it using the following code:</p>
  
  <pre><code>df = sparkSession.read.format('com.databricks.spark.xml').options(rowTag='Panel', compression='gzip').load('/user/tester/datafiles')
  </code></pre>
  
  <p>But I am having issue to read it. If i manually upload one gzip file into HDFS folder and re-run the above Spark code, it able to read it without any issue. I am not sure is it due to flume.</p>
  
  <p>I tried to download the file streamed by flume and unzip it, when i viewed the contents, it no longer showing the xml format, it is some unreadable character. Could anyone shed me some light on this? Thanks.</p>