title: Flume Twitter Stream rolling small files in HDFS
body: <p>I think I have tried every combination of altering my config file. I also saw somewhere that it might be due to my replication factor being 3 so I changed it to 1. I am using cloudera manager on AWS. Below is my config file, any ideas?  </p>
  
  <p>In HDFS, the file sizes are all under 20kb, trying to get at least 40-50mb. What is funny is that the same config file is writing ~60mb files on my virtual machine that I was practicing with (pre-installed hadoop + tools). See  below for config file, any ideas? </p>
  
  <pre><code># The configuration file needs to define the sources, 
  # the channels and the sinks.
  # Sources, channels and sinks are defined per agent, 
  # in this case called 'TwitterAgent'
  
  TwitterAgent.sources = Twitter
  TwitterAgent.channels = MemChannel
  TwitterAgent.sinks = HDFS
  
  TwitterAgent.sources.Twitter.type = com.cloudera.flume.source.TwitterSource
  TwitterAgent.sources.Twitter.channels = MemChannel
  TwitterAgent.sources.Twitter.consumerKey = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
  TwitterAgent.sources.Twitter.consumerSecret = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
  TwitterAgent.sources.Twitter.accessToken = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 
  TwitterAgent.sources.Twitter.accessTokenSecret = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
  TwitterAgent.sources.Twitter.keywords = apple, grapes, fruits, strawberry, mango, pear
  TwitterAgent.sinks.HDFS.channel = MemChannel
  TwitterAgent.sinks.HDFS.type = hdfs
  TwitterAgent.sinks.HDFS.hdfs.path = hdfs://123.456.789.us-west-2.compute.amazonaws.com:8020/user/flume/tweets
  TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream
  TwitterAgent.sinks.HDFS.hdfs.writeFormat = Text
  TwitterAgent.sinks.HDFS.hdfs.rollInterval = 0
  TwitterAgent.sinks.HDFS.hdfs.batchSize = 100000
  TwitterAgent.sinks.HDFS.hdfs.rollSize = 0
  TwitterAgent.sinks.HDFS.hdfs.rollCount = 0
  
  TwitterAgent.channels.MemChannel.type = memory
  TwitterAgent.channels.MemChannel.capacity = 10000
  TwitterAgent.channels.MemChannel.transactionCapacity = 1000
  </code></pre>