title: What causes flume with GCS sink to throw a OutOfMemoryException
body: <p>I am using flume to write to Google Cloud Storage. Flume listens on <code>HTTP:9000</code>. It took me some time to make it work (add gcs libaries, use a credentials file...) but now it seems to communicate over the network.</p>
  
  <p>I am sending very small HTTP request for my tests, and I have plenty of RAM available: </p>
  
  <pre><code>curl -X POST -d '[{ "headers" : { timestamp=1417444588182, env=dev, tenant=myTenant, type=myType }, "body" : "some body ONE"  }]' localhost:9000
  </code></pre>
  
  <p>I encounter this memory exception on first request (then of course, it stops working):</p>
  
  <pre><code>2014-11-28 16:59:47,748 (hdfs-hdfs_sink-call-runner-0) [INFO - com.google.cloud.hadoop.util.LogUtil.info(LogUtil.java:142)] GHFS version: 1.3.0-hadoop2
  2014-11-28 16:59:50,014 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:467)] process failed
  java.lang.OutOfMemoryError: Java heap space
          at java.io.BufferedOutputStream.&lt;init&gt;(BufferedOutputStream.java:76)
          at com.google.cloud.hadoop.fs.gcs.GoogleHadoopOutputStream.&lt;init&gt;(GoogleHadoopOutputStream.java:79)
          at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.create(GoogleHadoopFileSystemBase.java:820)
          at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:906)
  </code></pre>
  
  <p>(see <a href="https://gist.github.com/jpcaruana/3a960ff64ff0eb1c30e0" rel="nofollow noreferrer">complete stack trace as a gist</a> for full details)</p>
  
  <p>The strange part is that folders and files are created the way I want, but files are empty.</p>
  
  <pre><code>gs://my_bucket/dev/myTenant/myType/2014-12-01/14-36-28.1417445234193.json.tmp
  </code></pre>
  
  <p>Is it something wrong with the way I configured flume + GCS or is it a bug in <a href="https://storage.googleapis.com/hadoop-lib/gcs/gcs-connector-latest-hadoop2.jar" rel="nofollow noreferrer">GCS.jar</a> ?</p>
  
  <p>Where should I check to gather more data ?</p>
  
  <p>ps : I am running flume-ng inside docker.</p>
  
  <hr>
  
  <p>My <code>flume.conf</code> file:</p>
  
  <pre><code># Name the components on this agent
  a1.sources = http
  a1.sinks = hdfs_sink
  a1.channels = mem
  
  # Describe/configure the source
  a1.sources.http.type =  org.apache.flume.source.http.HTTPSource
  a1.sources.http.port = 9000
  
  # Describe the sink
  a1.sinks.hdfs_sink.type = hdfs
  a1.sinks.hdfs_sink.hdfs.path = gs://my_bucket/%{env}/%{tenant}/%{type}/%Y-%m-%d
  a1.sinks.hdfs_sink.hdfs.filePrefix = %H-%M-%S
  a1.sinks.hdfs_sink.hdfs.fileSuffix = .json
  a1.sinks.hdfs_sink.hdfs.round = true
  a1.sinks.hdfs_sink.hdfs.roundValue = 10
  a1.sinks.hdfs_sink.hdfs.roundUnit = minute
  
  # Use a channel which buffers events in memory
  a1.channels.mem.type = memory
  a1.channels.mem.capacity = 10000
  a1.channels.mem.transactionCapacity = 1000
  
  # Bind the source and sink to the channel
  a1.sources.http.channels = mem
  a1.sinks.hdfs_sink.channel = mem
  </code></pre>
  
  <hr>
  
  <p>related question in my flume/gcs journey: <a href="https://stackoverflow.com/questions/27174033/what-is-the-minimal-setup-needed-to-write-to-hdfs-gs-on-google-cloud-storage-wit">What is the minimal setup needed to write to HDFS/GS on Google Cloud Storage with flume?</a></p>