title: Flume java.lang.IllegalStateException: File has changed size since being read
body: <p>I have a java application that gather data from different sources and write the output into files under a specific directory. </p>
  
  <p>And I have a flume agent configured to use spooldir source to read from that directory and write the output to Solr using <code>MorphlineSolrSink</code>. </p>
  
  <p>The flume agent throws the following exception </p>
  
  <pre><code>java.lang.IllegalStateException: File has changed size since being read
  </code></pre>
  
  <p>Here is the configuration of the flume agent</p>
  
  <pre><code>agent02.sources = s1
  agent02.sinks = solrSink
  agent02.channels = ch1
  
  agent02.channels.ch1.type = file
  agent02.channels.ch1.checkpointDir=/home/flume/prod_solr_chkpoint/file-channel/checkpoint
  agent02.channels.ch1.dataDirs= /home/flume/prod_solr_chkpoint/file-channel/data
  
  agent02.sources.s1.type = spooldir
  agent02.sources.s1.channels = ch1
  
  agent02.sources.s1.spoolDir = /DataCollection/json_output/solr/
  agent02.sources.s1.deserializer.maxLineLength = 100000
  
  agent02.sinks.solrSink.type = org.apache.flume.sink.solr.morphline.MorphlineSolrSink
  agent02.sinks.solrSink.channel = ch1
  agent02.sinks.solrSink.batchSize = 10000
  agent02.sinks.solrSink.batchDurationMillis = 10000
  agent02.sinks.solrSink.morphlineFile = morphlines.conf 
  agent02.sinks.solrSink.morphlineId = morphline
  </code></pre>
  
  <p>What I understand from the exception is that the flume agent start working on a file while the java application did not finish writing it. </p>
  
  <p>How can I fix this problem ? </p>
  
  <h2>Edit</h2>
  
  <p>I have no idea this information is valuable or not. 
  These configurations were working before without any problem. We faced a hard desk failure in the machine we run flume from. After recovering from that failure flume throws this exception. </p>