title: Logstash doesn't process all input files
body: <p>I'm using logstash to process several Zanox csv exports and export them to elasticsearch.
  However, for some reason logstash often processes only some of the input files.</p>
  
  <p>The input files definitely exist in the given directory. To avoid the Logstash Inode buch I've set sincedb_path to /dev/null, stop logstash every day before the new files are downloaded and start it after the downloads have been completed.</p>
  
  <p>Logstash and Elasticsearch are currently on the same server</p>
  
  <p>The first file (1) is the only one that always is being imported. It is a rather big feed with about half a gigabyte.
  Also the Zanox csv has a small glitch: the first line starts with a point, invalidating the csv format for this line.</p>
  
  <pre><code>input { 
  file {
       path =&gt; ["/var/app/1/*.csv"]
       sincedb_path =&gt; "/dev/null"
       start_position =&gt; beginning
       type =&gt; "1"
    }  
      file {
       path =&gt; ["/var/app/2/*.csv"]
       sincedb_path =&gt; "/dev/null"
       start_position =&gt; beginning
       type =&gt; "2"
    }
      file {
       path =&gt; ["/var/app/3/*.csv"]
       sincedb_path =&gt; "/dev/null"
       start_position =&gt; beginning
       type =&gt; "3"
    }
      file {
       path =&gt; ["/var/app/4/*.csv"]
       sincedb_path =&gt; "/dev/null"
       start_position =&gt; beginning
       type =&gt; "4"
    }
      file {
       path =&gt; ["/var/app/5/*.csv"]
       sincedb_path =&gt; "/dev/null"
       start_position =&gt; beginning
       type =&gt; "5"
    }
      file {
       path =&gt; ["/var/app/6/*.csv"]
       sincedb_path =&gt; "/dev/null"
       start_position =&gt; beginning
       type =&gt; "6"
    }
  }
  filter {
    if [type] == "1" {
        csv { 
          columns =&gt; [ "title", "price", "image", "deeplink", "color" ] 
      separator =&gt; ";" 
        }
  
    } else {
        csv { 
          columns =&gt; [ "title", "price", "image", "deeplink" ] 
      separator =&gt; ";" 
        }
    }
    mutate {
      convert =&gt; ["price", "float"]
      add_field =&gt; {"source" =&gt; "%{type}"}
    }
    if ![title] { 
      drop { } 
    }
  }
  output {
    elasticsearch{
      index =&gt; products
      index_type =&gt; products
      host =&gt; localhost
      document_id =&gt; "%{deeplink}"
      flush_size =&gt; 5000
    }
  }
  </code></pre>
  
  <p>What could be the reason for logstash not processing all the files?</p>
  
  <p><strong>EDIT:</strong>
  Removed CSV processing errors by doing some pre-processing. Now I got the following error in the logstash log:</p>
  
  <pre><code>log4j, [2014-09-11T03:41:50.075]  WARN: org.elasticsearch.monitor.jvm: 
    [logstash-17675126.onlinehome-server.info-13551-4016] [gc][young][1345505][23190] duration [2.4s], 
    collections [1]/[2.8s], 
    total [2.4s]/[10.3m], 
    memory [298.8mb]-&gt;[122.3mb]/[483.3mb], 
    all_pools {
      [young] [1.9mb]-&gt;[1.2mb]/[133.3mb]
    }
    {
      [survivor] [16.6mb]-&gt;[0b]/[16.6mb]}{[old] [280.2mb]-&gt;[121.1mb]/[333.3mb]
    }
  </code></pre>