title: Import CSV into Elasticsearch with Logstash
body: <p>I use the ELK stack to parse CSV files and send them to ElasticSearch after parsing them with logstash.</p>
  
  <p>Unfortunately, I have a problem:</p>
  
  <p>When I send my files to the listening directory of the "input" of my logstash pipeline, the records are doubled, see triplets, without my asking anything ...</p>
  
  <p>Indeed :</p>
  
  <p>This is what my pipeline looks like:</p>
  
  <pre><code>input {
    file {
      path =&gt; "/home/XXX/report/*.csv"
      start_position =&gt; "beginning"
      sincedb_path =&gt; "/dev/null"
    }
  }
  filter {
    csv {
        separator =&gt; ";"
       columns =&gt; ["Name", "Status", "Category", "Type", "EndPoint", "Group", "Policy", "Scanned At", "Reported At", "Affected Application"]
    }
  }
  output {
     elasticsearch {
       hosts =&gt; "http://localhost:9200"
       index =&gt; "malwarebytes-report"
    }
  stdout {}
  }
  </code></pre>
  
  <p>When I send my first file containing 28 records in "/home/XXX/report/", this is what ElasticSearch says:</p>
  
  <pre><code>[root @ lrtstfpe1 confd]#curl -XGET 'localhost:9200/_cat/indices?v&amp;pretty'
  health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
  yellow open malwarebytes-report PO4g6rKRTb6yuMDb7i-6sg 5 1 28 0 25.3kb 25.3kb
  </code></pre>
  
  <p>So far so good, but when I send my second file of 150 records ...:</p>
  
  <pre><code>[root @ lrtstfpe1 confd]#curl -XGET 'localhost:9200/_cat/indices?v&amp;pretty'
  health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
  yellow open malwarebytes-report PO4g6rKRTb6yuMDb7i-6sg 5 1 328 0 263.3kb 263.3kb
  </code></pre>
  
  <p>The 150 recordings have been doubled and added to the first 28 ...</p>
  
  <p>What's going on ??</p>
  
  <p>Several days that I am stuck on the problem, I really need you ..</p>
  
  <p>UPDATE : </p>
  
  <pre><code>You need to look in /etc/logstash/conf.d and see if there are any other config files there
  </code></pre>
  
  <p>The problem is that I only have one pipeline in this folder ... So:</p>
  
  <p>I just completely uninstalled the ELK stack (<code>rpm -e elasticsearch kibana logstash filebeat</code>) as well as any ELK traces (<code>rm -rf /var/lib/ELK/ var/log/ELK/ etc/default/ELK /usr/share/ELK</code> ...) So, nothing anywhere.</p>
  
  <p>I just reinstall everything:</p>
  
  <pre><code>rpm -ivh elasticsearch-6.2.3.rpm
  rpm -ivh kibana-6.2.3-x86_64.rpm
  rpm -ivh logstash-6.2.3.rpm
  </code></pre>
  
  <p>And start the services: <code>service ELK restart</code></p>
  
  <p>Then, in terms of configurations:
  <code>/etc/elasticsearch.yml</code> is completely by default.
  <code>/etc/kibana.ym</code>l is completely by default.
  <code>/etc/logstash.yml</code> is completely by default.</p>
  
  <p>Then, I put my one and ONLY pipeline named "pip.conf" in <code>/etc/logstash/conf.d/</code>
  Its configuration:</p>
  
  <pre><code>input {
     file {
       path =&gt; "/home/report/*.csv"
       start_position =&gt; "beginning"
       sincedb_path =&gt; "/dev/null"
    }
  }
  filter {
    csv {
       separator =&gt; ";"
       columns =&gt; ["Name","Status","Category","Type","EndPoint","Group","Policy","Scanned At","Reported At","Affected Application"]
    }
  }
  output {
     elasticsearch {
       hosts =&gt; "http://localhost:9200"
       index =&gt; "malwarebytes-report"
    }
  stdout{}
  }
  </code></pre>
  
  <p>And finally, I launch my pipeline :
  I go into <code>/usr/share/logstash</code> and I execute :</p>
  
  <pre><code>bin/logstash -f /etc/logstash/conf.d/pip.conf
  </code></pre>
  
  <p>After few secondes, my pipeline is listening, and now, I put my <code>file1.csv</code> and my <code>file2.csv</code> into <code>/home/report/</code>.</p>
  
  <p><code>file1.csv</code> contains 28 records and <code>file2.csv</code> contains 150 records.</p>
  
  <p>But now, when I check my index : <code>curl -XGET 'localhost:9200/_cat/indices?v&amp;pretty'</code>
  My index "malwarebytes-report" contains 357 records ... (150x2 + 28x2 ...)</p>
  
  <p>I don't understand NOTHING ....</p>