title: How to add only new SQL rows in Elasticsearch - Logstash - Kibana?
body: <p>I'm trying to add some data from an SQL query to my Elasticsearch, and it's currently running as a Logstash jdbc input like this:</p>
  
  <pre><code> jdbc {
          type =&gt; "database_log"
          jdbc_connection_string =&gt; "jdbc:sqlserver://(redacted)"
          jdbc_user =&gt; (redacted)
          jdbc_password =&gt; (redacted)
          jdbc_driver_library =&gt; "D:\ELK_56\sqljdbc4.jar"
          jdbc_driver_class =&gt; "com.microsoft.sqlserver.jdbc.SQLServerDriver"
          schedule =&gt; "0 * * * *"
          statement =&gt; "select user_id,First_Name from user_login"
  </code></pre>
  
  <p>Now the problem is that like each hour when the input runs, it just adds the entire result of the query, some 124k rows. My intent is to just add the new rows (of which there are about 3-4 per hour) that have appeared since the last time i queried, and avoid duplicates.</p>
  
  <p>I can't just solve it by adding a WHERE clause to the query since this particular table has no date column or any sort of such identifier.</p>
  
  <p>Is there any way to have logstash or elasticsearch itself substract the previous result from the latest one and just add the new rows?</p>
  
  <p>Thank you for your reply.</p>