title: Apache Camel ftp consumer loads the same files again and again
body: <p>I have following spring configuration</p>
  
  <pre><code>&lt;?xml version="1.0" encoding="UTF-8"?&gt;
  &lt;beans xmlns="http://www.springframework.org/schema/beans"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="
         http://www.springframework.org/schema/beans 
         http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
         http://camel.apache.org/schema/spring 
         http://camel.apache.org/schema/spring/camel-spring.xsd"&gt;
  
      &lt;bean id="downloadLogger" class="com.thomsonreuters.oa.sdi.camel.DownloadLogger" /&gt;
  
      &lt;bean id="fileFilter" class="com.thomsonreuters.oa.sdi.camel.IgnoreReadyFilesFilter" /&gt;
  
      &lt;camelContext xmlns="http://camel.apache.org/schema/spring"&gt;
          &lt;route&gt;
              &lt;from uri="ftp://url_to_ftp?password=*******&amp;amp;noop=true&amp;amp;stepwise=false&amp;amp;binary=true&amp;amp;consumer.delay=10s&amp;amp;recursive=true&amp;amp;filter=#fileFilter" /&gt;
              &lt;process ref="downloadLogger" /&gt;
              &lt;to uri="file:data/outbox" /&gt;
          &lt;/route&gt;
      &lt;/camelContext&gt;
  
  &lt;/beans&gt;
  </code></pre>
  
  <p>At the ftp side I have 3 folders with files which I want to download. I want to achieve following scenario:</p>
  
  <ol>
  <li>On ftp is fixed amount of files (for isntance 5) at the first data pull consumer loads these files to the destination folder</li>
  <li>At the second attempt to load files, ftp state still the same (5 files) and camel ftp consumer just does nothing (except check for new files)</li>
  <li>To ftp arrives new 2 files, and at this data pull consumer downloads only these new two files</li>
  </ol>
  
  <p>At the moment my current solutions downloads all files each time when I run dataload process, how I can manage information about downloaded files to prevent downloads of duplicates (I mean already copied files from ftp), I can write my own filter which will filter out already downloaded files but I belive there should be built in feature which will give me controle of this (maybe idempotentRepository, actually I am not sure)...</p>