title: Camel route move files from local folder to sftp folder using multiple threads
body: <p>So I am moving the files from  a local folder to an sftp location. The files potentially could also be in order of 500,000.
  I have seda for writer process where multiple writers are writing the files, but moving the files have become bottleneck. Following is my camel route currently :</p>
  
  <pre><code>from(props.splitFolderUri)
              .routeId('move-files')
              .log(LoggingLevel.INFO, 'file : ${exchange.getIn().headers[\'CamelFileName\']} :: moving this split file to sftp location ')
              .to(props.remoteFolderUri)
              .log(LoggingLevel.INFO, 'file : ${exchange.getIn().headers[\'CamelFileName\']} :: moved this split file to mailbox')
  </code></pre>
  
  <p>I want to make this route multi threaded so that more than one thread keeps on moving the files.
  I tried using threads(10) like as follows </p>
  
  <pre><code>rom(props.splitFolderUri)
          .threads(10)
          .routeId('move-files')
          .log(LoggingLevel.INFO, 'file : ${exchange.getIn().headers[\'CamelFileName\']} :: moving this split file to sftp location ')
          .to(props.remoteFolderUri)
          .log(LoggingLevel.INFO, 'file : ${exchange.getIn().headers[\'CamelFileName\']} :: moved this split file to mailbox')
  </code></pre>
  
  <p>But, I have rename file operation fail. after some research i found that this is the scenario that occurs :
  Here is the scenario:</p>
  
  <pre><code>Thread#1 retrieves file1, moves to process folder
  Thread#2 retrieves same file: file1 at the same time. file1 is deleted
  Thread#2 cannot find file1 in source directory, rename fails.
  Thread#1 fails due to deleted file by Thread#2
  </code></pre>
  
  <p>Thus, I tried noop=true and that works but is not a good choice for me as the files in container should be retained or I have very limited memory inside container to work with. 
  So, what can be the good solution regarding the same.
  I am new to camel so can`t pin point exactly what might be the best way to achieving this. </p>
  
  <p>UPDATE:
  I tried even following way :</p>
  
  <pre><code>from(props.splitFolderUri)
     .to("seda:processSplitFiles")
  
  from("seda:processSplitFiles?concurrentConsumers=10")
      .to(log...)
      .to(props.remoteFolderUri)
      ...
  </code></pre>
  
  <p>But I keep on getting following error :</p>
  
  <pre><code>    {"@timestamp":"2018-05-27T22:25:48.220-05:00","@version":1,"message":"Error during commit. Exchange[ID-8c85902bd64c-60121-1527477152921-0-90793]. Caused by:w [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot rename file: GenericFile[/apps/splitFiles/spl/stressTesting.txt_3843_sc_3843_t_10000] to: GenericFile[/apps/splitFiles/spl/.camel/stressTesting.txt_3843_sc_3843_t_10000]]","logger_name":"org.apache.camel.component.file.GenericFileOnCompletion","thread_name":"Camel (camel-1) thread #58 - seda://moveSplitFiles","level":"WARN","level_value":30000,"stack_trace":"org.apache.camel.component.file.GenericFileOperationFailedException: Cannot rename file: GenericFile[/apps/splitFiles/spl/stressTesting.txt_3843_sc_3843_t_10000] to: GenericFile[/apps/splitFiles/spl/.camel/stressTesting.txt_3843_sc_3843_t_10000]
   org.apache.camel.component.file.strategy.GenericFileProcessStrategySupport.renameFile(GenericFileProcessStrategySupport.java:115)
    org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy.commit(GenericFileRenameProcessStrategy.java:88)
     org.apache.camel.component.file.GenericFileOnCompletion.processStrategyCommit(GenericFileOnCompletion.java:127)
      org.apache.camel.component.file.GenericFileOnCompletion.onCompletion(GenericFileOnCompletion.java:83) org.apache.camel.component.file.GenericFileOnCompletion.onComplete(GenericFileOnCompletion.java:57)
       org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(UnitOfWorkHelper.java:104)
        org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWork.java:229)
         org.apache.camel.util.UnitOfWorkHelper.doneUow(UnitOfWorkHelper.java:65)
          org.apache.camel.processor.CamelInternalProcessor$UnitOfWorkProcessorAdvice.after(CamelInternalProcessor.java:654)
           org.apache.camel.processor.CamelInternalProcessor$UnitOfWorkProcessorAdvice.after(CamelInternalProcessor.java:609)
            org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(CamelInternalProcessor.java:239)
             org.apache.camel.processor.Pipeline.process(Pipeline.java:109)
              org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
               org.apache.camel.component.seda.SedaConsumer.sendToConsumers(SedaConsumer.java:298)
                org.apache.camel.component.seda.SedaConsumer.doRun(SedaConsumer.java:207)
                 org.apache.camel.component.seda.SedaConsumer.run(SedaConsumer.java:154)
                  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
                   java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
                    java.lang.Thread.run(Thread.java:748)
                    ","HOSTNAME":"8c85902bd64c"}
  </code></pre>
  
  <p>Please let me know what can be done for this.</p>