title: Read file line by line as soon as these lines appear in the file using Apache Camel
body: <p>I have a process A that from time to time writes to a file (newline separated strings, each string is a separate message) and when either that file exceeds N megabytes or after M seconds it deletes the file and creates a new one at the same location. I need to write a program that will fetch messages from this file and if the file is updated, reads another line from it, transforms to some message, applies special business logic and sends it somewhere.</p>
  
  <p>This problem boils down to the following separate problems:</p>
  
  <ul>
  <li>File input component needs to read input up to the nearest newline and do not try to read the whole file at once (again, as I said before writes are occasional and it is critical to read and process each new line separately in the file as soon as it appears in that file).</li>
  <li>File input component needs to track file by file name. Once an old file is deleted, it should start trying to read a new file at the same location once an old one is completely processed.</li>
  </ul>
  
  <p>The overall problem is actually similar to what tools like logstash are solving but I really would not like to use them as I have complex java business logic in place and I want to have automated and extensive test suite around different aspects of message processing as this task is much more critical than log processing.</p>
  
  <p>My current prototype is implemented as an ad hoc Endpoint (I'm almost not using any standard Camel components), it is hairy and not very flexible.</p>
  
  <p>Also: changing the way in which process A communicates its messages is not possible. I would really prefer JMS or something like that but unfortunately it is not an option.</p>
  
  <p>UPDATE 1: I obviously tried to use file component but I had no luck in solving both of file input-related requirements - that component can't read additions to the file done after some time and it can't track file at the same location.</p>
  
  <p>UPDATE 2: Surely, I read a lot of similar Camel-related questions on stackoverflow but they all seem to have much simpler input-related requirements.</p>