title: Processing log files: Apache Storm or Spark
body: <p>I have a requirement to process log file data. It is relatively trivial. I have 4 servers with 2 web applications running on each for a total of 8 log files. These get rotated on a regular basis. I'm writing data in the following format into these log files</p>
  
  <p>Source Timestamp :9340398;39048039;930483;3940830</p>
  
  <p>Where the numbers are identifiers in a data store. I want to set up a process to read these logs and for each id it will update a count depending on the number of times its id has been logged. It can either be real time or batch. My interface language to the datastore is Java. The process runs in production so needs to be robust but also needs to have a relatively simple architecture so it is maintainable. We also run zookeeper.</p>
  
  <p>My initial thought was to do this in a batch whenever the log file is rotated running an Apache spark on each server. However I then got to looking at log agregators such as Apache Flume, Kafka and Storm, but this seems like overkill.</p>
  
  <p>Given the multitude of choices has anyone got any good suggestions as to which tools to use to handle this problem based on experience?</p>