title: Apache Hadoop : Can it do "time-varying" input?
body: <p>I haven't found an answer to this even after a bit of googling. My input files are generated by a process which chunks them out at say, when the file touches 1GB. Now, if I were to run a mapreduce job, which processes an input directory in the dfs, how do I make sure that this job picks up the files added to the same input directory, while the hadoop job is running?</p>
  
  <p>I have a feeling this is close to impossible, coz when a hadoop job runs, it would calculate remaining time and all those stuff, so when my input keeps on piling or in other terms is "variable", Hadoop wouldn't know how to manage it - this is my guess. I would like to know your take on this and also on the best possible alternate ways to this! Appreciate your help.</p>