title: How to process new files in HDFS directory once their writing has eventually finished?
body: <p>In my scenario I have CSV files continuously uploaded to HDFS.</p>
  
  <p>As soon as a new file gets uploaded I'd like to process the new file with Spark SQL (e.g., compute the maximum of a field in the file,  transform the file into <code>parquet</code>). i.e. I have a one-to-one mapping between each input file and a transformed/processed output file.</p>
  
  <p>I was evaluating Spark Streaming to listen to the HDFS directory, then to process the "streamed file" with Spark.</p>
  
  <p>However, in order to process the whole file I would need to know when the "file stream" completes. I'd like to apply the transformation to the whole file in order to preserve the end-to-end one-to-one mapping between files.</p>
  
  <p>How can I transform the whole file and not its micro-batches?</p>
  
  <p>As far as I know, Spark Streaming can only apply transformation to batches (<code>DStreams</code> mapped to <code>RDDs</code>) and not to the whole file at once (when its finite stream has completed).</p>
  
  <p>Is that correct? If so, what alternative should I consider for my scenario?</p>