title: How do I delete files in hdfs directory after reading it using scala?
body: <p>I use fileStream to read files in the hdfs directory from Spark (streaming context). In case my Spark shut down and starts after some time, I would like to read the new files in the directory. I don't want to read old files in the directory which was already read and processed by Spark. I am trying to avoid duplicates here.</p>
  
  <pre><code>val lines = ssc.fileStream[LongWritable, Text, TextInputFormat]("/home/File")
  </code></pre>
  
  <p>any code snippets to help?</p>