title: Reading fileStream with Spark Streaming
body: <p>I have a directory on HDFS where every <em>10 minutes</em> a file is copied (the existing one is overwritten).
  I'd like to read the content of a file with Spark streaming (<strong>1.6.0</strong>) and use it as a reference data to join it to an other stream.</p>
  
  <p>I set the "<strong>remember window</strong>" <code>spark.streaming.fileStream.minRememberDuration</code> to "<strong>600s</strong>" and set <strong>newFilesOnly</strong> to <strong>false</strong>, because
  when I start the application I wan't to fetch the initial data from HDFS which is already there. </p>
  
  <pre><code>val ssc = new StreamingContext(sparkConf, Seconds(2))
  def defaultFilter(path: Path): Boolean = !path.getName().startsWith(".")
  val lines: DStream[String] = 
     ssc.fileStream[LongWritable, Text, TextInputFormat](loc, defaultFilter(_), false).map(_._2.toString)
  lines.foreachRDD { x =&gt; x.foreach(println) }
  </code></pre>
  
  <p>My idea is to persist the content of this DStream into memory and delegate the task of maintaining 
  this "<strong>batch lookup cache</strong>" to Spark.
  I expect to automatically have fresh data after each change on the HDFS directory which I can join to the other stream.</p>
  
  <p>What I don't understand:</p>
  
  <ul>
  <li>when I start the application the data is loaded but then if
  I touch the file locally and overwrite the one on HDFS I won't see
  its content printed out anymore</li>
  <li>how to cache and reload this data?</li>
  <li>When I cache it will this be available on the worker nodes or
  this (along with the join) will happen in the driver?</li>
  </ul>
  
  <p>Should I also set the <code>StreamingContext</code> time interval to 10 minutes as I will only have changes every 10 minutes?</p>