title: Pyspark - FileInputDStream: Error finding new files
body: <p>Hi I'm new to Python Spark and I'm trying out this example from Spark github in order to Counts words in new text files created in the given directory :</p>
  
  <pre><code>import sys
  
  from pyspark import SparkContext
  from pyspark.streaming import StreamingContext
  
  if __name__ == "__main__":
  if len(sys.argv) != 2:
      print("Usage: hdfs_wordcount.py &lt;directory&gt;", file=sys.stderr)
      exit(-1)
  
  sc = SparkContext(appName="PythonStreamingHDFSWordCount")
  ssc = StreamingContext(sc, 1)
  
  lines = ssc.textFileStream("hdfs:///home/my-logs/")
  counts = lines.flatMap(lambda line: line.split(" "))\
                .map(lambda x: (x, 1))\
                .reduceByKey(lambda a, b: a+b)
  counts.pprint()
  
  ssc.start()
  ssc.awaitTermination()
  </code></pre>
  
  <p>And this is what I get : 
  <a href="https://i.stack.imgur.com/8DFcn.png" rel="nofollow noreferrer">a warning saying : WARN FileInputDStream: Error finding new files</a></p>
  
  <p>a warning message saying : <code>WARN FileInputDStream: Error finding new files</code>.</p>
  
  <p>and I got empty results even i'm adding files in this dir :/</p>
  
  <p>Any suggested solution for this ?
  thanks.</p>