title: pyspark duplicates in workers. Count words in text file
body: <p>I have <strong>3</strong> workers running with a single master.</p>
  <p>It's a <strong>standalone</strong> system</p>
  <p>here is the script</p>
  <pre><code>import sys
   
  from pyspark import SparkContext, SparkConf
   
  if __name__ == &quot;__main__&quot;:
      
      # create Spark context with necessary configuration
          conf = SparkConf()
          conf.setMaster('spark://172.31.60.190:7077')
          conf.setAppName('spark-basic')
          sc = SparkContext(conf=conf)    
      # read data from text file and split each line into words
          words = sc.textFile(&quot;/home/ubuntu/sparkscripts/files/&quot;).flatMap(lambda line: line.split(&quot; &quot;))
      
      # count the occurrence of each word
          wordCounts = words.map(lambda word: (word, 1)).reduceByKey(lambda a,b:a +b)
      
      # save the counts to output
          wordCounts.coalesce(1).saveAsTextFile(&quot;/home/ubuntu/sparkscripts/files/output/&quot;)
  </code></pre>
  <p>A single file gets created with word counts for each word but the word gets repeated exactly 3 times e g the word <strong>'nibh'</strong></p>
  <pre><code>('on', 1)
  ('nibh', 2871678)
  ('amet.', 1147222)
  ('Pulvinar', 337496)
  ('Ridiculus', 51053)
  ('Rhoncus', 215521)
  ('nibh', 9234234)
  ('quam.', 453811)
  ('will', 1)
  ('cras', 2108764)
  ('dignissim.', 303452)
  ('iaculis', 1072101)
  ('bibendum.', 331816
  ('nibh', 234234)
  </code></pre>
  <p>This duplication happens for all words. May I know how to solve this? Thanks</p>