title: hadoop-streaming : writing output to different files
body: <p>Here is the scenario  </p>
  
  <pre><code>           Reducer1  
           /  
  Mapper - - Reducer2  
           \   
             ReducerN  
  </code></pre>
  
  <p>In reducer I want to write the data on different files, lets say the reducer looks like  </p>
  
  <pre><code>def reduce():  
    for line in sys.STDIN:  
      if(line == type1):
        create_type_1_file(line)
      if(line == type2):
        create_type_2_file(line)
      if(line == type3):
        create_type3_file(line)
        ... and so on  
  def create_type_1_file(line):
    # writes to file1  
  def create_type2_file(line):
    # writes to file2  
  def create_type_3_file(line):
    # write to file 3  
  </code></pre>
  
  <p>Consider the paths to write as :  </p>
  
  <pre><code>file1 = /home/user/data/file1  
  file2 = /home/user/data/file2  
  file3 = /home/user/data/file3  
  </code></pre>
  
  <p>When I run in <code>pseudo-distributed mode(machine with one node and hdfs daemons running)</code>, things are good since all daemons will write to the same set of files  </p>
  
  <p>Question: 
  - If I run this in cluster of 1000 machines, will they write to the same set of files even then? I am <code>writing to local filesystem</code> in this case, Is there a better way to perform this operation in <code>hadoop streaming</code>?</p>