Amazon MapReduce with my own reducer for streaming
Question: I wrote a simple map and reduce program in python to count the numbers for
each sentence, and then group the same number together. i.e suppose sentence 1
has 10 words, sentence 2 has 17 words and sentence 3 has 10 words. The final
result will be:

    
    
    10 \t  2
    17 \t  1
    

The mapper function is:

    
    
        import sys
        import re
    
        pattern = re.compile("[a-zA-Z][a-zA-Z0-9]*")
        for line in sys.stdin:
    
            word = str(len(line.split()))  # calculate how many words for each line
            count = str(1)
            print "%s\t%s" % (word, count)
    

The reducer function is:

    
    
        import sys
    
    
        current_word = None
        current_count = 0
        word = None
    
        for line in sys.stdin:
            line = line.strip()
            word, count = line.split('\t')
            try:
                count = int(count)
                word = int(word)
            except ValueError:
                continue
            if current_word == word:
                current_count += count
            else:
                if current_word:
                    print "%s\t%s" % (current_word, current_count)
                current_count = count
                current_word = word
    
        if current_word == word:
            print "%s\t%s" %(current_word, current_count)
    

I tested on my local machine with the first 200 lines of the file : head -n 200 sentences.txt | python mapper.py | sort | python reducer.py The results are correct. Then I used Amazon MapReduce streaming service, it failed at the reducer step. So I changed the print in the mapper function to:
    
    
       print "LongValueSum" + word + "\t" + "1"
    

This fits into the default aggregate in mapreduce streaming service. In this
case, I don't need the reducer.py function. I get the final results from the
big file sentences.txt. But I don't know why my reducer.py function failed.
Thank you!


Answer: Got it! A "stupid" mistake. When I tested it, I use something like python
mapper.py. But for mapreduce, I need make it executable. So just add

    
    
        # !/usr/bin/env python 
    

in the beginning.

