MapReduce is not sorting
Question: I'm using python to develop a mapreduce program, when I use map.py and
reduce.py by comand line:

    
    
    cat passengers.dat | python map.py | sort | python reduce.py
    

The result is good. But if I try to use mapreduce:

    
    
    hadoop jar /usr/lib/hadoop-mapreduce/hadoop-streaming-2.2.0.2.0.10.0-1.jar -input passengers.dat -output /out -file map.py -file reduce.py -mapper map.py -reducer  reduce.py
    

The result is wrong because it seems not sorting before reducing.

I draw this conclusion because by command line:

    
    
    cat passengers.dat | python map.py | sort 
    

The result is:

    
    
    141181  2014    5   1   0   STA 267
    141181  2014    5   1   1   END 1031
    141181  2014    5   1   4   STA 1031
    141181  2014    5   1   5   END 267
    

But when I rewrite my reduce to write the lines 'reduce' receives to
understand what is the problem:

    
    
    #!/usr/bin/env python
    import sys 
    for line in sys.stdin:
            print line
    

The result is not sorted:

    
    
    141181  2014    5   1   1   END 1031
    141181  2014    5   1   0   STA 267
    141181  2014    5   1   5   END 267
    141181  2014    5   1   4   STA 1031
    

the fifth column is disordered !!! Why??!

thank you

I post my SOLUTION below!!


Answer: I solved the problem, the problem was that mapreduce uses key-value and there
are too many keys, so the solution I have implemented is to rewrite mapper to
return:

    
    
    return 'value1#value2#value3#value4#value5#value6 \t value7'
    

By this way mapreduce sort perfectly the key and I can split on reducer.

I hope it will be useful

