title: Cross Product in Map Reduce using Hadoop Streaming and Python
body: <p>I am learning Python and Hadoop. I completed the setup and basic examples provided in official site using pythong+hadoop streaming. I considered implementing join of 2 files. I completed equi-join which checks if same key appears in both input files, then it outputs the key along with values from file 1 and file 2 in that order. The equality join is working as it is supposed.</p>
  
  <p>Now, I wish to do inequality join which involves finding Cross Product before applying the inequality condition. I am using the same mapper (do I need to change it) and I changed the reducer so that it contains a nested loop (since every key-value pair in file1 must be matched with all key-values pairs in file2). This doesn't work since you can only go through the stream once. Now, I thought of an option of storing 'some' values in reducer and comparing them but I have no idea 'how' many. Naive method is to store whole file2 content in a array (or similar structure) but thats stupid and goes against the idea of distributed processing. Finally, my questions are</p>
  
  <ol>
  <li><p>How can I store values in reducer so that I can have cross product between two files?</p></li>
  <li><p>In equi-join, Hadoop seems to be sending all key value pairs with same key to same reducer which is perfectly fine and works well for that case. However, how I do change this behaviour (if needed) so that required grouping of key-value pairs go correct reducer?</p></li>
  </ol>
  
  <p>Sample Files:
      <a href="http://pastebin.com/ufYydiPu" rel="noreferrer">http://pastebin.com/ufYydiPu</a></p>
  
  <p>Python Map/Reduce Scripts:
      <a href="http://pastebin.com/kEJwd2u1" rel="noreferrer">http://pastebin.com/kEJwd2u1</a></p>
  
  <p>Hadoop Command I am using:</p>
  
  <pre><code>bin/hadoop jar contrib/streaming/hadoop-*streaming*.jar -file /home/hduser/mapper.py -mapper mapper.py -file /home/hduser/ireducer.py -reducer reducer.py -input /user/hduser/inputfiles/* -output /user/hduser/join-output
  </code></pre>
  
  <p>Any help/hint is much appreciated.</p>