title: MapReduce on more than one datastore kind in Google App Engine
body: <p>I just watched <a href="http://code.google.com/intl/en/events/io/2010/sessions/batch-data-processing-app-engine.html" rel="nofollow noreferrer">Batch data processing with App Engine session of Google I/O 2010</a>, read some parts of <a href="http://labs.google.com/papers/mapreduce.html" rel="nofollow noreferrer">MapReduce article from Google Research</a> and now I am thinking to use <a href="http://code.google.com/p/appengine-mapreduce/" rel="nofollow noreferrer">MapReduce on Google App Engine</a> to implement a recommender system in Python.</p>
  
  <p>I prefer using appengine-mapreduce instead of Task Queue API because the former offers easy iteration over all instances of some kind, automatic batching, automatic task chaining, etc. The problem is: my recommender system needs to calculate correlation between instances of two different Models, i.e., instances of two distinct kinds.</p>
  
  <p>Example:
  I have these two Models: User and Item. Each one has a list of tags as an attribute. Below are the functions to calculate correlation between users and items. Note that <code>calculateCorrelation</code> should be called for every combination of users and items:</p>
  
  <pre><code>def calculateCorrelation(user, item):
      return calculateCorrelationAverage(u.tags, i.tags)
  
  def calculateCorrelationAverage(tags1, tags2):
      correlationSum = 0.0
      for (tag1, tag2) in allCombinations(tags1, tags2):
          correlationSum += correlation(tag1, tag2)
      return correlationSum / (len(tags1) + len(tags2))
  
  def allCombinations(list1, list2):
      combinations = []
      for x in list1:
          for y in list2:
              combinations.append((x, y))
      return combinations             
  </code></pre>
  
  <p>But that <code>calculateCorrelation</code> is not a valid Mapper in appengine-mapreduce and maybe this function is not even compatible with MapReduce computation concept. Yet, I need to be sure... it would be really great for me having those appengine-mapreduce advantages like automatic batching and task chaining.</p>
  
  <p><strong>Is there any solution for that?</strong></p>
  
  <p><strong>Should I define my own InputReader?</strong> A new InputReader that reads all instances of two different kinds is compatible with the current appengine-mapreduce implementation?</p>
  
  <p><strong>Or should I try the following?</strong></p>
  
  <ul>
  <li>Combine all keys of all entities of these two kinds, two by two, into instances of a new Model (possibly using MapReduce)</li>
  <li>Iterate using mappers over instances of this new Model</li>
  <li>For each instance, use keys inside it to get the two entities of different kinds and calculate the correlation between them. </li>
  </ul>