title: Cohort analysis using python and mongodb
body: <p>I've made function to keep tracking active users which have been using my app and see their progress. Actually, I've made function in order to implement Cohort analysis of my users!</p>
  
  <p>My idea was (Cohort analysis) to select time period (e.g 01/Oct/2012 up to 01/Oct/2013), take all users that used app first month in my case ( 01/Oct/2012 up to 01/Nov/2012), then tacking their progress in rest of selected months!(I mean "progress", if they used app next month, if they make activities)</p>
  
  <p>So, I made function to do that job, but it's too too slow... Here is my function:</p>
  
  <pre><code>def query(self):
      start = fields['start'].value
      end = fields['end'].value
      start_month, start_year = start.month, start.year
      end_month, end_year = end.month + 1, end.year + 1
      col = self.db[self.__collection__]
      # we only follow users that were in the first month and
      # we have to filter those activities by month
      users_funnel = []
      data = []
      for dt in rrule(MONTHLY, dtstart=start, until=end):
          query = {
              "activities": {
                  "$exists": 1
              },
              "activities.started": {
                  "$exists": 1,
                  "$type": MONGODB_DATE_TYPE,
                  "$gte": dt,
                  "$lt": datetime(dt.year + dt.month / 12, (dt.month % 12) + 1, 1)
              }
          }
  
          if users_funnel:
              query['_id'] = {"$in": users_funnel}
  
          pipeline = [
              {'$project': {
                  '_id': 1,
                  'first_name': 1,
                  'last_name': 1,
                  'email': 1,
                  'activities': 1
              }},
              {'$unwind': "$activities"},
              {"$match": query},
              {'$group': {
                  '_id': {"y": {'$year': "$activities.started"},
                          "m": {'$month': "$activities.started"}},
                  'users': {'$addToSet': "$_id"},
                  'activities_count': {"$sum": 1},
                  'users_data': {"$addToSet": {
                      "_id": "$_id",
                      "first_name": "$first_name",
                      "last_name": "$last_name"}}
              }},
              {"$sort": {
                  "_id.y": 1,
                  "_id.m": 1
              }}
          ]
          results = col.aggregate(pipeline)
          result = results.get("result", [])
          if result:
              data.append(result[0])
              users_funnel = result[0]['users']
      return data
  </code></pre>
  
  <p>Then I've got <strong>Cohort analysis</strong> graph like this <a href="http://cl.ly/image/0C2P3y1d1b38" rel="nofollow">graph</a>.</p>
  
  <p>Problem is about that function that I've made is too slow. I need better solution! </p>
  
  <p>any can help me to develop Cohort analysis function querying mongodb in python ???</p>