title: Recency weighted moving average on previous dates in pandas
body: <p>I have the following df:</p>
  
  <pre><code>index = pd.to_datetime(['2017-03-01', '2017-03-01', '2017-02-15', '2017-02-01',
          '2017-01-20', '2017-01-20', '2017-01-20', '2017-01-02', 
          '2016-12-04', '2016-12-04', '2016-12-04', '2016-11-16'])
  
  df = pd.DataFrame(data = {'val': [8, 1, 5, 2, 3 , 5, 9, 14, 13, 2, 1, 12],
                 'group': ['one', 'two', 'one', 'one', 'two', 'two', 'one', 'two', 
                 'two', 'one', 'one', 'two']}, 
                 index=index)
  
  df = df.sort_index()
  
               group val
  2016-11-16   two   12
  2016-12-04   two   13
  2016-12-04   one    2
  2016-12-04   one    1
  2017-01-02   two   14
  2017-01-20   two    3
  2017-01-20   two    5
  2017-01-20   one    9
  2017-02-01   one    2
  2017-02-15   one    5
  2017-03-01   one    8
  2017-03-01   two    1
  </code></pre>
  
  <p>In every group (one, two) I would like to a <strong>recency weighted mean of previous val.</strong> So for example looking at group one:</p>
  
  <pre><code>           group  val
  2016-12-04   one    2
  2016-12-04   one    1
  2017-01-20   one    9
  2017-02-01   one    2
  2017-02-15   one    5
  2017-03-01   one    8
  </code></pre>
  
  <p>For instance, for the date <code>2017-02-15</code>, I wish to calculate a new column having as a value for this date a recency weighted version (higher weights for closer dates in the past) of the previous values that are [2,9,1,2]. Notice there is the possibility to have dates multiple times within one group and those should get the same weight.</p>
  
  <p>I thought pandas exponentially weighted function would be good for this. I figured that is the date within one group is the same I would first take the mean of those values so that I can apply a  simple shift() later. I tried the following:</p>
  
  <pre><code>df =  df.reset_index().set_index(['index', 'group']).groupby(
        level=[0,1]).mean().reset_index().set_index('index')
  </code></pre>
  
  <p>Now if I would not be interested in recency weighting I could to something like</p>
  
  <pre><code>df = df.groupby('group')['val'].expanding().mean().groupby(level=0).shift()
  </code></pre>
  
  <p>and then merge with the original on date and group.
  But when I try to make use of pandas.ewma I am missing something like:</p>
  
  <pre><code>df.groupby('group')['val'].ewm(span=27).groupby(level=0).shift()
  </code></pre>
  
  <p>I can iterate through the groups:</p>
  
  <pre><code>grouped = df.groupby('group')['val']
  for key, group in grouped:
      print pd.ewma(group, span=27).shift()
  
  index
  2016-12-04         NaN
  2017-01-20    1.500000
  2017-02-01    5.388889
  2017-02-15    4.174589
  2017-03-01    4.404414
  Name: val, dtype: float64
  index
  2016-11-16          NaN
  2016-12-04    12.000000
  2017-01-02    12.518519
  2017-01-20    13.049360
  2017-03-01    10.529680
  </code></pre>
  
  <p>and then somehow merge on group and date with the original <code>df</code> but this seems overly complicated. Is there a better way to do this?</p>