title: How to add a Hodrick-Prescott filtered trend on a pandas groupby?
body: <p>I'm currently stuck trying to get the <a href="https://www.statsmodels.org/dev/generated/statsmodels.tsa.filters.hp_filter.hpfilter.html" rel="nofollow noreferrer">Hodrick-Prescott</a> trend from different groups within a monthly dataset. Here's a replica of the dataset:</p>
  
  <pre><code>import pandas as pd
  import numpy as np
  import statsmodels.api as sm
  
  df = pd.DataFrame(np.random.randint(0,100,size=(100, 4)),
                    columns=list('abcd'))
  
  df['date'] = pd.date_range(start='2018-01-01', 
                             periods=100, freq='M')
  
  df['id'] = ['Group 1', 'Group 2', 'Group 3', 'Group 4'] * 25
  
  df.rename({'a': 'target'}, axis=1, inplace=True)
  
  
  final_df = df.groupby('id', 
                        group_keys=False).apply(
      lambda x: x.sort_values('date'))
  </code></pre>
  
  <p>The Dataset looks like this:</p>
  
  <pre><code>    target  b   c   d   date    id
  0   28  45  17  46  2018-01-31  Group 1
  4   58  23  34  76  2018-05-31  Group 1
  8   30  98  91  79  2018-09-30  Group 1
  12  15  23  25  96  2019-01-31  Group 1
  16  67  45  41  38  2019-05-31  Group 1
  20  28  40  36  38  2019-09-30  Group 1
  24  8   95  28  86  2020-01-31  Group 1
  28  14  53  58  75  2020-05-31  Group 1
  32  46  3   26  61  2020-09-30  Group 1
  36  50  71  80  34  2021-01-31  Group 1
  40  78  38  97  75  2021-05-31  Group 1
  44  15  74  83  25  2021-09-30  Group 1
  48  27  43  18  84  2022-01-31  Group 1
  52  84  38  11  24  2022-05-31  Group 1
  56  23  29  81  22  2022-09-30  Group 1
  60  87  56  92  65  2023-01-31  Group 1
  64  24  99  55  86  2023-05-31  Group 1
  68  16  68  36  63  2023-09-30  Group 1
  72  43  29  80  44  2024-01-31  Group 1
  76  0   48  35  49  2024-05-31  Group 1
  80  17  50  51  51  2024-09-30  Group 1
  84  17  16  40  87  2025-01-31  Group 1
  88  98  13  70  27  2025-05-31  Group 1
  92  21  30  96  87  2025-09-30  Group 1
  96  19  35  32  47  2026-01-31  Group 1
  1   21  45  34  61  2018-02-28  Group 2
  5   35  15  95  11  2018-06-30  Group 2
  9   3   31  94  25  2018-10-31  Group 2
  13  65  89  1   7   2019-02-28  Group 2
  17  77  41  12  58  2019-06-30  Group 2
  ... ... ... ... ... ... ...
  82  32  99  54  27  2024-11-30  Group 3
  86  67  5   71  44  2025-03-31  Group 3
  90  79  94  34  53  2025-07-31  Group 3
  94  4   60  37  85  2025-11-30  Group 3
  98  20  16  32  97  2026-03-31  Group 3
  3   70  63  94  98  2018-04-30  Group 4
  7   2   13  14  5   2018-08-31  Group 4
  11  49  44  20  27  2018-12-31  Group 4
  15  11  60  39  10  2019-04-30  Group 4
  19  22  96  48  5   2019-08-31  Group 4
  23  23  22  30  8   2019-12-31  Group 4
  27  39  11  58  89  2020-04-30  Group 4
  31  61  72  68  78  2020-08-31  Group 4
  35  29  20  7   30  2020-12-31  Group 4
  39  53  20  32  98  2021-04-30  Group 4
  43  97  31  60  74  2021-08-31  Group 4
  47  46  65  15  93  2021-12-31  Group 4
  51  31  24  5   75  2022-04-30  Group 4
  55  42  59  87  68  2022-08-31  Group 4
  59  75  50  62  60  2022-12-31  Group 4
  63  5   24  15  83  2023-04-30  Group 4
  67  77  12  81  44  2023-08-31  Group 4
  71  74  15  11  90  2023-12-31  Group 4
  75  34  0   19  81  2024-04-30  Group 4
  79  2   26  36  98  2024-08-31  Group 4
  83  45  66  9   23  2024-12-31  Group 4
  87  74  67  35  98  2025-04-30  Group 4
  91  69  78  46  7   2025-08-31  Group 4
  95  66  77  91  41  2025-12-31  Group 4
  99  66  11  96  91  2026-04-30  Group 4
  </code></pre>
  
  <p>Here's my current approach:</p>
  
  <pre><code>groups = final_df.groupby('id')
  
  group_keys = list(groups.groups.keys())
  
  bs = pd.DataFrame()
  
  for key in group_keys:
  
      g = groups.get_group(key)
  
      target = g['target']
  
      cycle, trend = sm.tsa.filters.hpfilter(target, lamb=129600)
  
      g['hp_trend'] = trend
  
      bs.append(g)
  </code></pre>
  
  <p>My goal is to simply generate the trend from HP-Filter for each group and append it to that group as a column such that each group will have its own trend based on the target field specified.</p>
  
  <p>Currently, the <code>bs</code> dataframe only returns the empty dataframe that it started with. How can I get the result that I need?</p>
  
  <p>Thanks for reading. </p>