Looking for performance improvement of my custom dataframe aggregation function (Python/Pandas)
Normally, I really despise the fact that itertools.groupby only groups adjacent elements with the same key... in your case, though, this seems ideal. Forgive me for just rewriting the code rather than critique'ing what you have, but using this grouping function completely changes how the overall task is best approached.

Let's use itertools.groupby to perform the grouping rather than pandas.groupby specifically because of this adjacency behavior:

In [1]: grouped = itertools.groupby(df.itertuples(False), key=lambda x: x.Id)

In [2]: {k: list(v) for k, v in grouped}
Out[2]:
{'foo': [Pandas(Anticipation=10, Id='foo', Size=10),
  Pandas(Anticipation=9, Id='foo', Size=11),
  Pandas(Anticipation=8, Id='foo', Size=30)],
 'bar': [Pandas(Anticipation=10, Id='bar', Size=10),
  Pandas(Anticipation=9, Id='bar', Size=9),
  Pandas(Anticipation=8, Id='bar', Size=10)],
 'baz': [Pandas(Anticipation=10, Id='baz', Size=7)]}


Note that non-adjacent rows won't be aggregated together:

In [3]: [list(v) for _, v in itertools.groupby([1, 1, 2, 1], key=lambda x: x)]
Out[3]: [[1, 1], [2], [1]]


So then we just need a custom aggregation function. Let's use a generator function that can produce multiple outputs per input, so that we can aggregate each group and output one or more rows as appropriate. It'll operate on a single adjacent group of rows and handle the funkiness of your grouping logic:

def funky_aggregate(k, vs, max_size=10):
    cur_size = 0
    cur_ant = 0
    for v in vs:
        cur_size += v.Size
        cur_ant += v.Anticipation * v.Size
        if cur_size &gt; max_size:
            yield {'Id': k, 'Size': cur_size, 'Anticipation': cur_ant / cur_size}
            cur_size = cur_ant = 0
    if cur_size != 0:
        yield {'Id': k, 'Size': cur_size, 'Anticipation': cur_ant / cur_size}


This can then be easily joined back together into a dataframe:

In [4]: pd.DataFrame([row
            for key, group in itertools.groupby(df.itertuples(False), key=lambda x: x.Id)
            for row in funky_aggregate(key, group)
        ])
Out[4]:
   Anticipation   Id  Size
0      9.476190  foo    21
1      8.000000  foo    30
2      9.526316  bar    19
3      8.000000  bar    10
4     10.000000  baz     7