I will start by describing the task that I am performing. I have to repeatedly calculate a grouped sum, typically between 5 and 10 times. The values in the column that I am performing the grouped sum on change with each iteration, but the columns that I am grouping by do not. Below is an example table, in which w, x, and y together constitute the grouping and z is the column whose values will be summed.

```
DT <- data.table(w = sample(1:10, size  = 1000000, replace = TRUE), 
                 x = sample(1:100, size  = 1000000, replace = TRUE), 
                 y = sample(1:1000, size  = 1000000, replace = TRUE),
                 z = runif(n = 1000000, min = 0, max = 1))

setkey(DT, w, x, y)

```

My initial solution was, I think, the most obvious:

```
DT[, Group_Total := sum(z), keyby = list(w, x, y)]

microbenchmark(DT[, Group_Total := sum(z), keyby = list(w, x, y)])
Unit: milliseconds
min       lq        mean      median    uq        max  neval
447.8674  467.3366  481.1989  479.3057  490.5499  691  100

```

I found that this solution was not nearly as fast as I hoped, especially since I had to perform this calculation several times. However, as far as I could tell, there was not a method within data.table that would provide better performance than this.
Recently, I began experimenting with Rccp and I wrote the below function to perform a grouped sum:

```
cppFunction('NumericVector Group_Total(NumericVector Grouping, NumericVector x) {

                          int n = Grouping.size();
                          NumericVector out(n);

                          int curr_start_index = 0;
                          double grp_total = x[0];

                          for(int i = 1; i < (n-1); ++i) {

                            if(Grouping[i] == 1) {

                              for(int j = curr_start_index; j < i; ++j) {

                                out[j] = grp_total;  

                              }

                              curr_start_index = i;
                              grp_total = x[i];

                            } else {

                                grp_total = grp_total + x[i];

                              } 

                            }

                          if(Grouping[(n-1)] == 1) {

                              for(int j = curr_start_index; j < (n-1); ++j) {

                                out[j] = grp_total;  

                              }

                              out[(n-1)] = x[(n-1)];

                            } else {

                                grp_total = grp_total + x[(n-1)];

                                for(int j = curr_start_index; j < n; ++j) {

                                  out[j] = grp_total;  

                                }

                            }

                        return out;
              }')

```

To use this function, I calculate a within group index column once the data is ordered:

```
DT[, Within_Group_Index := 1][, Within_Group_Index := cumsum(Within_Group_Index), keyby = list(w, x, y)]

```

Since I only need to calculate this column once, the time added is not significant. Once this column is created, the grouped sum can be calculated via :

```
DT[, Group_Total := Group_Total(Within_Group_Index, z)]

microbenchmark(DT[, Group_Total := Group_Total(Within_Group_Index, z)])
Unit: milliseconds
min     lq      mean      median  uq      max       neval
5.3286  6.9879  9.078267  7.4814  8.0171  161.6406  100

```

This is faster by a significant margin, and now we come to my question: why?
My understanding is that grouped operations are very efficient in data.table. Is there something that I can do to speed up the data.table grouped operation, or is this an example of a fundamental speed vs. flexibility tradeoff, i.e. the flexibility of data.table's grouped operations comes at the cost of speed when considering specific use cases?