Repeated sequential subsetting of data.table
I am not a data.table expert so I can't tell if there is a faster approach, though https://stackoverflow.com/q/16573995/1201032 suggests you are already using something efficient at each step.

The changes I have to suggest would however make your code a lot easier to read and maintain. First, define a function that can be applied at each step:

max_subset &lt;- function(dt, var, by)
  dt[dt[, .I[which.max(get(var))], by = by][,V1]]


Then, you can make your successive calls as follows:

step &lt;- payoff_mat
step &lt;- max_subset(step, "V3", c("V4", "V5"))
step &lt;- max_subset(step, "V2", c("V4"))
step &lt;- max_subset(step, "V1", c())


Or using the magrittr package:

library(magrittr)
payoff_mat %&gt;% max_subset("V3", c("V4", "V5")) %&gt;%
               max_subset("V2", c("V4")) %&gt;%
               max_subset("V1", c()) -&gt; step3


Note that assigning intermediate outputs to a variable in the first approach (step &lt;- ...) should not carry any significant time (feel free to test). And your code will be a lot more readable than using nested calls: max_subset(max_subset(max_subset(...), ..., ...).