2.12
   
10.51
   
14.2
   



A10

-2.34
   
8.99
   
10.57
   


V100

18.63
   
6.39
   
10.43
   


A100

38.5
   
20.33
   
12.17
   


One can notice the following:

The improvements are significant for powerful GPUs like A100 and V100. For those GPUs the improvement is most pronounced for batch size 1
For less powerful GPUs we observe smaller speedups (or in two cases slight regressions). The batch size trend is reversed here: improvement is larger for larger batches

In the following sections we describe the applied optimizations and provide detailed benchmarking data, comparing the generation time with various optimization features on/off.