title: Speeding up pytorch cuda for small-ish tensors
body: <p>For index operations on a tensor of size around 10,000 elements I am finding Pytorch CUDA slower than CPU (whereas if I size up to around 1,000,000,000 elements, CUDA beats CPU).</p>
  <p>According to the profiler (code and results below), most of the execution time seems to be taken by <code>cudaLaunchKernel</code>.</p>
  <p>Unfortunately my requirement is to run lots of different operations in series on tensors around 10,000 elements in size. Is there a way I can speed this up?</p>
  <pre><code>import torch
  from torch.profiler import profile, ProfilerActivity
  
  size=10000
  repeats=1000
  
  def test(device):
      print (&quot;testing &quot;+device)
      device = torch.device(device)
      x = torch.zeros(size,device=device)
      mask = torch.randint(2,(size,),device=device)==1
  
      with profile(activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA]) as prof:
          for _ in range(repeats):
              x[mask]=1
      print(prof.key_averages().table(sort_by=&quot;cpu_time_total&quot;, row_limit=10))
  
  test(&quot;cuda&quot;)
  test(&quot;cpu&quot;)
  </code></pre>
  <p>Output:</p>
  <pre><code>testing cuda
  -------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------
                                                     Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg     Self CUDA   Self CUDA %    CUDA total  CUDA time avg    # of Calls
  -------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------
                                         cudaLaunchKernel        93.93%        2.065s        93.93%        2.065s     344.132us       0.000us         0.00%       0.000us       0.000us          6000
                                      aten::scalar_tensor         0.25%       5.574ms        93.46%        2.055s       2.055ms       0.000us         0.00%       1.001ms       1.001us          1000
                                              aten::fill_         0.24%       5.326ms        93.00%        2.044s       2.044ms       1.001ms         8.99%       1.001ms       1.001us          1000
                                         aten::index_put_         0.19%       4.163ms         6.48%     142.452ms     142.452us       0.000us         0.00%      10.132ms      10.132us          1000
                                   aten::_index_put_impl_         0.81%      17.773ms         6.29%     138.289ms     138.289us       3.005ms        26.99%      10.132ms      10.132us          1000
                                            aten::nonzero         2.33%      51.137ms         4.64%     101.946ms     101.946us       7.127ms        64.02%       7.127ms       7.127us          1000
                                          cudaMemcpyAsync         0.56%      12.363ms         0.56%      12.363ms      12.363us       0.000us         0.00%       0.000us       0.000us          1000
                                                  aten::t         0.15%       3.234ms         0.38%       8.296ms       8.296us       0.000us         0.00%       0.000us       0.000us          1000
                                               aten::set_         0.27%       6.005ms         0.27%       6.005ms       6.005us       0.000us         0.00%       0.000us       0.000us          1000
                                            aten::reshape         0.16%       3.526ms         0.23%       5.117ms       5.117us       0.000us         0.00%       0.000us       0.000us          1000
  -------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------
  Self CPU time total: 2.198s
  Self CUDA time total: 11.133ms
  
  testing cpu
  --------------------------  ------------  ------------  ------------  ------------  ------------  ------------
                        Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg    # of Calls
  --------------------------  ------------  ------------  ------------  ------------  ------------  ------------
            aten::index_put_         2.79%       3.367ms        98.99%     119.352ms     119.352us          1000
      aten::_index_put_impl_        29.47%      35.529ms        96.20%     115.985ms     115.985us          1000
               aten::nonzero        57.51%      69.331ms        57.51%      69.331ms      69.331us          1000
                aten::select         3.16%       3.811ms         4.36%       5.251ms       5.251us          1000
               aten::reshape         2.58%       3.113ms         3.78%       4.554ms       4.554us          1000
            aten::as_strided         2.29%       2.760ms         2.29%       2.760ms       1.380us          2000
                  aten::view         1.20%       1.441ms         1.20%       1.441ms       1.441us          1000
                    aten::to         0.99%       1.191ms         0.99%       1.191ms       1.191us          1000
       cudaDeviceSynchronize         0.02%      22.000us         0.02%      22.000us      22.000us             1
  --------------------------  ------------  ------------  ------------  ------------  ------------  ------------
  Self CPU time total: 120.565ms
  </code></pre>