text: I know the init process takes more time on machines with more GPUs, but since we are calling the Pytorch script from external scripts it is kind of a bottleneck for our process.
  See the basic example below:
  import torch
  cuda0 = torch.device(‘cuda:0’)
  cuda1 = torch.device(‘cuda:1’)
  cuda2 = torch.device(‘cuda:2’)
  cuda3 = torch.device(‘cuda:3’)
  x0 = torch.tensor([1], device=cuda0)
  x1 = torch.tensor([1], device=cuda1)
  x2 = torch.tensor([1], device=cuda2)
  x3 = torch.tensor([1], device=cuda3)
  So the example takes
  - ~24s on a machine with 8GPUs. Tested with 8x2080TI and 8xA6000.
  - ~15s on machine with 4GPUs. Tested with 4xA6000.
  - ~5s on machine with 2GPUs (in example above removed x2 and x3). Test on 2x3090.
  - ~8s on machine with 8GPUs when init only one GPU (i.e. x1,x2,x3 are removed). Test on 8xA6000
  And since we call it again and again it becomes an issue.
  My understanding is that it is that slow because the initialization process of the GPUs.
  So I was wonder whether I can cache something so it is slow only the first time and then when we call the script again it is faster?