Titan XP vs Quadro P400 GPU in Pytorch
I gave the the two GPUs on my machine a try and I expected the Titan-XP to be faster than the Quadro-P400. However, both gave almost the same execution time. 

I need to know if PyTorch will dynamically choose one GPU over another, or, I myself will have to specify which one PyTorch will use, during run-time.

Here is the code snippet used in the test:

import torch
import time

def do_something(gpu_device):
    torch.cuda.set_device(gpu_device)  # torch.cuda.set_device(device_num)
    print("current GPU device ", torch.cuda.current_device())
    strt = time.time()
    a = torch.randn(100000000).cuda()   
    xx = time.time() - strt
    print("execution time, to create 1E8 random numbers, is ", xx)
    # print(a)
    # print(a + 2)

no_of_GPUs= torch.cuda.device_count()
print("how many GPUs are there:", no_of_GPUs)
for i  in range(0, no_of_GPUs):
    print(i, "th GPU is", torch.cuda.get_device_name(i))
    do_something(i)


Sample output:

how many GPUs are there: 2
0 th GPU is TITAN Xp COLLECTORS EDITION
current GPU device  0
execution time, to create 1E8 random numbers, is  5.527713775634766

1 th GPU is Quadro P400
current GPU device  1
execution time, to create 1E8 random numbers, is  5.511776685714722


Despite what you might believe, the lack of performance difference which you see is because the random number generation is being run on your host CPU not the GPU. If I modify your do_something routine like this:

def do_something(gpu_device, ongpu=False, N=100000000):
    torch.cuda.set_device(gpu_device)
    print("current GPU device ", torch.cuda.current_device())
    strt = time.time()
    if ongpu:
        a = torch.cuda.FloatTensor(N).normal_()
    else:
        a = torch.randn(N).cuda()
    print("execution time, to create 1E8 random no, is ", time.time() - strt)
    return a


and run it two ways, I get very different execution times:

In [4]: do_something(0)
current GPU device  0
execution time, to create 1E8 random no, is  7.736972808837891
Out[4]: 

-9.3955e-01
-1.9721e-01
-1.1502e+00
     ......     
-1.2428e+00
 3.1547e-01
-2.1870e+00
[torch.cuda.FloatTensor of size 100000000 (GPU 0)]

In [5]: do_something(0,True)
current GPU device  0
execution time, to create 1E8 random no, is  0.001735687255859375
Out[5]: 

 4.1403e+06
 5.7016e+06
 1.2710e+07
     ......     
 8.9790e+06
 1.3779e+07
 8.0731e+06
[torch.cuda.FloatTensor of size 100000000 (GPU 0)]


i.e. your version takes 7 seconds and mine takes 1.7ms.  I think it is obvious which one ran on the GPU....
