Why does creating a single tensor on the GPU take 2.5 seconds in PyTorch?
I'm just going through the beginner tutorial on PyTorch and noticed that one of the many different ways to put a tensor (basically the same as a numpy array) on the GPU takes a suspiciously long amount compared to the other methods:

import time
import torch

if torch.cuda.is_available():
    print('time =', time.time())
    x = torch.randn(4, 4)
    device = torch.device("cuda")
    print('time =', time.time())
    y = torch.ones_like(x, device=device)  # directly create a tensor on GPU  =&gt; 2.5 secs??
    print('time =', time.time())
    x = x.to(device)                       # or just use strings ``.to("cuda")``
    z = x + y
    print(z)
    print(z.to("cpu", torch.double))       # ``.to`` can also change dtype together!
    a = torch.ones(5)
    print(a.cuda())
    print('time =', time.time())
else:
    print('I recommend you get CUDA to work, my good friend!')


Output (just times):

time = 1551809363.28284
time = 1551809363.282943
time = 1551809365.7204516  # (!)
time = 1551809365.7236063


Version details:

1 CUDA device: GeForce GTX 1050, driver version 415.27
CUDA          = 9.0.176
PyTorch       = 1.0.0
cuDNN         = 7401
Python        = 3.5.2
GCC           = 5.4.0
OS            = Linux Mint 18.3
Linux kernel  = 4.15.0-45-generic


As you can see this one operation ("y = ...") takes much longer (2.5 seconds) than the rest combined (.003 seconds). I'm confused about this as I expect all these methods to basically do the same. I've tried making sure the types in this line are 32 bit or have different shapes but that didn't change anything.

When I re-order the commands, whatever command is on top takes 2.5 seconds. So this leads me to believe there is a delayed one-time setup of the device happening here, and future on-GPU allocations will be faster.
