Why does it take so long print the value of a GPU tensor in Pytorch?
I wrote this pytorch program to compute a 5000*5000 matrix multiplication on GPU, 100 iterations.
import torch
import numpy as np
import time

N = 5000
x1 = np.random.rand(N, N)

######## a 5000*5000 matrix multiplication on GPU, 100 iterations #######
x2 = torch.tensor(x1, dtype=torch.float32).to(&quot;cuda:0&quot;)

start_time = time.time()
for n in range(100):
    G2 = x2.t() @ x2
print(G2.size())
print(&quot;It takes&quot;, time.time() - start_time, &quot;seconds to compute&quot;)
print(&quot;G2.device:&quot;, G2.device)

start_time2 = time.time()
# G4 = torch.zeros((5,5),device=&quot;cuda:0&quot;)
G4 = G2[:5, :5]
print(&quot;G4.device:&quot;, G4.device)
print(&quot;G4======&quot;, G4)
# G5=G4.cpu()
# print(&quot;G5.device:&quot;,G5.device)
print(&quot;It takes&quot;, time.time() - start_time2, &quot;seconds to transfer or display&quot;)

Here is the result on my laptop:

torch.Size([5000, 5000])


It takes 0.22243595123291016 seconds to compute


G2.device: cuda:0


G4.device: cuda:0


G4====== tensor([[1636.3195, 1227.1913, 1252.6871, 1242.4584, 1235.8160],
[1227.1913, 1653.0522, 1260.2621, 1246.9526, 1250.2871],
[1252.6871, 1260.2621, 1685.1147, 1257.2373, 1266.2213],
[1242.4584, 1246.9526, 1257.2373, 1660.5951, 1239.5414],
[1235.8160, 1250.2871, 1266.2213, 1239.5414, 1670.0034]],
device='cuda:0')


It takes 60.13639569282532 seconds to transfer or display
Process finished with exit code 0

I am confused why it takes so much time to display the variable G5 on GPU, since is only 5*5 in size.
BTW, I use &quot;G5=G4.cpu()&quot; to transfer the variable on GPU to CPU, it takes so much time too.
My develop enviroment (rather old laptop):

pytorch 1.0.0

CUDA 8.0

Nvidia GeForce GT 730m

Windows 10 Professional


when increasing iteration times, the compute time do not increase obviously, but the transfer or display increased obviously, Why? Can somebody interpret, thanks so much.

Pytorch CUDA operations are asynchronous. Most operations on GPU tensors are actually non blocking until a derived result is requested. This means that until you ask for a CPU version of a tensor, commands like matrix multiply are basically being processed in parallel to your code. When you stop the timer there's no guarantee that the operation has been completed. You can read more about this in the docs.
To time chunks of your code properly you should add calls to torch.cuda.synchronize. This function should be called twice, once right before you start your timer, and once right before you stop your timer. Outside of profiling your code you should avoid calls to this function as it may slow down overall performance.
