Why putting value on GPU slow the calculation?
I'm a new one in GPU parallezation. I found putting values on GPU in advance slows the calculations and indexing. My code is as follows:
import torch

A = torch.rand(600, 600, device='cuda:0')
row0 = torch.tensor(100, device='cuda:0')
col0 = torch.tensor(100, device='cuda:0')
row1 = torch.tensor(356, device='cuda:0')
col1 = torch.tensor(356, device='cuda:0')
B = torch.rand(256, 256, device='cuda:0')
a = 10

%timeit B[:] = A[row0:row1, col0:col1]
# 395 µs ± 4.01 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
%timeit a*A + a**2
# 17 µs ± 256 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

A = torch.rand(600, 600, device='cuda:0')
row0 = 100
col0 = 100
row1 = 356
col1 = 356
B = torch.rand(256, 256, device='cuda:0')
a1 = torch.as_tensor(a).cuda()

%timeit B[:] = A[row0:row1, col0:col1]
# 10.6 µs ± 141 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
%timeit a1*A + a1**2
# 30.2 µs ± 584 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)

Could anyone explain the mechanism behind it?

Your code is slow because there is nothing to parallelize, and you're just taking unnecessary GPU overheads.
GPU parallelism works by launching a large number of threads, and simultaneously computing chunks of some operation. Things like matrix multiplication and convolution are very GPU-friendly, because you can break them down into a lot of similar smaller operations.
However, there is also an overhead when doing things on the GPU.
Only when sufficient amount of threads have been launched to beat the CUDA overhead, we observe a speedup. Let's see an example:
import torch

device = torch.device('cuda:0')

A = torch.randn(5, 10, device=device)
B = torch.randn(10, 5, device=device)
A_ = torch.randn(5, 10)
B_ = torch.randn(10, 5)

%timeit A @ B
# 10.5 µs ± 745 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

%timeit A_ @ B_
# 5.21 µs ± 120 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

You may think this is going against common sense -- how can CPU matrix multiplication be faster than its GPU counterpart? It's simply because we haven't had a big enough operation to parallelize. Let's use retry the same thing, but on bigger inputs:
A = torch.randn(100, 200, device=device)
B = torch.randn(200, 100, device=device)
A_ = torch.randn(100, 200)
B_ = torch.randn(200, 100)

%timeit A @ B
# 10.4 µs ± 333 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

%timeit A_ @ B_
# 45.3 µs ± 647 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)

We increased the input size by a factor of 20. GPU's still basically showing us the same time, the overhead, while CPU time has blown up. Because the input was bigger, GPU parallelism can show its magic.
In your case, you're not doing any parallelization at all. You're simply trying to slice your tensor with a GPU scalar, thus getting some sort of overhead but no benefits. Similar case in the other operation: there is nothing to parallelize.
%timeit a**2
# 200 ns ± 11.9 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)
%timeit a1**2
# 16.8 µs ± 1.43 µs per loop (mean ± std. dev. of 7 runs, 100000 loops each)

There is no way to break the operation a1 ** 2 into smaller repeatable chunks. Knowing when and when not to use the GPU is quite important. This can also be a useful starting point to get a glimpse of how CUDA works under the hood.
