Instruction: Why the computing efficiency of torch.tanh is much higher than the direct expression?

Input:
The two ways of computing 'tanh' are shown as follows. Why the computing efficiency of torch.tanh(1) is much higher than the direct expression(2)? I am confused. And where can I find the original code of torch.tanh in pytorch? Dose it written by C/C++?
import torch
import time

def tanh(x):
    return (torch.exp(x) - torch.exp(-x)) / (torch.exp(x) + torch.exp(-x))

class Function(torch.nn.Module):
    def __init__(self):
        super(Function, self).__init__()
        self.Linear1 = torch.nn.Linear(3, 50)
        self.Linear2 = torch.nn.Linear(50, 50)
        self.Linear3 = torch.nn.Linear(50, 50)
        self.Linear4 = torch.nn.Linear(50, 1)
    def forward(self, x):
        # (1) for torch.torch
        x = torch.tanh(self.Linear1(x))
        x = torch.tanh(self.Linear2(x))
        x = torch.tanh(self.Linear3(x))
        x = torch.tanh(self.Linear4(x))

        # (2) for direct expression
        # x = tanh(self.Linear1(x))
        # x = tanh(self.Linear2(x))
        # x = tanh(self.Linear3(x))
        # x = tanh(self.Linear4(x))

        return x

func = Function()

x= torch.ones(1000,3)

T1 = time.time()
for i in range(10000):
    y = func(x)
T2 = time.time()
print(T2-T1)

Output:
The mathematical functions are writen in higly optimized code, they can use  advanced CPU features and multiple cores, it can even take advantage of GPUs.
in your tanh function it evaluates the exp function four times, does 2 subtraction and one division, creating temporary tensors require memory allocation that can be slow as well, not to mention the overhead of the python interpreter, being 4 to 10 times slow is reasonable.


Instruction: Neural network not training, parameters.grad is None

Input:
I implemented a simple NN and a custom objective function for a minimization problem. Everything seems to be working fine, except for the fact that the model does not seem to learn.
I checked if list(network.parameters())[0].grad was None, and indeed this seems to be the problem. Based on previously asked questions, the problem seems to be the detachment of the graph, but I don't know what I am doing wrong.
Here's the link to the code that you can run on colab: Colab code
Thank you!!

Output:
This part seems problematic in your code.
  output_grad, _ = torch.autograd.grad(q, (x,y))
  output_grad.requires_grad_()

Your loss depends on output_grad and so when you do loss.backward() are trying to compute the gradient of parameters w.r.t to output_grad.
You cannot compute the gradient of output_grad since create_graph is False by default. And so output_grad is implicitly detached from the rest of the graph. To fix this, just pass create_graph=True in the autograd.grad. You do not need to set requires_grad either for output_grad, i.e., the second line is not needed.


Instruction: Alternative concatenation of tensors

Input:
I have 2 tensors of shape [2, 1, 9] and [2, 1, 3]. I'd like to concatenate across the 3rd dimension alternatively (once every 4).
For example:
a = [[[1,2,3,4,5,6,7,8,9]],[[11,12,13,14,15,16,17,18,19]]]
b = [[[10, 20, 30]], [[1, 2, 3]]]
result = [[[1,2,3,10,4,5,6,20,7,8,9,30]],[[11,12,13,1,14,15,16,2,17,18,19,3]]]

How can I do this in pytorch?

Output:
This would do the trick:
torch.concat([a.reshape((2, 1, 3, 3)), b.reshape(2, 1, 3, 1)], axis=-1).reshape((2, 1, -1))

There's probably a smarter way to do this, but hey, it works.


Instruction: How to transform a variable to bucketed variable which tells us which bucket/range it lies to in pytorch

Input:
I have a variable a = [0.129, 0.369, 0.758, 0.012, 0.925]. I want to transform this variable into a bucketed variable. What I mean by this is explained below.
min_bucket_value, max_bucket_value = 0, 1 (Can be anything, for example, 0 to 800, but the min value is always going to be 0)
num_divisions = 10 (For this example I've taken 10, but it can be higher as well, like 80 divisions instead of 10)
Bucket/division ranges are as shown below.
0 - 0.1 -> 0
0.1 - 0.2 -> 1
0.2 - 0.3 -> 2
0.3 - 0.4 -> 3
0.4 - 0.5 -> 4
0.5 - 0.6 -> 5
0.6 - 0.7 -> 6
0.7 - 0.8 -> 7
0.8 - 0.9 -> 8
0.9 - 1.0 -> 9

so, transformed_a = [1, 3, 7, 0, 9]
So it's like I divide min_bucket_value, max_bucket_value in num_divisions  different ranges/buckets and then transform original a to tell which bucket it lies in
I've tried creating torch.linspace(min_bucket_value, max_bucket_value, num_divisions), but not sure how to move forward and map it to a range so that I can get the bucket index to which it belongs to
Can you guys please help
EDIT
There's an extension to this problem.
Let's say that we've got a = [127, 362, 799] and I want to create two buckets. One is a coarse bucket, so a_transform = [12, 36, 89], but what if I want a fine bucket as well so that my second transformation becomes a_fine_transform = [7, 2, 9].
Sub-range index within the range. Basically, coarse division has 80 buckets (giving 127 in 12th bucket) and then the fine bucket which has 10 divisions which tells us that 127 lies in 12th coarse bucket and 7th fine bucket
a can be in float as well. eg, a = [127.36, 362.456, 789.646].
so a_coarse_transform = [12, 36, 78] & a_fine_transform = [7, 2, 6]
where min_bucket_value, max_bucket_value, num_coarse_divisions, num_fine_divisions  = 0, 1, 80, 10

Output:
For torch tensor, you can simply use the following code (part of the answer from Bob and partly different to work it will tensor since numpy won't work unless and until you call .cpu() method on the tensor, which I'm not sure is the right thing to do)
So instead to this,
a1 = (a - min_bucket_value) / (max_bucket_value - min_bucket_value)

a_coarse_transform, r = ((a1 * coarse_divisions)//1).type(torch.long), (a1 * coarse_divisions)%1
a_fine_transform, r = ((r * fine_divisions)//1).type(torch.long), (r * fine_divisions)%1


Instruction: TypeError: relu(): argument 'input' (position 1) must be Tensor, not tuple. I believe it's because I have an LSTM layer

Input: I believe error is because I have an LSTM layer. How can I modify the code so it will work fine? Any help?

Output:
Move the LSTM layer out of the Sequential layer.
LSTM returns a tuple of output, (hn, cn) where hn, cn are the last hidden states.
For example your init function will contain something like
class module(nn.Module):
    def __init__(self):
        super(nn.Module, self).__init__()
        self.lstm = nn.LSTM(...)
        self.seq = nn.Sequential(...)

and your forward function will be
def forward(self, x):
    lstm_out= self.lstm(x)
    out = self.seq(lstm_out[0])
    return out


Instruction: Appending zero rows to a 2D Tensor in PyTorch

Input:
Suppose I have a tensor 2D tensor x of shape (n,m). How can I extend the first dimension of the tensor by appending zero rows in x by specifying the indices of where the zero rows will be located in the resulting tensor? For a concrete example:
x = torch.tensor([[1,1,1],
                  [2,2,2],
                  [3,3,3],
                  [4,4,4]])

And I want to append 2 zero rows such that their row-index will be 1,3, respectively, in the resulting tensor? I.e. in the example the result would be
X = torch.tensor([1,1,1],
                 [0,0,0],
                 [2,2,2],
                 [0,0,0],
                 [3,3,3],
                 [4,4,4]])

I tried using F.pad and reshape.

Output:
You can use torch.tensor.index_add_.
import torch

zero_index = [1, 3]
size = (6, 3)

x = torch.tensor([[1,1,1],
                  [2,2,2],
                  [3,3,3],
                  [4,4,4]])

t = torch.zeros(size, dtype=torch.int64)
index = torch.tensor([i for i in range(size[0]) if i not in zero_index])
# index -> tensor([0, 2, 4, 5])

t.index_add_(0, index, x)
print(t)

Output:
tensor([[1, 1, 1],
        [0, 0, 0],
        [2, 2, 2],
        [0, 0, 0],
        [3, 3, 3],
        [4, 4, 4]])


Instruction: Pytorch Lightning Loss not decreasing

Input:
I've set up a basic UNET model. When using a function to train the model directly, it optimizes fine. However, when using a similar loop in pytorch lightning with the train step defined, the loss does not change from the original value. I took out the zero_grad/backward/step bits based on this tutorial. What am I doing wrong?
# Optimizes well
def train(dataloader, model, loss_fn, optimizer):
    size = len(dataloader.dataset)
    model.train()
    for batch, (X, y) in enumerate(dataloader):
        X, y = X.to('cuda',dtype=torch.float), y.to('cuda',dtype=torch.float)

        # Compute prediction error
        pred = model(X)
        loss = loss_fn(pred, y)

        # Backpropagation
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

# Using this as a function inside the UNet class, which I feed to pytorch_lightning.Trainer.
# Loss does not update from initial value. Model predictions do not improve.
def training_step(self, batch, batch_idx):
    X,y = batch
    X, y = X.to(self.device,dtype=torch.float), y.to(self.device,dtype=torch.float)

    # Compute prediction error
    pred = self.forward(X)
    loss = self.loss_fn(pred, y)
    self.log("train_loss", loss)
    return loss

Output:
This issue was caused by the following line in the model class:
def configure_optimizers(self):
    return super().configure_optimizers()

One of the threads online recommended having this together with training_step and train_dataloader as a minimum set of methods to run pytorch lightning. However, in fact this line interferes with optimization - perhaps, the same batch is loaded every time so that loss does not improve. Simply deleting this method fixes the issue. LightningModule.fit takes in a data loader and uses that to pass batches to training_step.


Instruction: Weird 1d shape result on pytorch 3d ResNet

Input:
I have a 3dResNet model from PyTorch. I also commented out the flatten line in the resnet.py source code so my output shouldn't be 1D.
Here is the code I have:
class VideoModel(nn.Module):
    def __init__(self,num_channels=3):
        super(VideoModel, self).__init__()
        self.r2plus1d = models.video.r2plus1d_18(pretrained=True)
        self.r2plus1d.fc = Identity()
        for layer in self.r2plus1d.children():
            layer.requires_grad_ = False

    def forward(self, x):
        print(x.shape)
        x = self.r2plus1d(x)
        print(x.shape)
        return x

My identity class exists just to ignore a layer:

class Identity(nn.Module):
    def __init__(self):
        super().__init__()

    def forward(self, x):
        return x

When I run torch.randn(1, 3, 8, 112, 112) as my input, I get the following output:
torch.Size([1, 3, 8, 112, 112])
torch.Size([1, 512, 1, 1, 1])

Why do I have a 1D output even though I removed fc layer and the flatten operation?
Is there a better way to remove the flatten operation?

Output:
The cause is the AdaptiveAvgPool3d layer right before the flatten step. It is called with the argument output_size=(1,1,1), and so pools the last three dimensions to (1,1,1) regardless of their original dimensions.
In your case, the output after the average pool has the shape (1,512,1,1,1), after flatten has the shape (1,512), and after the fc layer has the shape (1,400).
So the flatten operation is not responsible, disable the average pool and all subsequent steps to get the desired result.


Instruction: nn.Parameter() doesn't register as a model parameter with torch.randn()

Input:
I'm trying to create a module, which contains certain layers of nn.Parameters().
If I initialize the layer as following -
self.W = nn.Parameter(torch.randn(4,4), requires_grad=True).double()

then this layer doesn't appear to register in the module parameters.
However, this initialization does work -
self.W = nn.Parameter(torch.FloatTensor(4,4), requires_grad=True)

Full example -
class TestNet(nn.Module):
    def __init__(self):
        super(TestNet, self).__init__()
        self.W = nn.Parameter(torch.randn(4,4), requires_grad=True).double()

    def forward(self, x):
        x = torch.matmul(x, self.W.T)
        x = torch.sigmoid(x)
        return x

tnet = TestNet()
print(list(tnet.parameters()))
### Output = [] (an empty list)

Compared to -
class TestNet(nn.Module):
    def __init__(self):
        super(TestNet, self).__init__()
        self.W = nn.Parameter(torch.FloatTensor(4,4), requires_grad=True)

    def forward(self, x):
        x = torch.matmul(x, self.W.T)
        x = torch.sigmoid(x)
        return x

tnet = TestNet()
print(list(tnet.parameters()))

Which prints -
[Parameter containing:
 tensor([[-1.8859e+26,  6.0240e-01,  1.0842e-19,  3.8177e-05],
         [ 1.5229e-27, -8.5899e+09,  1.5226e-27, -3.6893e+19],
         [ 4.2039e-45, -4.6566e-10,  1.5229e-27, -2.0000e+00],
         [ 2.8026e-45,  0.0000e+00,  0.0000e+00,  4.5918e-40]],
        requires_grad=True)]

So what is the difference? Why doesn't the torch.randn() version work?
I couldn't find anything about this in the docs or in previous answers online.

Output:
Calling randn is completely fine. The issue is that .double() is being called at the end of the operation:
class TestNet(nn.Module):
    def __init__(self):
        super(TestNet, self).__init__()
        self.W = nn.Parameter(torch.randn(4,4, dtype = torch.double), requires_grad=True)
        # self.W = nn.Parameter(torch.randn(4,4).double(), requires_grad=True) # also works

    def forward(self, x):
        x = torch.matmul(x, self.W.T)
        x = torch.sigmoid(x)
        return x

tnet = TestNet()
print(tnet.W.dtype)
# torch.float64

print(list(tnet.parameters()))
# [Parameter containing:
# tensor([[-1.9645, -1.5445,  0.2435,  0.4380],
#         [ 1.1403,  0.8836,  0.1811, -0.1212],
#         [ 1.5983, -0.1854, -0.2626,  0.2881],
#         [-1.2364, -0.4802, -0.6038,  0.1164]], requires_grad=True)]

Now the code registers the parameters. I added dtype = torch.double in the initialization of randn to make sure that self.W contains doubles as before.

In summary, we cannot call nn.Parameter, and then register its conversion to another data type as our neural network weights for the deep learning system.


Instruction: Torch custom pairwise distance

Input:
i'm working on big distance matrix (10-80k row ; 3k cols) and i want to get custom pairwise distance on that matrix ; and do it fast.
I have trying with armadillo but with huge data it still "slow"
I try with torch with cuda acceleration and with built in euclidean distance that realy so fast (100 times faster).
So now i want to make custom pairwise distance like :
for pairwise row (a and b): get the standard deviation of ai*bi (where i is cols)
for example :
my_mat:
   |1    |2    |3    |4
a  |5    |3    |0    |4
b  |1    |6    |2    |3

a//b dist = std(5*1,3*6,0*2,4*3)
          = std(5,18,0,12)
          = 7.889867

i think about :
start with my two dimension (N,M) tensor (my_mat)
create a new tensor with 3 dimension (N,N,P) and in P dimension store a "list" with each pairwise product by cols :
3_dim_tens :

   |a                      |b
a  |Pdim(5*5,3*3,0*0,4*4)  |Pdim(5*1,3*6,0*2,4*3)
b  |Pdim(5*1,3*6,0*2,4*3)  |Pdim(5*5,3*3,0*0,4*4)

then if i reduce Pdim by std() i will have 2 dims (N,N) pairwise matrix with my custom distance.
(typically is like matmul my_mat * t(my_mat) but with std in place of addition)
is it possible to do this with torch or is there another way for custom pairwise distance?

Output:
I think the most intuitive way is using einsum for this:
import torch
a = torch.tensor([[5.0, 3, 0, 4],[1, 6, 2, 3]])
b = torch.einsum('ij,kj->ikj', a, a).std(dim=2)
print(b)


Instruction: How can I get the nearest entity in python

Input:
My code:
from mss import mss
import math
import cv2
import numpy as np
import torc

with mss() as sct:
   monitor = {"top": 220, "left": 640, "width": 640, "height":640}

while True:
    screenshot = np.array(sct.grab(monitor))
    results = model(screenshot, size=600)
    df = results.pandas().xyxy[0]
    distances = []
    closest = 1000

       try:

          xmin = int(df.iloc[0, 0])
          ymin = int(df.iloc[0, 1])
          xmax = int(df.iloc[0, 2])
          ymax = int(df.iloc[0, 3])

          centerX = (xmax + xmin) / 2 + xmin
          centerY = (ymax + ymin) / 2 + ymin

          distance2 = math.sqrt(((centerX - 320) ** 2) + ((centerY - 320) ** 2))
          distances.append(distance2)
          if closest > distances[i]:
              closest = distances[i]
              closestEnemy = i

Only problem now is that it doesn't seem to get the closest enemy, is my math wrong? If my math should be wrong, how can I improve it? Also if my math is correct, how can I improve it in order to achieve my goal of getting the nearest entity? Any help will be very appriciated. Thanks in advance to everyone who invests his / her time in helping me :)

Output:
It's not entirely clear, what you are after... but my guess is, that there is a small mistake when calculating the center of the enemies. Either use:
centerX = (xmax + xmin) / 2  # do not add xmin here
centerY = (ymax + ymin) / 2  # do not add ymin here

or calculate the distance between the minimum and maximum values and add the minim again:
centerX = (xmax - xmin) / 2 + xmin  # subtract minimum from maximum
centerY = (ymax - ymin) / 2 + ymin # subtract minimum from maximum

Additional remark:
Performance wise it is mostly not a good idea to iterate over a pandas data frame. Another approach is to add a new column distance to the data frame and then search for the index of the minimum value:
df['distance'] = (
    (( (df['xmax']+df['xmin'])/2 - 320) ** 2) +
    (( (df['ymax']+df['ymin'])/2 - 320) ** 2)
    ) **0.5

closest_enemy = df['distance'].idxmin()


Instruction: Data collation step causing "ValueError: Unable to create tensor..." due to unnecessary padding attempts to extra inputs

Input:
I am trying to fine-tune a Bart model from the huggingface transformers framework on a dialogue summarisation task. The Bart model by default takes in the conversations as a monolithic piece of text as the input and takes the summaries as the decoder input while training. I want to explicitly train the model on dialogue speaker and utterance information rather than waiting for the model to implicitly learn them. For this reason, I am extracting the position IDs of the speaker name tokens and their utterance tokens when I send them to the model along with the original input tokens and summary tokens and send them separately. However, the model's data collator/padding automation expects this information to also be the same size as the inputs (I need to disable this behaviour/change the way I am encoding the speaker to utterance mapping).
Please find the code and description for the above issue below:
I am using the SAMSum dataset for the dialogue summarisation task. The dataset looks like this
Conversation:
Amanda: I baked  cookies. Do you want some?
Jerry: Sure!
Amanda: I'll bring you tomorrow :-)

Summary:
Amanda baked cookies and will bring Jerry some tomorrow.

The conversation gets tokenized as:
tokens = [0, 10127, 5219, 35, 38, 17241, 1437, 15269, 4, 1832, 47, 236, 103, 116, 50121, 50118, 39237, 35, 9136, 328, 50121, 50118, 10127, 5219, 35, 38, 581, 836, 47, 3859, 48433, 2]

The explicit speaker-utterance information is encoded as:
[0, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 2, 2, 2, 2, 2, 1, 1, 2, 2, 2, 2, 2, 2, 2, 0]

Where 1s indicate that tokens[1:3] map to a name "Amanda" and the 2s indicate that tokens[3:16] map to an utterance ": I baked  cookies. Do you want some?"
I am trying to send this speaker utterance association information to the forward function in the hopes of adding a loss on the basis of this information. I intend to override the compute_loss method of the Trainer class from huggingface framework to edit the loss after I can successfully relay this explicit information.
I am currently trying the following:
tokenized_dataset_train = train_datasets.map(preprocess_function, batched=True)

where the preprocess_function tokenizes and adds the speaker-utterance information in the form of a key-value pair. tokenized_dataset_train is of the form {'input_ids':[...], 'attention_mask':[...], 'spk_utt_pos':[...], ...}
The preprocess function makes sure that the lengths for each of 'input_ids', 'attention_masks', and 'spk_utt_pos' is the same.
The data_collator from the DataCollatorForSeq2Seq pads 'input_ids' and 'attention_masks', but also tries to pad 'spk_utt_pos' which gives an error:
Unable to create tensor, you should probably activate truncation and/or padding with 'padding=True' 'truncation=True' to have batched tensors with the same length. Perhaps your features (`spk_utt_pos` in this case) have excessive nesting (inputs type `list` where type `int` is expected).
Upon printing the sizes of 'input_ids', 'attention_masks', and 'spk_utt_pos' inside the train loop during the data collation step I found that the sizes of were not the same.
Example: (A 32 instance batch)
'input_ids' sizes      357 357 357 357 357 357 357 357 357 357 357 357 357 357 357 357 357 357 357 357 357 357 357 357 357 357 357 357 357 357 357 357
'attention_mask' sizes 357 357 357 357 357 357 357 357 357 357 357 357 357 357 357 357 357 357 357 357 357 357 357 357 357 357 357 357 357 357 357 357
'spk_utt_pos' sizes    285 276 276 321 58 93 77 69 198 266 55 107 85 235 47 280 209 357 86 186 27 52 80 77 85 231 266 237 322 125 251 126

My question is: Is there something wrong with my approach to adding this explicit information to my model? What can be another method to send the speaker-utterance information to my model?

Output: I solved this by extending the DataCollatorForSeq2Seq class and overriding the __call__ method in it to also pad my 'spk_utt_pos' list appropriately.


Instruction: How to print the model's parameters'shape and print the parameters while loading a .pt file?

Input:
Thanks to everyone reading this.
I'm a beginner to pytorch. I now have a .pt file and I wanna print the parameter's shape of this module. As I can see, it's a MLP model and the size of input layer is 168, hidden layer is 32 and output layer is 12.
I tried torch.load() but it returned a dict and I don't know how to deal with it. Also, I wanna print the weight of input layer to hidden layer(that maybe a 168*32 matrix) but I don't know how to do that. Thanks for helping me!

Output: The state dictionary of does not contain any information about the structure of forward logic of its corresponding nn.Module. Without prior knowledge about it's content, you can't get which key of the dict contains the first layer of the module... it's possibly the first one but this method is rather limited if you want to beyond just the first layer. You can inspect the content of the nn.Module but you won't be able to extract much more from it, without having the actual nn.Module class at your disposal.


Instruction: Hyperparameter Tuning with Wandb Sweep for custom parameters

Input:
I'm trying to tune the hyperparameters using the Stable-Baseline-3 Library for the network architecture.
My configuration file is:
program: main.py
method: bayes
name: sweep
metric:
  goal: minimize
  name: train/loss
parameters:
  batch_size:
    values: [16, 32, 64, 128, 256, 512, 1024]
  epochs:
    values: [20, 50, 100, 200, 250, 300]
  lr:
    max: 0.1
    min: 0.000001

But if I try to add to the parameters:
  policy_kwargs:
    net_arch:
      pi:
        values: [[ 128, 128 ],[ 256, 256 ],[ 512, 512 ]]
      vf:
        values: [[ 128, 128 ],[ 256, 256 ],[ 512, 512 ]]

I got the following error:
wandb.errors.CommError: Invalid sweep config: invalid hyperparameter configuration: policy_kwargs

Is it possible to use wandb sweep with Stable-Baseline-3 for the network architecture?

Output:
You are trying to create a nested config. Please refer to this documentation here.
Your configuration should be:
program: main.py
method: bayes
name: sweep
metric:
  goal: minimize
  name: train/loss
parameters:
  batch_size:
    values: [16, 32, 64, 128, 256, 512, 1024]
  epochs:
    values: [20, 50, 100, 200, 250, 300]
  lr:
    max: 0.1
    min: 0.000001
  policy_kwargs:
    parameters:
        net_arch:
            parameters:
                pi:
                    values: [[ 128, 128 ],[ 256, 256 ],[ 512, 512 ]]
                vf:
                    values: [[ 128, 128 ],[ 256, 256 ],[ 512, 512 ]]


Instruction: How can I visualize single image with only one convolutional layer and one pooling layers?

Input:
I wrote this sample code to show only a single image after passing it to my model. The model should have only one convolutional layer and one pooling layer. Or in another way, how can I visualize a single image by passing it to a simple neural network that has one convolutional and one pooling layer?
import torch
import torch.nn as nn #creating neural network
from PIL import Image
from numpy import asarray

# Set up GPU
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

# load the image
image = Image.open('./img.png')

# convert image to numpy array
data = asarray(image)
print(type(data))
print(data.shape)

now building the arch.
class ConvNet(nn.Module):
    def __init__(self):
        super().__init__()
        #convolutional layer
        self.layer = nn.Sequential(
            nn.Conv2d(in_channels=3, out_channels=3, kernel_size=2, stride=1, padding=0),
            nn.MaxPool2d(kernel_size=2, stride=2))

    def forward(self, x):
        out = self.layer(x)
        return out

convnet = ConvNet().to(device) #set up for GPU if available
convnet

pass image to my model
outputs = convnet(data) imshow(outputs)
got the error below
TypeError                                 Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_3184/1768392595.py in <module>
----> 1 outputs = convnet(data)
      2 imshow(outputs)

TypeError: conv2d() received an invalid combination of arguments - got (numpy.ndarray, Parameter, Parameter, tuple, tuple, tuple, int), but expected one of:
 * (Tensor input, Tensor weight, Tensor bias, tuple of ints stride, tuple of ints padding, tuple of ints dilation, int groups)
      didn't match because some of the arguments have invalid types: (numpy.ndarray, Parameter, Parameter, tuple, tuple, tuple, int)
 * (Tensor input, Tensor weight, Tensor bias, tuple of ints stride, str padding, tuple of ints dilation, int groups)
      didn't match because some of the arguments have invalid types: (numpy.ndarray, Parameter, Parameter, tuple, tuple, tuple, int)

I expect to show image after passed during this sample network

Output:
as GoodDeeds mentioned, CNN expects the data to be of type Tensor you have read the image using PIL and then converted it to NumPy array, you will need to convert the NumPy array to Tensor using  torch.from_numpy(data)
Below code will solve the issue
import torch
import torch.nn as nn #creating neural network
from PIL import Image
from numpy import asarray

#Set up GPU
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
Her I am loading my image
# load the image
image = Image.open('./img.png')
# convert image to numpy array
data = asarray(image)
data=torch.from_numpy(data)
print(type(data))
print(data.shape)`


Instruction: How to split multi-dimensional arrays based on the unique indices of another array?

Input:
I have two torch tensors a and b:
import torch
torch.manual_seed(0) # for reproducibility

a = torch.rand(size = (5, 10, 1))
b = torch.tensor([3, 3, 1, 5, 3, 1, 0, 2, 1, 2])

I want to split the 2nd dimension of a (which is dim = 1 in the Python language) based on the unique values in b.
What I have tried so far:
# find the unique values and unique indices of b
unique_values, unique_indices = torch.unique(b, return_inverse = True)

# split a in where dim = 1, based on unique indices
l = torch.tensor_split(a, unique_indices, dim = 1)

I was expecting l to be a list of n number of tensors where n is the number of unique values in b. I was also expecting the tensors to have the shape (5, number of elements corresponding to unique_values, 1).
However, I get the following:
print(l)

(tensor([[[0.8198],
         [0.9971],
         [0.6984]],

        [[0.7262],
         [0.7011],
         [0.2038]],

        [[0.1147],
         [0.3168],
         [0.6965]],

        [[0.0340],
         [0.9442],
         [0.8802]],

        [[0.6833],
         [0.7529],
         [0.8579]]]), tensor([], size=(5, 0, 1)), tensor([], size=(5, 0, 1)), tensor([[[0.9971],
         [0.6984],
         [0.5675]],

        [[0.7011],
         [0.2038],
         [0.6511]],

        [[0.3168],
         [0.6965],
         [0.9143]],

        [[0.9442],
         [0.8802],
         [0.0012]],

        [[0.7529],
         [0.8579],
         [0.6870]]]), tensor([], size=(5, 0, 1)), tensor([], size=(5, 0, 1)), tensor([], size=(5, 0, 1)), tensor([[[0.8198],
         [0.9971]],

        [[0.7262],
         [0.7011]],

        [[0.1147],
         [0.3168]],

        [[0.0340],
         [0.9442]],

        [[0.6833],
         [0.7529]]]), tensor([], size=(5, 0, 1)), tensor([[[0.9971]],

        [[0.7011]],

        [[0.3168]],

        [[0.9442]],

        [[0.7529]]]), tensor([[[0.6984],
         [0.5675],
         [0.8352],
         [0.2056],
         [0.5932],
         [0.1123],
         [0.1535],
         [0.2417]],

        [[0.2038],
         [0.6511],
         [0.7745],
         [0.4369],
         [0.5191],
         [0.6159],
         [0.8102],
         [0.9801]],

        [[0.6965],
         [0.9143],
         [0.9351],
         [0.9412],
         [0.5995],
         [0.0652],
         [0.5460],
         [0.1872]],

        [[0.8802],
         [0.0012],
         [0.5936],
         [0.4158],
         [0.4177],
         [0.2711],
         [0.6923],
         [0.2038]],

        [[0.8579],
         [0.6870],
         [0.0051],
         [0.1757],
         [0.7497],
         [0.6047],
         [0.1100],
         [0.2121]]]))

Why do I get empty tensors like tensor([], size=(5, 0, 1)) and how would I achieve what I want to achieve?

Output:
From your description of the desired result:

I was also expecting the tensors to have the shape (5, number of elements corresponding to unique_values, 1).

I believe you are looking for the count (or frequency) of unique values. If you want to keep using torch.unique, then you can provide the return_counts argument combined with a call to torch.cumsum.
Something like this should work:
>>> indices = torch.cumsum(counts, dim=0)
>>> splits = torch.tensor_split(a, indices[:-1], dim = 1)

Let's have a look:
>>> for x in splits:
...     print(x.shape)
torch.Size([5, 1, 1])
torch.Size([5, 3, 1])
torch.Size([5, 2, 1])
torch.Size([5, 3, 1])
torch.Size([5, 1, 1])


Instruction: Add loss in SRGAN

Input:
I want to add loss to SRGAN
https://github.com/leftthomas/SRGAN
in train.py
g_loss = generator_criterion(fake_out, fake_img, real_img)

Can I write a function myself like:
def ContentLoss(a, b):
     result = 0

     for x, y in zip(a, b):
         shape = x.shape
         k = np.prod(shape[0:])
         diff = x - y
         #l2 norm
         diff = np. sqrt(np. sum(np. square(diff)))
         diff = diff*diff
         diff = diff / k
         result = result + diff

     return result

And add it to the original loss as follows:
a = ContentLoss(a,b)
g_loss = generator_criterion(fake_out, fake_img, real_img) + a

Is there a way to calculate the gradient of this loss during training?

Output:
The project you linked uses PyTorch. Assuming that you are also using it, you can just implement your loss using PyTorch instead of numpy and your're covered.
import torch
import math

def ContentLoss(a, b):
     result = 0
     for x, y in zip(a, b):
         shape = x.shape
         k = math.prod(shape[0:])  # you can also use np, but not torch
         diff = x - y
         #l2 norm
         # just replacing np with torch
         diff = torch.sqrt(torch.sum(torch.square(diff)))
         diff = diff*diff  # torch differentiates through these as well
         diff = diff / k
         result = result + diff
     return result

# I assume you only want gradients for a (are these your model outputs?)
# This is just test data anyway. If this is your model output,
# it will pass on the gradients to the weights.
a = [
    torch.tensor([1.0, 1.0, 1.0], requires_grad=True),
    torch.tensor([0.0, -1.0, 1.0], requires_grad=True),
    torch.tensor([7.0, 6.0, -5.0], requires_grad=True),
]
b = [
    torch.tensor([1.0, 0.0, 1.0]),
    torch.tensor([1.0, -2.0, 3.0]),
    torch.tensor([0.0, 0.0, 0.0]),
]

loss = ContentLoss(a, b)
loss.backward()  # computes the gradients
for x in a:
    print(f"a={a}, gradient={a.grad}")

If you need to keep numpy, you have to create a torch.autograd.Function and implement forward and backward: https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html.


Instruction: One hidden layer MLP not training in pytorch

Input:
I tried to implement a simple gradient descent without using an optimizer but the following MLP doesn't train as the loss is always around 2.30. I can't figure out what's wrong with my code . Any advice on what I am doing wrong will be appreciated.(Sorry if the structure of the code is weird, it was implemented in a notebook)
import torch
import numpy as np
import torchvision.datasets as datasets
from torchvision.transforms import ToTensor

# download train and test set
train = datasets.MNIST(root='.data', train=True, download=True, transform=ToTensor())
test = datasets.MNIST(root='.data', train=False, download=True, transform=ToTensor())

# calculate mean and std
mean = train.data.double().mean()
std = train.data.double().std()
print(f"Mean: {mean}, std: {std}")

# standardize data
train.data = (train.data - mean) / std
test.data = (test.data - mean) / std

from torch.utils.data import DataLoader

train_loader1 = DataLoader(train, shuffle=True, batch_size=1)
train_loader2 = DataLoader(train, shuffle=True, batch_size=256)

device = "cuda" if torch.cuda.is_available() else "cpu"
print(f"Device: {device}")

import torch.nn as nn

class SimpleNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.flatten = nn.Flatten() # flatten the 28x28 images to 784 features vectors
        self.in_to_hidden = nn.Linear(784, 784) # input to 784 neurons hidden layer
        self.hidden_to_out = nn.Linear(784, 10) # 784 to 10 neuron output layer
        self.softmax = nn.Softmax(dim=1) # make the output into a probability distribution

    def forward(self, x):
        activation_fn = nn.Tanh()
        x = self.flatten(x)
        x = activation_fn(self.in_to_hidden(x))
        x = self.softmax(self.hidden_to_out(x))
        return x

    def train_single_epoch(self, data_loader, loss_fn, lr, device):
        for x, y in data_loader:
            x, y = x.to(device), y.to(device)

            # calculate loss
            pred = self(x)
            loss = loss_fn(pred, y)

            # backpropagate error and update weights
            loss.backward()
            with torch.no_grad(): # the gradient mustn't be calculated for the weight updates
                for weights in self.parameters():
                    weights -= lr * weights.grad
                    weights.grad.zero_()

        print(f"loss: {loss.item()}")

net = SimpleNN().to(device)
for i in range(2):
    net.train_single_epoch(train_loader2, nn.CrossEntropyLoss(), 0.001, device)

Output:
My first observation is that you're printing the wrong loss value. You are printing the last batch loss at the end of each epoch, which doesn't represent the model's performance for the given epoch.
The right way would be to accumulate batches' losses and then compute the average accros all batches at the end of the epoch.
I modified your train_single_epoch method to print the right epoch loss. I also added an attribute self.list_losses to record the losses and plot them later :
class SimpleNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.flatten = nn.Flatten() # flatten the 28x28 images to 784 features vectors
        self.in_to_hidden = nn.Linear(784, 784) # input to 784 neurons hidden layer
        self.hidden_to_out = nn.Linear(784, 10) # 784 to 10 neuron output layer
        self.softmax = nn.Softmax(dim=1) # make the output into a probability distribution
        self.list_losses = []

    def forward(self, x):
        activation_fn = nn.Tanh()
        x = self.flatten(x)
        x = activation_fn(self.in_to_hidden(x))
        x = self.softmax(self.hidden_to_out(x))
        return x

    def train_single_epoch(self, epoch, data_loader, loss_fn, lr, device):
        epoch_losses = []
        sample_counts = 0
        for x, y in data_loader:
            x, y = x.to(device), y.to(device)

            # calculate loss
            pred = self(x)
            loss = loss_fn(pred, y)
            epoch_losses.append(loss.item()*y.shape[0])
            sample_counts += y.shape[0]

            # backpropagate error and update weights
            loss.backward()
            with torch.no_grad(): # the gradient mustn't be calculated for the weight updates
                for weights in self.parameters():
                    weights -= lr * weights.grad
                    weights.grad.zero_()

        epoch_loss = sum(epoch_losses)/sample_counts
        self.list_losses.append(epoch_loss)
        print(f"[Epoch]: {epoch+1} \t----\t [loss]: {epoch_loss}")

And then I train for 100 epochs and and plot the losses progress across epochs :
net = SimpleNN().to(device)
for epoch in range(100):
    net.train_single_epoch(epoch, train_loader2, nn.CrossEntropyLoss(), 0.001, device)

import matplotlib.pyplot as plt
plt.plot(range(len(net.list_losses)), net.list_losses)
plt.show()

As you can see from the above plot the loss is decreasing, even though quite slowly, but the model is learning.
The model you used here is a toy one (I guess for learning purpose), and that could explain the slow learning we are observing.
There are lots of things you can do to improve learning performance of a neural network (increase model size in terms of depth and layer width, change activation function from tanh to relu, use convolutional network, change learning rate, ...), but that is out of scope of a stackoverflow post.
As an example of improvement I tried to change the learning rate from 0.001 to 0.01 and you can see the model is learning faster now :
net2 = SimpleNN().to(device)
for epoch in range(100):
    net2.train_single_epoch(epoch, train_loader2, nn.CrossEntropyLoss(), 0.01, device)

plt.plot(range(len(net2.list_losses)), net2.list_losses)
plt.show()


Instruction: AttributeError: 'numpy.float64' object has no attribute 'cpu'

Input:
I am trying to run BERT and train a model using pytorch.
I am not sure why I am getting this error after finishing the first Epoch.
I am using this code link
history = defaultdict(list)
best_accuracy = 0

for epoch in range(EPOCHS):

    # Show details
    print(f"Epoch {epoch + 1}/{EPOCHS}")
    print("-" * 10)

    train_acc, train_loss = train_epoch(
        model,
        train_data_loader,
        loss_fn,
        optimizer,
        device,
        scheduler,
        len(df_train)
    )

    print(f"Train loss {train_loss} accuracy {train_acc}")

    # Get model performance (accuracy and loss)
    val_acc, val_loss = eval_model(
        model,
        val_data_loader,
        loss_fn,
        device,
        len(df_val)
    )

    print(f"Val   loss {val_loss} accuracy {val_acc}")
    print()

    history['train_acc'].append(train_acc.cpu())
    history['train_loss'].append(train_loss.cpu())
    history['val_acc'].append(val_acc.cpu())
    history['val_loss'].append(val_loss.cpu())

    # If we beat prev performance
    if val_acc > best_accuracy:
        torch.save(model.state_dict(), 'best_model_state.bin')
        best_accuracy = val_acc

Here is the output and the error message
Image
It is a first time for me to work with pytorch. Any ideas how to fix the error>

Output:
I checked kaggle link and I see that there is no cpu() reference as you have posted in your code. It should simply be:
history['train_acc'].append(train_acc)
history['train_loss'].append(train_loss)
history['val_acc'].append(val_acc)
history['val_loss'].append(val_loss)


Instruction: Pytorch gradient descent keeps sending me NaNs mean squared errors

Input:
I am trying to apply, within the framework of a course, a gradient descent to estimate a linear model. My code is the following :
model = torch.nn.Linear(1,1)
myModel = model(X)
ds = torch.utils.data.TensorDataset(X, Y)
dl = torch.utils.data.DataLoader(ds)
optimiser = torch.optim.SGD(model.parameters(), lr=0.01)
loss = torch.nn.functional.mse_loss

for epoch in range(100):
    for (Xb, yb) in dl:
        yb_pred = model(Xb)
        c_loss = loss(yb_pred, yb)
        print(c_loss)
        optimiser.zero_grad()
        c_loss.backward()
        optimiser.step()

Yet it keeps printing NaNs, which I do not understand. Have I done a mistake in the implementation ? I have the following output (x numerous times) :

tensor(nan, grad_fn=<MseLossBackward0>)

Output:
There is nothing wrong with your code but Nan values can be explained by gradients exploding depending on the data X and Y. You can try with a lower learning rate (1e-3 or 1e-4).
For instance if you test with this toy linear example:
X = torch.randn(100, 1)
Y = X * 2 + 3

The loss will converge to 0 quickly.


Instruction: Display variable label instead of value

Input:
I created two lists as dropdown menus in streamlit.
list = [2, 3, 5, 7, 11]
list1 = [1, 2, 5, 7, 11]
select_options = [list,list1]

dropdown = st.selectbox('Select Option', options=select_options)

Right now the dropdown menu displays 2,3,5,7,11 & 1,2,5,7,11. Is there a way to display the variable label instead so the dropdown menu would show list & list1?

Output:
You can have two select boxes, the first selectbox is to make the choice as to which list you want to use and the second selectbox will let you make the choice of the elements in the selected list.
Note: It's not appropriate to name a variable after list because it is a python built-in function. So I renamed your list variable to list1
import streamlit as st

list1 = [2, 3, 5, 7, 11]
list2 = [1, 2, 5, 7, 11]

get_list = st.selectbox('Select List Option', ("List1", "List2"))
select_options = []
if get_list == "List1":
    select_options = list1
elif get_list == "List2":
    select_options = list2

dropdown = st.selectbox(f'Select Option of {get_list}:', select_options)

Output:


Instruction: Unexpected calculation of number of trainable parameters (Pytorch)

Input:
Consider the following code
from torch import nn
from torchsummary import summary
from torchvision import models

model = models.efficientnet_b7(pretrained=True)
model.classifier[-1].out_features = 4   # because i have a 4-class problem; initially the output is 1000 classes
model.classifier = nn.Sequential(*model.classifier, nn.Softmax(dim=1))  # add softmax

# freeze features
for child in model.features:
    for param in child.parameters():
        param.requires_grad = False

When I run
model.classifier

I get the below (expected) output

which as per my calculations implies that the total trainable parameters should be (2560 + 1) * 4 output nodes = 10244 tranable params.
However, when i attempt to calculate the total number of trainable params by
summary(model, (3,128,128))

I get

and by
sum(p.numel() for p in model.parameters() if p.requires_grad)

I get

The 2,561,000, in both cases, comes from (2560 + 1) * 1000 classes.
But, why does it still consider 1000 classes though ?

Output: Resetting an attribute of an initialized layer does not necessarily re-initialize it with the newly-set attribute. What you need is model.classifier[-1] = nn.Linear(2560, 4).


Instruction: TorchVision using pretrained weights for entire model vs backbone

Input: TorchVision Detection models have a weights and a weights_backbone parameter. Does using pretrained weights imply that the model uses pretrained weights_backbone under the hood? I am training a RetinaNet model and um unsure which of the two options I should use and what the differences are.

Output:
The difference is pretty simple: you can either choose to do transfer learning on the backbone only or on the whole network.
RetinaNet from Torchvision has a Resnet50 backbone. You should be able to do both of:

retinanet_resnet50_fpn(weights=RetinaNet_ResNet50_FPN_Weights.COCO_V1)
retinanet_resnet50_fpn(backbone_weights=ResNet50_Weights.IMAGENET1K_V1)

As implied by their names, the backbone weights are different. The former were trained on COCO (object detection) while the later were trained on ImageNet (classification).
To answer your question, pretrained weights implies that the whole network, including backbone weights, are initialized. However, I don't think that it calls backbone_weights under the hood.


Instruction: Enabling CAFFE2 while building pytorch from source on Windows command prompt

Input:
So, I was doing a model train up using Yolo7 on Windows platform and

C:\Users\LENOVO>python train.py --weights yolov7.pt --data "data/custom.yaml" --workers 4 --batch-size 4 --img 416 --cfg cfg/training/yolov7.yaml --name yolov7 --hyp data/hyp.scratch.p5.yaml

After running the above command the below stack trace of error showed up in my command prompt on windows. My question is:

How to do the suggestions of the error below? How to enable the BUILD_CAFFE2=1 while building pytorch on my Windows? Not using Conda of course. On my Windows command prompt only.
I installed pytorch from using the following source
https://github.com/pytorch/pytorch#from-source
Install caffe2 using commands from this source
https://caffe2.ai/docs/getting-started.html?platform=windows&configuration=compile
But the following error still shows while training my model.

I just need to know the command of enabling build_caffe2=1 on windows command prompt.
    C:\Users\LENOVO\AppData\Local\Programs\Python\Python310\lib\site-packages\caffe2\__init__.py:5: UserWarning: Caffe2 support is not fully enabled in this PyTorch build. Please enable Caffe2 by building PyTorch from source with `BUILD_CAFFE2=1` flag.
     warnings.warn("Caffe2 support is not fully enabled in this PyTorch build. "
    C:\Users\LENOVO\AppData\Local\Programs\Python\Python310\lib\site-packages\caffe2\proto\__init__.py:17: UserWarning: Caffe2 support is not enabled in this PyTorch build. Please enable Caffe2 by building PyTorch from source with `BUILD_CAFFE2=1` flag.
     warnings.warn('Caffe2 support is not enabled in this PyTorch build. '
    C:\Users\LENOVO\AppData\Local\Programs\Python\Python310\lib\site-packages\caffe2\python\__init__.py:9: UserWarning: Caffe2 support is not enabled in this PyTorch build. Please enable Caffe2 by building PyTorch from source with `BUILD_CAFFE2=1` flag.
     warnings.warn('Caffe2 support is not enabled in this PyTorch build. '
    Traceback (most recent call last):
     File "C:\Users\LENOVO\train.py", line 8, in <module>
    from caffe2.python import core, scope
      File "C:\Users\LENOVO\AppData\Local\Programs\Python\Python310\lib\site-packages\caffe2\python\__init__.py", line 7, in <module>
    from caffe2.proto import caffe2_pb2
      File "C:\Users\LENOVO\AppData\Local\Programs\Python\Python310\lib\site-packages\caffe2\proto\__init__.py", line 15, in <module>
    from caffe2.proto import caffe2_pb2, metanet_pb2, torch_pb2
    ImportError: cannot import name 'metanet_pb2' from partially initialized module 'caffe2.proto' (most likely due to a circular import) (C:\Users\LENOVO\AppData\Local\Programs\Python\Python310\lib\site-packages\caffe2\proto\__init__.py)

Output:
I have solved the issue but setting BUILD_CAFFE2=1 on the command prompt before installing pytorch, with the following code.
set BUILD_CAFFE2=1


Instruction: Why is my mac1 gpu not working in pytorch?

Input:
python train_torch.py --train --max_epochs 2
PyTorch version:1.12.1
MPS 장치를 지원하도록 build 되었는지: True
MPS 장치가 사용 가능한지: True

INFO:root:Namespace(chat=False, sentiment='0', model_params='model_chp/model_-last.ckpt', train=True, max_len=32, batch_size=40, lr=5e-05, warmup_ratio=0.1, logger=True, checkpoint_callback=True, default_root_dir=None, gradient_clip_val=0, process_position=0, num_nodes=1, num_processes=1, gpus=None, auto_select_gpus=False, tpu_cores=None, log_gpu_memory=None, progress_bar_refresh_rate=None, overfit_batches=0.0, track_grad_norm=-1, check_val_every_n_epoch=1, fast_dev_run=False, accumulate_grad_batches=1, max_epochs=2, min_epochs=None, max_steps=None, min_steps=None, limit_train_batches=1.0, limit_val_batches=1.0, limit_test_batches=1.0, limit_predict_batches=1.0, val_check_interval=1.0, flush_logs_every_n_steps=100, log_every_n_steps=50, accelerator=None, sync_batchnorm=False, precision=32, weights_summary='top', weights_save_path=None, num_sanity_val_steps=2, truncated_bptt_steps=None, resume_from_checkpoint=None, profiler=None, benchmark=False, deterministic=False, reload_dataloaders_every_epoch=False, auto_lr_find=False, replace_sampler_ddp=True, terminate_on_nan=False, auto_scale_batch_size=False, prepare_data_per_node=True, plugins=None, amp_backend='native', amp_level='O2', distributed_backend=None, automatic_optimization=None, move_metrics_to_cpu=False, enable_pl_optimizer=None, multiple_trainloader_mode='max_size_cycle', stochastic_weight_avg=False)
GPU available: False, used: False
TPU available: False, using: 0 TPU cores

-> AttributeError: Can't pickle local object 'get_cosine_schedule_with_warmup..lr_lambda'
To do deep learning,
python train_torch.py --train --max_epochs 2
Syntax used successfully.
However, the following error appears and it says accelerator=None
I don't know what to do. I ask for your help me.
I was able to see the tracing in Google colab.

Output:
I think the pytorch version you are using does not have mps acceleration, you should download the nightly version of pytorch. Pytorch version 1.13 does how mps acceleration.

It will have some issues because it is not the stable version.


Instruction: how to avoid TO_TENSOR() clips values to 1

Input:
I have a black and white image that needs to be converted into tensor.
The shape of the image is (400, 600, 3).
Originally, the values of the image have max = 255; for example:
org_img[0]

# result:
array([[255, 255, 255],
               [255, 255, 255],
               [255, 255, 255],
               ...,
               [255, 255, 255],
               [255, 255, 255],
               [255, 255, 255]], dtype=uint8)

but after i convert into tensor using to_tensor(), it clips my value into 1.
torchvision.transforms.functional.to_tensor(org_img[0])
# result:
tensor([[[1., 1., 1.],
         [1., 1., 1.],
         [1., 1., 1.],
         ...,
         [1., 1., 1.],
         [1., 1., 1.],
         [1., 1., 1.]]])

This makes my image all black, how can i avoid this problem?
Thank you.

Output: The images are not clipped but instead re-scaled from uint8 0..255 to float32 [0, 1] by ToTensor. Library such as matplotlib can naturally handle RGB images with single-precision pixel values within [0, 1] after re-scaling.


Instruction: Visualising the last layer node embeddings of a model in torch geometric

Input:
I'm doing my first graph convolutional neural network project with torch_geometric. I want to visualize the last layer node embeddings of my model and don't know how I should get it.
I trained my model on the CiteSeer dataset. You can get the full dataset as easily as this:
from torch_geometric.datasets import Planetoid
from torch_geometric.transforms import NormalizeFeatures
dataset = Planetoid(root="data/Planetoid", name='CiteSeer', transform=NormalizeFeatures())

My model is a simple two-layered model as this:
class GraphClassifier(torch.nn.Module):
    def __init__(self, dataset, hidden_dim):
        super(GraphClassifier, self).__init__()
        self.conv1 = GCNConv(dataset.num_features, hidden_dim)
        self.conv2 = GCNConv(hidden_dim, dataset.num_classes)

    def forward(self, data):
        x, edge_index = data.x, data.edge_index
        x = F.relu(self.conv1(x, edge_index))
        x = F.relu(self.conv2(x, edge_index))
        return F.log_softmax(x, dim=1)

If you print my model you will get this:
model = GraphClassifier(dataset, 64)
print(model)

>>>
GraphClassifier(
  (conv1): GCNConv(3703, 64)
  (conv2): GCNConv(64, 6)
)

My model is trained successfully. I only want to visualize its last-layer node embeddings. To visualize that I have this function to use:
%matplotlib inline
import matplotlib.pyplot as plt
from sklearn.manifold import TSNE
import torch
# emb: (nNodes, hidden_dim)
# node_type: (nNodes,). Entries are torch.int64 ranged from 0 to num_class - 1

def visualize(emb: torch.tensor, node_type: torch.tensor):
  z = TSNE(n_components=2).fit_transform(emb.detach().cpu().numpy())
  plt.figure(figsize=(10,10))
  plt.scatter(z[:, 0], z[:, 1], s=70, c=node_type, cmap="Set2")
  plt.show()

I don't know how I should extract emb and node_type from my model to give to the visualize function. emb is the last layer of node embeddings of the model. How can I get these from my model?

Output:
It is solve by changing the model to this:
class GraphClassifier(torch.nn.Module):
    def __init__(self, dataset, hidden_dim):
        super(GraphClassifier, self).__init__()
        self.conv1 = GCNConv(dataset.num_features, hidden_dim)
        self.conv2 = GCNConv(hidden_dim, dataset.num_classes)

    def forward(self, data, do_visualize=False):
        x, edge_index = data.x, data.edge_index
        x = F.relu(self.conv1(x, edge_index))
        x = F.relu(self.conv2(x, edge_index))
        if do_visualize:                          # NEW LINE
            visualize(x, data.y)                  # NEW LINE
        return F.log_softmax(x, dim=1)

Now if you call the forward function with do_visualize=Ture it will visualize. like this:
model = GraphClassifier(dataset, hidden_dim)
model.to(device)
model(dataset[0].to(device), do_visualize=True)


Instruction: How can I convert this tensoflow code to pytorch?

Input:
How can I convert this tensoflow code to pytorch?
#tensoflow
Conv2D(
    self.filter_1, (1, 64),
    activation='elu',
    padding="same",
    kernel_constraint=max_norm(2., axis=(0, 1, 2))
)

nn.Sequential(
        nn.Conv2D(16, (1, 64),
        padding="same",
        kernel_constraint=max_norm(2., axis=(0, 1, 2)),
        nn.ELU()
)

Output:
You need two things:

You need to know what the input channel size is. In your example, you've only given the number of output channels, 16. Keras calculates this on its own during runtime, but you have to specify input channels when making torch nn.Conv2d.
You need to implement the max_norm constraint on the conv kernel yourself.

With this in mind, let's write a simple wrapper around the nn.Conv2d, that just enforces the constraint on the weights each time forward is called:
import torch
from torch import nn
import torch.nn.functional as F

class Conv2D_Norm_Constrained(nn.Conv2d):
    def __init__(self, max_norm_val, norm_dim, **kwargs):
        super().__init__(**kwargs)
        self.max_norm_val = max_norm_val
        self.norm_dim = norm_dim

    def get_constrained_weights(self, epsilon=1e-8):
        norm = self.weight.norm(2, dim=self.norm_dim, keepdim=True)
        return self.weight * (torch.clamp(norm, 0, self.max_norm_val) / (norm + epsilon))

    def forward(self, input):
        return F.conv2d(input, self.get_constrained_weights(), self.bias, self.stride, self.padding, self.dilation, self.groups)

Assuming your input channels are something like 8, we can write:
nn.Sequential(
    Conv2D_Norm_Constrained(in_channels=8, out_channels=16, kernel_size=(1, 64), padding="same", max_norm_val=2.0, norm_dim=(0, 1, 2)),
    nn.ELU()
)


Instruction: LibTorch (PyTorch C++) LNK2001 errors

Input:
I was following the tutorial in LibTorch here.
With the following changes:

example-app => Ceres
example-app.cpp => main.cxx

Everything worked until the CMake command cmake --build . --config Release.
It produced the following errors:
main.obj : error LNK2001: unresolved external symbol __imp___tls_index_?init@?1??lazy_init_num_threads@internal@at@@YAXXZ@4_NA [D:\Silverous Black\CPE42S2-CPE42S2\CPE 406\ProjectDumagan\src\Ceres\build\Ceres.vcxproj]
main.obj : error LNK2001: unresolved external symbol __imp___tls_offset_?init@?1??lazy_init_num_threads@internal@at@@YAXXZ@4_NA [D:\Silverous Black\CPE42S2-CPE42S2\CPE 406\ProjectDumagan\src\Ceres\build\Ceres.vcxproj]
D:\Silverous Black\CPE42S2-CPE42S2\CPE 406\ProjectDumagan\src\Ceres\build\Release\Ceres.exe : fatal error LNK1120: 2 unresolved externals [D:\Silverous Black\CPE42S2-CPE42S2\CPE 406\ProjectDumagan\src\Ceres\build\Ceres.vcxproj]

I don't believe these are from the changes I placed, since the problem is with the linking.
I also am trying to replicate this directly into Visual Studio. I am using Visual Studio 17 2022 which the LibTorch extension is not yet compatible with (Visual Studio 16 2019 is no longer available for install from the website).
The replication is through a blank C++ template (no starting files). And I have set the following macros:

LibTorchTarget = CPU specifies libtorch for CPU is to be used (useful for other macros
LibTorchDir = C:/libtorch/ directory where the libtorch installation(s) can be found (for multiple installations)
LibTorchInstall = $(LibTorchDir)libtorch_$(LibTorchTarget)/ expresses to C:/libtorch/libtorch_CPU/
LibTorchInclude = $(LibTorchInstall)include/ expresses to C:/libtorch/libtorch_CPU/include/
LibTorchLib = $(LibTorchInstall)lib/ expresses to C:/libtorch/libtorch_CPU/lib/

And have put the Include and Lib macros at their respective VC++ Directories positions. As well as $(LibTorchLib)*.lib (C:/libtorch/libtorch_CPU/lib/*.lib) in the Linker > Input > Additional Dependencies to specify all the .libs for linking (prevents a lot of LNK2009 errors).
And lastly, I have put start xcopy /s "$(LibTorchLib)*.dll" "$(OutDir)" /Y /E /D /R command at the Build Events > Pre-Link Event > Command Line to replicate the if clause in the CMakeLists.txt in the tutorial (apparently to avoid memory errors).
Result is same exact error with a final LNK1120 error:
Error   LNK2001 unresolved external symbol __imp___tls_index_?init@?1??lazy_init_num_threads@internal@at@@YAXXZ@4_NA    Ceres   D:\Silverous Black\CPE42S2-CPE42S2\CPE 406\ProjectDumagan\src\Ceres\main.obj    1
Error   LNK2001 unresolved external symbol __imp___tls_offset_?init@?1??lazy_init_num_threads@internal@at@@YAXXZ@4_NA   Ceres   D:\Silverous Black\CPE42S2-CPE42S2\CPE 406\ProjectDumagan\src\Ceres\main.obj    1
Error   LNK1120 2 unresolved externals  Ceres   D:\Silverous Black\CPE42S2-CPE42S2\CPE 406\ProjectDumagan\out\Debug_64\Ceres\Ceres.exe  1

I don't exactly understand the reason for the LNK errors, so if anyone could help that'll be really nice. Thank you in advance.

Output:
Look to: Updating to Visual Studio 17.4.0 Yields linker errors related to TLS
You most likely need to rebuild PyTorch after MSVC update.


Instruction: Why does pytorch's transforms.Normalize() not do the described action from the documentation?

Input:
According to the documentation normalize is supposed to do (tensor - mean)/std, but it doesn't. Why?
Docs:

Normalize a tensor image with mean and standard deviation.
Given mean: (mean[1],...,mean[n]) and std: (std[1],..,std[n]) for n
channels, this transform will normalize each channel of the input
torch.*Tensor i.e.,
output[channel] = (input[channel] - mean[channel]) / std[channel]

a = T.Tensor([[[1, 2, 3], [4, 5, 6], [7, 8, 9]]])
m = a.mean()
std = a.std()
print((m, std))
print(transforms.Normalize(mean, std)(T.unsqueeze(a, 0)))
print(transforms.Normalize(mean, std)(T.unsqueeze(a, 0)).mean())
print(transforms.Normalize(mean, std)(T.unsqueeze(a, 0)).std())
a = (a - m)/std
m = a.mean()
std = a.std()
print((m, std))

Output:
(tensor(5.), tensor(2.7386))
tensor([[[[1.0150, 1.3802, 1.7453],
          [2.1105, 2.4756, 2.8408],
          [3.2059, 3.5711, 3.9362]]]])
tensor(2.4756)
tensor(1.0000)
(tensor(0.), tensor(1.0000))

The std is correct, but the mean is something random. What gives?

Output:
Mean value of your tensor is stored in variable m, not mean.
After replace m with mean on Line 2
a = T.Tensor([[[1, 2, 3], [4, 5, 6], [7, 8, 9]]])
mean = a.mean()
std = a.std()
print((mean, std))
print(transforms.Normalize(mean, std)(T.unsqueeze(a, 0)))
print(transforms.Normalize(mean, std)(T.unsqueeze(a, 0)).mean())
print(transforms.Normalize(mean, std)(T.unsqueeze(a, 0)).std())
a = (a - mean)/std
mean = a.mean()
std = a.std()
print((mean, std))

Output:
(tensor(5.), tensor(2.7386))
tensor([[[[-1.4606, -1.0954, -0.7303],
          [-0.3651,  0.0000,  0.3651],
          [ 0.7303,  1.0954,  1.4606]]]])
tensor(0.)
tensor(1.0000)
(tensor(0.), tensor(1.0000))


Instruction: Using detectron2 for training, pytorch's runtime error 'default process group has not been initialized'

Input:
Instructions To Reproduce the Issue:

Full runnable code or full changes you made:

I tried to train DeepLabV3+ architecture with a customized config having ResNet18 (converted to .pkl from https://download.pytorch.org/models/resnet18-f37072fd.pth) as the backbone:
_BASE_: "detectron2/projects/DeepLab/configs/Cityscapes-SemanticSegmentation/deeplab_v3_plus_R_103_os16_mg124_poly_90k_bs16.yaml"
MODEL:
  WEIGHTS: "r18.pkl"
  BACKBONE:
    NAME: "build_resnet_backbone"
  RESNETS:
    DEPTH: 18
    RES2_OUT_CHANNELS: 64
    STEM_OUT_CHANNELS: 64
    RES5_DILATION: 1
    NUM_GROUPS: 1
  ROI_HEADS:
    NUM_CLASSES: 1

on MoNuSeg 2020 dataset.

What exact command you run:

cfg = get_cfg()
add_deeplab_config(cfg)
cfg.merge_from_file('/kaggle/input/deeplab-v3-plus-models/deeplab_v3_plus_R_18_os16_mg124_poly_90k_bs16.yaml')
cfg.DATASETS.TRAIN = ('monuseg_train',)
cfg.DATASETS.TEST = ()
cfg.DATALOADER.NUM_WORKERS = 2
cfg.SOLVER.IMS_PER_BATCH = 16
cfg.SOLVER.BASE_LR = 0.01
cfg.SOLVER.MAX_ITER = 300
cfg.SOLVER.LR_SCHEDULER_NAME = 'WarmupMultiStepLR'
cfg.SOLVER.STEPS = []
cfg.MODEL.ROI_HEADS.NUM_CLASSES = 1

os.makedirs(cfg.OUTPUT_DIR, exist_ok=True)
trainer = DefaultTrainer(cfg)
trainer.resume_or_load(resume=False)
trainer.train()

I also used the following lines to also try Trainer code (from DeepLab project train_net.py):
cfg.SOLVER.LR_SCHEDULER_NAME = 'WarmupPolyLR'
trainer = Trainer(cfg)

Full logs or other relevant observations:

[11/20 16:16:06 d2.engine.defaults]: Model:
SemanticSegmentor(
  (backbone): ResNet(
    (stem): BasicStem(
      (conv1): Conv2d(
        3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False
        (norm): SyncBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (res2): Sequential(
      (0): BasicBlock(
        (conv1): Conv2d(
          64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
          (norm): SyncBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
        (conv2): Conv2d(
          64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
          (norm): SyncBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
      )
      (1): BasicBlock(
        (conv1): Conv2d(
          64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
          (norm): SyncBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
        (conv2): Conv2d(
          64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
          (norm): SyncBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
      )
    )
    (res3): Sequential(
      (0): BasicBlock(
        (shortcut): Conv2d(
          64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False
          (norm): SyncBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
        (conv1): Conv2d(
          64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False
          (norm): SyncBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
        (conv2): Conv2d(
          128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
          (norm): SyncBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
      )
      (1): BasicBlock(
        (conv1): Conv2d(
          128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
          (norm): SyncBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
        (conv2): Conv2d(
          128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
          (norm): SyncBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
      )
    )
    (res4): Sequential(
      (0): BasicBlock(
        (shortcut): Conv2d(
          128, 256, kernel_size=(1, 1), stride=(2, 2), bias=False
          (norm): SyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
        (conv1): Conv2d(
          128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False
          (norm): SyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
        (conv2): Conv2d(
          256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
          (norm): SyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
      )
      (1): BasicBlock(
        (conv1): Conv2d(
          256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
          (norm): SyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
        (conv2): Conv2d(
          256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
          (norm): SyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
      )
    )
    (res5): Sequential(
      (0): BasicBlock(
        (shortcut): Conv2d(
          256, 512, kernel_size=(1, 1), stride=(2, 2), bias=False
          (norm): SyncBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
        (conv1): Conv2d(
          256, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False
          (norm): SyncBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
        (conv2): Conv2d(
          512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
          (norm): SyncBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
      )
      (1): BasicBlock(
        (conv1): Conv2d(
          512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
          (norm): SyncBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
        (conv2): Conv2d(
          512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
          (norm): SyncBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
      )
    )
  )
  (sem_seg_head): DeepLabV3PlusHead(
    (decoder): ModuleDict(
      (res2): ModuleDict(
        (project_conv): Conv2d(
          64, 48, kernel_size=(1, 1), stride=(1, 1), bias=False
          (norm): SyncBatchNorm(48, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
        (fuse_conv): Sequential(
          (0): Conv2d(
            304, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
            (norm): SyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          )
          (1): Conv2d(
            256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
            (norm): SyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          )
        )
      )
      (res5): ModuleDict(
        (project_conv): ASPP(
          (convs): ModuleList(
            (0): Conv2d(
              512, 256, kernel_size=(1, 1), stride=(1, 1), bias=False
              (norm): SyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            )
            (1): Conv2d(
              512, 256, kernel_size=(3, 3), stride=(1, 1), padding=(6, 6), dilation=(6, 6), bias=False
              (norm): SyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            )
            (2): Conv2d(
              512, 256, kernel_size=(3, 3), stride=(1, 1), padding=(12, 12), dilation=(12, 12), bias=False
              (norm): SyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            )
            (3): Conv2d(
              512, 256, kernel_size=(3, 3), stride=(1, 1), padding=(18, 18), dilation=(18, 18), bias=False
              (norm): SyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            )
            (4): Sequential(
              (0): AvgPool2d(kernel_size=(16, 32), stride=1, padding=0)
              (1): Conv2d(512, 256, kernel_size=(1, 1), stride=(1, 1))
            )
          )
          (project): Conv2d(
            1280, 256, kernel_size=(1, 1), stride=(1, 1), bias=False
            (norm): SyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          )
        )
        (fuse_conv): None
      )
    )
    (predictor): Conv2d(256, 19, kernel_size=(1, 1), stride=(1, 1))
    (loss): DeepLabCE(
      (criterion): CrossEntropyLoss()
    )
  )
)
[11/20 16:16:07 d2.data.dataset_mapper]: [DatasetMapper] Augmentations used in training: [RandomCrop(crop_type='absolute', crop_size=[512, 1024]), ResizeShortestEdge(short_edge_length=(512, 768, 1024, 1280, 1536, 1792, 2048), max_size=4096, sample_style='choice'), RandomFlip()]
[11/20 16:16:07 d2.data.build]: Using training sampler TrainingSampler
[11/20 16:16:07 d2.data.common]: Serializing the dataset using: <class 'detectron2.data.common.NumpySerializedList'>
[11/20 16:16:07 d2.data.common]: Serializing 37 elements to byte tensors and concatenating them all ...
[11/20 16:16:07 d2.data.common]: Serialized dataset takes 0.01 MiB
[11/20 16:16:12 d2.checkpoint.c2_model_loading]: Following weights matched with submodule backbone:
| Names in Model    | Names in Checkpoint                                                               | Shapes                                    |
|:------------------|:----------------------------------------------------------------------------------|:------------------------------------------|
| res2.0.conv1.*    | res2.0.conv1.{norm.bias,norm.running_mean,norm.running_var,norm.weight,weight}    | (64,) (64,) (64,) (64,) (64,64,3,3)       |
| res2.0.conv2.*    | res2.0.conv2.{norm.bias,norm.running_mean,norm.running_var,norm.weight,weight}    | (64,) (64,) (64,) (64,) (64,64,3,3)       |
| res2.1.conv1.*    | res2.1.conv1.{norm.bias,norm.running_mean,norm.running_var,norm.weight,weight}    | (64,) (64,) (64,) (64,) (64,64,3,3)       |
| res2.1.conv2.*    | res2.1.conv2.{norm.bias,norm.running_mean,norm.running_var,norm.weight,weight}    | (64,) (64,) (64,) (64,) (64,64,3,3)       |
| res3.0.conv1.*    | res3.0.conv1.{norm.bias,norm.running_mean,norm.running_var,norm.weight,weight}    | (128,) (128,) (128,) (128,) (128,64,3,3)  |
| res3.0.conv2.*    | res3.0.conv2.{norm.bias,norm.running_mean,norm.running_var,norm.weight,weight}    | (128,) (128,) (128,) (128,) (128,128,3,3) |
| res3.0.shortcut.* | res3.0.shortcut.{norm.bias,norm.running_mean,norm.running_var,norm.weight,weight} | (128,) (128,) (128,) (128,) (128,64,1,1)  |
| res3.1.conv1.*    | res3.1.conv1.{norm.bias,norm.running_mean,norm.running_var,norm.weight,weight}    | (128,) (128,) (128,) (128,) (128,128,3,3) |
| res3.1.conv2.*    | res3.1.conv2.{norm.bias,norm.running_mean,norm.running_var,norm.weight,weight}    | (128,) (128,) (128,) (128,) (128,128,3,3) |
| res4.0.conv1.*    | res4.0.conv1.{norm.bias,norm.running_mean,norm.running_var,norm.weight,weight}    | (256,) (256,) (256,) (256,) (256,128,3,3) |
| res4.0.conv2.*    | res4.0.conv2.{norm.bias,norm.running_mean,norm.running_var,norm.weight,weight}    | (256,) (256,) (256,) (256,) (256,256,3,3) |
| res4.0.shortcut.* | res4.0.shortcut.{norm.bias,norm.running_mean,norm.running_var,norm.weight,weight} | (256,) (256,) (256,) (256,) (256,128,1,1) |
| res4.1.conv1.*    | res4.1.conv1.{norm.bias,norm.running_mean,norm.running_var,norm.weight,weight}    | (256,) (256,) (256,) (256,) (256,256,3,3) |
| res4.1.conv2.*    | res4.1.conv2.{norm.bias,norm.running_mean,norm.running_var,norm.weight,weight}    | (256,) (256,) (256,) (256,) (256,256,3,3) |
| res5.0.conv1.*    | res5.0.conv1.{norm.bias,norm.running_mean,norm.running_var,norm.weight,weight}    | (512,) (512,) (512,) (512,) (512,256,3,3) |
| res5.0.conv2.*    | res5.0.conv2.{norm.bias,norm.running_mean,norm.running_var,norm.weight,weight}    | (512,) (512,) (512,) (512,) (512,512,3,3) |
| res5.0.shortcut.* | res5.0.shortcut.{norm.bias,norm.running_mean,norm.running_var,norm.weight,weight} | (512,) (512,) (512,) (512,) (512,256,1,1) |
| res5.1.conv1.*    | res5.1.conv1.{norm.bias,norm.running_mean,norm.running_var,norm.weight,weight}    | (512,) (512,) (512,) (512,) (512,512,3,3) |
| res5.1.conv2.*    | res5.1.conv2.{norm.bias,norm.running_mean,norm.running_var,norm.weight,weight}    | (512,) (512,) (512,) (512,) (512,512,3,3) |
| stem.conv1.*      | stem.conv1.{norm.bias,norm.running_mean,norm.running_var,norm.weight,weight}      | (64,) (64,) (64,) (64,) (64,3,7,7)        |
[11/20 16:16:14 d2.engine.train_loop]: Starting training from iteration 0
ERROR [11/20 16:16:22 d2.engine.train_loop]: Exception during training:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/detectron2/engine/train_loop.py", line 149, in train
    self.run_step()
  File "/opt/conda/lib/python3.7/site-packages/detectron2/engine/defaults.py", line 494, in run_step
    self._trainer.run_step()
  File "/opt/conda/lib/python3.7/site-packages/detectron2/engine/train_loop.py", line 274, in run_step
    loss_dict = self.model(data)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/detectron2/modeling/meta_arch/semantic_seg.py", line 108, in forward
    features = self.backbone(images.tensor)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/detectron2/modeling/backbone/resnet.py", line 445, in forward
    x = self.stem(x)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/detectron2/modeling/backbone/resnet.py", line 356, in forward
    x = self.conv1(x)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/detectron2/layers/wrappers.py", line 117, in forward
    x = self.norm(x)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/batchnorm.py", line 731, in forward
    world_size = torch.distributed.get_world_size(process_group)
  File "/opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 867, in get_world_size
    return _get_group_size(group)
  File "/opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 325, in _get_group_size
    default_pg = _get_default_group()
  File "/opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 430, in _get_default_group
    "Default process group has not been initialized, "
RuntimeError: Default process group has not been initialized, please make sure to call init_process_group.
[11/20 16:16:22 d2.engine.hooks]: Total training time: 0:00:08 (0:00:00 on hooks)
[11/20 16:16:22 d2.utils.events]:  iter: 0    lr: N/A  max_mem: 7348M

Environment:

Kaggle platform with both accelerators: GPU T4 x2 and GPU P100:

----------------------  -------------------------------------------------------------------------------
sys.platform            linux
Python                  3.7.12 | packaged by conda-forge | (default, Oct 26 2021, 06:08:53) [GCC 9.4.0]
numpy                   1.21.6
detectron2              0.6 @/opt/conda/lib/python3.7/site-packages/detectron2
Compiler                GCC 9.4
CUDA compiler           CUDA 11.0
detectron2 arch flags   7.5
DETECTRON2_ENV_MODULE   <not set>
PyTorch                 1.11.0 @/opt/conda/lib/python3.7/site-packages/torch
PyTorch debug build     False
GPU available           Yes
GPU 0,1                 Tesla T4 (arch=7.5)
Driver version          470.82.01
CUDA_HOME               /usr/local/cuda
Pillow                  9.1.1
torchvision             0.12.0 @/opt/conda/lib/python3.7/site-packages/torchvision
torchvision arch flags  3.7, 6.0, 7.0, 7.5
fvcore                  0.1.5.post20220512
iopath                  0.1.9
cv2                     4.5.4
----------------------  -------------------------------------------------------------------------------
PyTorch built with:
  - GCC 9.4
  - C++ Version: 201402
  - Intel(R) oneAPI Math Kernel Library Version 2022.1-Product Build 20220311 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v2.5.2 (Git Hash a9302535553c73243c632ad3c4c80beec3d19a1e)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: AVX512
  - CUDA Runtime 11.0
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_70,code=compute_70;-gencode;arch=compute_75,code=compute_75
  - CuDNN 8.0.5
  - Magma 2.5.2
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.0, CUDNN_VERSION=8.0.5, CXX_COMPILER=/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.11.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=OFF, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF,

Testing NCCL connectivity ... this should not hang.
NCCL succeeded.

Google Colab:

----------------------  ----------------------------------------------------------------
sys.platform            linux
Python                  3.7.15 (default, Oct 12 2022, 19:14:55) [GCC 7.5.0]
numpy                   1.21.6
detectron2              0.6 @/usr/local/lib/python3.7/dist-packages/detectron2
Compiler                GCC 7.5
CUDA compiler           CUDA 11.2
detectron2 arch flags   7.5
DETECTRON2_ENV_MODULE   <not set>
PyTorch                 1.12.1+cu113 @/usr/local/lib/python3.7/dist-packages/torch
PyTorch debug build     False
GPU available           Yes
GPU 0                   Tesla T4 (arch=7.5)
Driver version          460.32.03
CUDA_HOME               /usr/local/cuda
Pillow                  7.1.2
torchvision             0.13.1+cu113 @/usr/local/lib/python3.7/dist-packages/torchvision
torchvision arch flags  3.5, 5.0, 6.0, 7.0, 7.5, 8.0, 8.6
fvcore                  0.1.5.post20220512
iopath                  0.1.9
cv2                     4.6.0
----------------------  ----------------------------------------------------------------
PyTorch built with:
  - GCC 9.3
  - C++ Version: 201402
  - Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v2.6.0 (Git Hash 52b5f107dd9cf10910aaa19cb47f3abf9b349815)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - CUDA Runtime 11.3
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86
  - CuDNN 8.3.2  (built against CUDA 11.5)
  - Magma 2.5.2
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.3, CUDNN_VERSION=8.3.2, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -fabi-version=11 -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.12.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=OFF, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF,

Output:
It was due to very SyncBatchNorm layers that I knew can bother on single GPU as a drawback in pytorch, but seemingly I was not able to find in the log!
Stated here as a workaround:

print the config
find all keys that has a value of "SyncBN" or similar
Edit the config file or code to set these values to "BN" instead


Instruction: Pytorch tensor broadcasting along axis

Input:
a=torch.rand(20)
b=torch.rand(20, 20)
a+b # Works!

a=torch.rand(32, 20)
b=torch.rand(32, 20, 20)
a+b # Doesn't work!

Does anyone know how broadcasting in the first example could be generalized to the second example along axis 0 with no for loops?
I tried normal addition but broadcasting in Pytorch doesn't seem to work this way!

Output:
The dimensions in the second case are incompatible. You need to insert a unitary dimension into a to achieve the same results as the first case: a.unsqueeze(1) + b.

PyTorch follows the same broadcasting rules as NumPy. See https://numpy.org/doc/stable/user/basics.broadcasting.html
See specifically the first paragraph of the General Broadcasting Rules section.

When operating on two arrays, NumPy compares their shapes element-wise. It starts with the trailing (i.e. rightmost) dimensions and works its way left. Two dimensions are compatible when

they are equal, or

one of them is 1

Furthermore

Arrays do not need to have the same number of dimensions. For example, if you have a 256x256x3 array of RGB values, and you want to scale each color in the image by a different value, you can multiply the image by a one-dimensional array with 3 values.

This is effectively saying that if we line up the shapes starting from the right, and then insert ones in any blank spots all the dimensions should be compatible.
Considering the first case. If we line up the tensor shapes starting from the right we have
a:       20
b:  20 x 20

and insert one into the missing spot
a:  1 x 20
b: 20 x 20

we see that the shapes are compatible because the first dimension has a 1 and the second dimension has both values equal. The output shape of the broadcasted operation is 20 x 20, taking the first 20 from the first dimension of b.
Considering the second case, if we try to do the same
a:      32 x 20
b: 32 x 20 x 20

after inserting one into the missing spot we have
a:  1 x 32 x 20
b: 32 x 20 x 20

!!! These shapes are incompatible since the second dimension of a is 32 and the second dimension of b is 20 (since 32 != 20 and neither is equal to 1).

For the second example, one way you could make these shapes compatible would be to reshape a so that it has shape 32 x 1 x 20. I.e. insert an explicit unitary dimension in the middle. This could be done with any of the three methods.
a.reshape(32, 1, 20)+b

or equivalently
a.unsqueeze(1)+b

or equivalently
a[:, None, :]+b


Instruction: Slicing a tensor with tensor

Input:
I want to make a tensor of moving window. I'm using a list comprehension, but, this is sequential making it extremely slow.
weight_list = [w[:, :, :, i : i + self.l_c] for i in range(n)]

I want to find
weight = torch.for_example.index_slice(w, start_indices=n, slice_length=self.l_c, dim=-1)

I've seen methods of "indexing". But not slicing. Is there a method for this?

Output: It seems like you are looking for unfolding of w with kernel size (1, self.l_c).


Instruction: How to save output after two layers of neural network in Pytorch

Input:
I wrote a convolutional autoencoder that was supposed to work on the ORL dataset (400 images in dataset, size 32*32) in csv. format. What I want is to observe how the data changes through the autoencoder. That's why I wrote a test1 function in the class that goes through only the first two layers.
class ConvAutoencoder(nn.Module):
    def __init__(self):
        super(ConvAutoencoder, self).__init__()
        ## encoder layers ##
        self.conv1 = nn.Conv2d(1, 3, 3)
        self.conv2 = nn.Conv2d(3 ,1, 3)
        self.conv3 = nn.Conv2d(1, 3, 3)
        self.conv4 = nn.Conv2d(3, 1, 3)

        ## decoder layers ##
        self.t_conv1 = nn.ConvTranspose2d(1, 3, 3)
        self.t_conv2 = nn.ConvTranspose2d(3, 1, 3)
        self.t_conv3 = nn.ConvTranspose2d(1, 3, 3)
        self.t_conv4 = nn.ConvTranspose2d(3, 1, 3)

    def forward(self, x):

        x = F.relu(self.conv1(x))
        x = F.relu(self.conv2(x))
        x = F.relu(self.conv3(x))
        x = F.relu(self.conv4(x))
        ## decode ##

        x = F.relu(self.t_conv1(x))
        x = F.relu(self.t_conv2(x))
        x = F.relu(self.t_conv3(x))
        x = (self.t_conv4(x))
        return x

    def test1(self, x):
      x = F.relu(self.conv1(x))
      x = F.relu(self.conv2(x))
      return x

But the problem arises when I really want to check what is in those two layers.
mn_dataset_loader = torch.utils.data.DataLoader(dataset=custom_mnist_from_csv,
                                                    batch_size=200,
                                                    shuffle=False)
for epoch in range(200):
  running_loss = 0
  br = 0
  for data in  mn_dataset_loader:
            inputs = data[0].to(device, non_blocking=True)
            optimizer.zero_grad()
            outputs = model(inputs).to(device)
            i1 = model.test1(inputs).to(device)
            i1 = torch.squeeze(i1, 0)
            i1 = i1.flatten().to(device)
            i1=i1.unsqueeze(0)
            print(i1.shape)
            loss = lossFn(data.to(device), outputs)
            loss.backward()
            optimizer.step()

            running_loss += loss.item()
  print('[Epoch %d] loss: %.3f' %
                      (epoch + 1, running_loss/len(mn_dataset_loader)))

  print('Done Training')

My question is actually why is i1 what is the output after only 2 layers of the form torch.Size([1, 784])?
Why not torch.Size([400, 784]) because there are actually so many images?
So how to actually see what is actually the output of the first two layers?

Output:
You specify a batch size of 200 but then take only the first element (inputs = data[0])
If you want to run it on all images change the batch size to 400 and don't take only the first element


Instruction: How can I get specific columns form txt file and save them to new file using python

Input:
I have this txt file sentences.txt that contains texts below
a01-000u-s00-00 0 ok 154 19 408 746 1661 89 A|MOVE|to|stop|Mr.|Gaitskell|from

a01-000u-s00-01 0 ok 156 19 395 932 1850 105 nominating|any|more|Labour|life|Peers
which contains 10 columns
I want to use the panda's data frame to extract only the file name (at column 0) and corresponding text (column 10) without the (|) character
I wrote this code
def load() -> pd.DataFrame:

 df = pd.read_csv('sentences.txt',sep=' ', header=None)
 data = []
 with open('sentences.txt') as infile:
    for line in infile:
        file_name, _, _, _, _, _, _, _, _, text = line.strip().split(' ')
        data.append((file_name, cl_txt(text)))

 df = pd.DataFrame(data, columns=['file_name', 'text'])
 df.rename(columns={0: 'file_name', 9: 'text'}, inplace=True)
 df['file_name'] = df['file_name'].apply(lambda x: x + '.jpg')
 df = df[['file_name', 'text']]
 return df

def cl_txt(input_text: str) -> str:
 text = input_text.replace('+', '-')
 text = text.replace('|', ' ')
 return text

load()

the error I got
ParserError: Error tokenizing data. C error: Expected 10 fields in line 4, saw 11
where my expected process.txt file results should look like below without \n
a01-000u-s00-00  A MOVE to stop Mr. Gaitskell from
a01-000u-s00-01  nominating any more Labour life Peers

Output:
IIUC, you just need pandas.read_csv to read your .txt and then select the two columns :
Try this :
import pandas as pd

df= (
        pd.read_csv("test.txt", header=None, sep=r"(\d+)\s(?=\D)", engine="python",
                    usecols=[0,4], names=["filename", "text"])
            .assign(filename= lambda x: x["filename"].str.strip().add(".jpg"),
                    text= lambda x: x["text"].str.replace(r'[\|"]', " ", regex=True)
                                             .str.replace(r"\s+", " ", regex=True))
    )

# Output :
print(df)

              filename                                         text
0  a01-000u-s00-00.jpg            A MOVE to stop Mr. Gaitskell from
1  a01-000u-s00-01.jpg        nominating any more Labour life Peers
2   a01-003-s00-01.jpg  large majority of Labour M Ps are likely to

# .txt used:


Instruction: torch.nn.functional.binary_cross_entropy and torch.nn.BCEloss() difference

Input:
I am trying to train a GAN model on anime face Dataset to generate anime faces. Here's my code-
from torch.utils.data import DataLoader
from torchvision.datasets import ImageFolder
import torchvision.transforms as T
import os
import torch
import torch.nn as nn
from torchvision.utils import make_grid
import matplotlib.pyplot as plt
%matplotlib inline

def denorm(img_tensors):
    return img_tensors * stats[1][0] + stats[0][0]

def show_images(images, nmax=64):
    fig, ax = plt.subplots(figsize=(8, 8))
    ax.set_xticks([]); ax.set_yticks([])
    ax.imshow(make_grid(denorm(images.detach()[:nmax]), nrow=8).permute(1, 2, 0))

def show_batch(dl, nmax=64):
    for images, _ in dl:
        show_images(images, nmax)
        break

def get_default_device():
    """Pick GPU if available, else CPU"""
    if torch.cuda.is_available():
        return torch.device('cuda')
    else:
        return torch.device('cpu')

def to_device(data, device):
    """Move tensor(s) to chosen device"""
    if isinstance(data, (list,tuple)):
        return [to_device(x, device) for x in data]
    return data.to(device, non_blocking=True)

class DeviceDataLoader():
    """Wrap a dataloader to move data to a device"""
    def __init__(self, dl, device):
        self.dl = dl
        self.device = device

    def __iter__(self):
        """Yield a batch of data after moving it to device"""
        for b in self.dl:
            yield to_device(b, self.device)

    def __len__(self):
        """Number of batches"""
        return len(self.dl)

device = get_default_device()
device

train_dl = DeviceDataLoader(train_dl, device)

discriminator = nn.Sequential(
    # in: 3 x 64 x 64

    nn.Conv2d(3, 64, kernel_size=4, stride=2, padding=1, bias=False),
    nn.BatchNorm2d(64),
    nn.LeakyReLU(0.2, inplace=True),
    # out: 64 x 32 x 32

    nn.Conv2d(64, 128, kernel_size=4, stride=2, padding=1, bias=False),
    nn.BatchNorm2d(128),
    nn.LeakyReLU(0.2, inplace=True),
    # out: 128 x 16 x 16

    nn.Conv2d(128, 256, kernel_size=4, stride=2, padding=1, bias=False),
    nn.BatchNorm2d(256),
    nn.LeakyReLU(0.2, inplace=True),
    # out: 256 x 8 x 8

    nn.Conv2d(256, 512, kernel_size=4, stride=2, padding=1, bias=False),
    nn.BatchNorm2d(512),
    nn.LeakyReLU(0.2, inplace=True),
    # out: 512 x 4 x 4

    nn.Conv2d(512, 1, kernel_size=4, stride=1, padding=0, bias=False),
    # out: 1 x 1 x 1

    nn.Flatten(),
    nn.Sigmoid())

discriminator = to_device(discriminator, device)

latent_size = 128

generator = nn.Sequential(
    # in: latent_size x 1 x 1

    nn.ConvTranspose2d(latent_size, 512, kernel_size=4, stride=1, padding=0, bias=False),
    nn.BatchNorm2d(512),
    nn.ReLU(True),
    # out: 512 x 4 x 4

    nn.ConvTranspose2d(512, 256, kernel_size=4, stride=2, padding=1, bias=False),
    nn.BatchNorm2d(256),
    nn.ReLU(True),
    # out: 256 x 8 x 8

    nn.ConvTranspose2d(256, 128, kernel_size=4, stride=2, padding=1, bias=False),
    nn.BatchNorm2d(128),
    nn.ReLU(True),
    # out: 128 x 16 x 16

    nn.ConvTranspose2d(128, 64, kernel_size=4, stride=2, padding=1, bias=False),
    nn.BatchNorm2d(64),
    nn.ReLU(True),
    # out: 64 x 32 x 32

    nn.ConvTranspose2d(64, 3, kernel_size=4, stride=2, padding=1, bias=False),
    nn.Tanh()
    # out: 3 x 64 x 64
)

xb = torch.randn(batch_size, latent_size, 1, 1) # random latent tensors
fake_images = generator(xb)
print(fake_images.shape)
show_images(fake_images)

generator = to_device(generator, device)

def train_discriminator(real_images, opt_d):
    # Clear discriminator gradients
    opt_d.zero_grad()

    # Pass real images through discriminator
    real_preds = discriminator(real_images)
    real_targets = torch.ones(real_images.size(0), 1, device=device)
    real_loss = F.binary_cross_entropy(real_preds, real_targets)     # here nn.BCELoss() not working
    real_score = torch.mean(real_preds).item()

    # Generate fake images
    latent = torch.randn(batch_size, latent_size, 1, 1, device=device)
    fake_images = generator(latent)

    # Pass fake images through discriminator
    fake_targets = torch.zeros(fake_images.size(0), 1, device=device)
    fake_preds = discriminator(fake_images)
    fake_loss = F.binary_cross_entropy(fake_preds, fake_targets)    # here nn.BCELoss() not working
    fake_score = torch.mean(fake_preds).item()

    # Update discriminator weights
    loss = real_loss + fake_loss
    loss.backward()
    opt_d.step()
    return loss.item(), real_score, fake_score

def train_generator(opt_g):
    # Clear generator gradients
    opt_g.zero_grad()

    # Generate fake images
    latent = torch.randn(batch_size, latent_size, 1, 1, device=device)
    fake_images = generator(latent)

    # Try to fool the discriminator
    preds = discriminator(fake_images)
    targets = torch.ones(batch_size, 1, device=device)
    loss = F.binary_cross_entropy(preds, targets)     # here nn.BCELoss() not working

    # Update generator weights
    loss.backward()
    opt_g.step()

    return loss.item()

from torchvision.utils import save_image

sample_dir = 'generated'
os.makedirs(sample_dir, exist_ok=True)

def save_samples(index, latent_tensors, show=True):
    fake_images = generator(latent_tensors)
    fake_fname = 'generated-images-{0:0=4d}.png'.format(index)
    save_image(denorm(fake_images), os.path.join(sample_dir, fake_fname), nrow=8)
    print('Saving', fake_fname)
    if show:
        fig, ax = plt.subplots(figsize=(8, 8))
        ax.set_xticks([]); ax.set_yticks([])
        ax.imshow(make_grid(fake_images.cpu().detach(), nrow=8).permute(1, 2, 0))

fixed_latent = torch.randn(64, latent_size, 1, 1, device=device)

save_samples(0, fixed_latent)

from tqdm.notebook import tqdm
import torch.nn.functional as F

def fit(epochs, lr, start_idx=1):
    torch.cuda.empty_cache()

    # Losses & scores
    losses_g = []
    losses_d = []
    real_scores = []
    fake_scores = []

    # Create optimizers
    opt_d = torch.optim.Adam(discriminator.parameters(), lr=lr, betas=(0.5, 0.999))
    opt_g = torch.optim.Adam(generator.parameters(), lr=lr, betas=(0.5, 0.999))

    for epoch in range(epochs):
        for real_images, _ in tqdm(train_dl):
            # Train discriminator
            loss_d, real_score, fake_score = train_discriminator(real_images, opt_d)
            # Train generator
            loss_g = train_generator(opt_g)

        # Record losses & scores
        losses_g.append(loss_g)
        losses_d.append(loss_d)
        real_scores.append(real_score)
        fake_scores.append(fake_score)

        # Log losses & scores (last batch)
        print("Epoch [{}/{}], loss_g: {:.4f}, loss_d: {:.4f}, real_score: {:.4f}, fake_score: {:.4f}".format(
            epoch+1, epochs, loss_g, loss_d, real_score, fake_score))

        # Save generated images
        save_samples(epoch+start_idx, fixed_latent, show=False)

    return losses_g, losses_d, real_scores, fake_scores

lr = 0.0002
epochs = 95

history = fit(epochs, lr)

The above code is working fine but before I was using nn.BCELoss from torch instead of binary_cross_entropy from torch.nn.functional in 'train_generator()' and 'train_discriminator()' methods above and I was getting the following error,
RuntimeError: Boolean value of Tensor with more than one value is ambiguous

I wonder if they both don't perform the same operation. Can you help me to understand the problem?

Output:
nn.BCELoss is a class. Unlike nn.functional.binary_cross_entropy, you have to instantiate it first before using it to calculate the loss. In you case,
F.binary_cross_entropy(preds, targets)

is equivalent to
nn.BCELoss()(preds, targets)