Instruction: `RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn` for linear regression with gradient descent using torch

Input:
I am trying to implement a simple gradient descent for linear regression with pytorch as shown in this example in the docs:
import torch
from torch.autograd import Variable

learning_rate = 0.01
y = 5
x = torch.tensor([3., 0., 1.])
w = torch.tensor([2., 3., 9.], requires_grad=True)
b = torch.tensor(1., requires_grad=True)

for z in range(100):
    y_pred = b + torch.sum(w * x)
    loss = (y_pred - y).pow(2)
    loss = Variable(loss, requires_grad = True)
    # loss.requires_grad = True
    loss.backward()

    with torch.no_grad():
        w = w - learning_rate * w.grad
        b = b - learning_rate * b.grad

        w.grad = None
        b.grad = None

When I run the code I get the error RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
I have read here and here that it could be solved
using

loss = Variable(loss, requires_grad = True) results in TypeError: unsupported operand type(s) for *: 'float' and 'NoneType'

or loss.requires_grad = True results in RuntimeError: you can only change requires_grad flags of leaf variables.

How can I fix this?

Output:
This error was actually caused by mixing calculation functions from torch with python built ins (same should go with numpy or other libraries which are not torch). It actually means that the autograd implementation from torch is breaking because they don't work with other functions.
a good explanation can be read here

This is more like a not fully appropiate hack:
calling .retain_grad() before backward solved the issue for me:
learning_rate = 0.01
y = 5
x = torch.tensor([3., 0., 1.])
w = torch.tensor([2., 3., 9.], requires_grad=True)
b = torch.tensor(1., requires_grad=True)

for z in range(100):
    y_pred = b + torch.sum(w * x)
    loss = (y_pred - y).pow(2)

    w.retain_grad()
    b.retain_grad()
    loss.backward()

    w = w - learning_rate * w.grad
    b = b - learning_rate * b.grad


Instruction: How to produce 4-dimensional input for 4-dimensional weight?

Input:
I am very new to Deep learning. I am working on the CIFAR10 dataset and created a CNN model which is as below.
class Net2(nn.Module):
  def __init__(self):
    super(Net2, self).__init__()
    self.conv1 = nn.Conv2d(3, 32, 5, 1)
    self.fc1 = nn.Linear(32 * 5 * 5, 512)
    self.fc2 = nn.Linear(512,10)

  def forward(self, x):
    x = x.view(x.size(0), -1)
    x = F.max_pool2d(F.relu(self.conv1(x)),(2,2))
    x = F.relu(self.fc1(x))
    x = self.fc2(x)
    return x

net2 = Net2().to(device)

My assignment requirements are to create a model with:
Convolutional layer with 32 filters, kernel size of 5x5 and stride of 1.
Max Pooling layer with kernel size of 2x2 and default stride.
ReLU Activation Layers.
Linear layer with output of 512.
ReLU Activation Layers.
A linear layer with output of 10.
Which I guess I wrote. But I am assuming that I am going to the wrong path. Please help me to write the correct model and also the reason behind those arguments in Conv2d and Linear layers.
The error which I am getting from my code is as below:
RuntimeError: Expected 4-dimensional input for 4-dimensional weight [32, 3, 5, 5], but got 2-dimensional input of size [1024, 3072] instead

Please help me!

Output:
There are two problems with the code:

Flattening of input
x = x.view(x.size(0), -1)

The convolutional layer expects a four dimensional input of dimensions (N, C, H, W), where N is the batch size, C = 3 is the number of channels, and (H, W) is the dimension of the image. By using the above statement, you are flattening your (1024, 3, 32, 32) input to (1024, 3072).

Number of input features in the first linear layer
self.fc1 = nn.Linear(32 * 5 * 5, 512)

The output dimensions of the convolutional layer for a (1024, 3, 32, 32) input will be (1024, 32, 28, 28), and after applying the 2 x 2 maxpooling, it is (1024, 32, 14, 14). So the number of input features for the linear layer should be 32 x 14 x 14 = 6272.


Instruction: How can I only train the classifier and freeze rest of the parameters in Pytorch?

Input:
I have taken the pretrained model of MoviNet, I have changed the last layer.
This is last parameters of pretrained model that I have taken;
classifier.0.conv_1.conv2d.weight  :  torch.Size([2048, 640, 1, 1])
classifier.0.conv_1.conv2d.bias  :  torch.Size([2048])
classifier.3.conv_1.conv2d.weight  :  torch.Size([600, 2048, 1, 1])
classifier.3.conv_1.conv2d.bias  :  torch.Size([600])

The following are the parameters that I have changed at the last layer;
clfr.0.multi_head.0.head2.0.conv_1.conv2d.weight  :  torch.Size([2048, 640, 1, 1])
clfr.0.multi_head.0.head2.0.conv_1.conv2d.bias  :  torch.Size([2048])
clfr.0.multi_head.0.head1.weight  :  torch.Size([600, 2048, 1, 1])
clfr.0.multi_head.0.head1.bias  :  torch.Size([600])

I want to train only classifier (clfr) based on previous layer weights, and freeze all previous laers in pytorch, can anyone one tell me how canI do this?

Output:
When creating your optimizer, only pass the parameters that you want to update during training. In your example, it could look something like:
optimizer = torch.optim.Adam(clfr.parameters())


Instruction: how to concate two tensors with different dimensions in pytorch

Input:
I have two tensors in pytorch with these shapes:

torch.Size([64, 100]) and torch.Size([64, 100, 256])

I want to concate them by torch.cat but they should be in the same shape and size. So I get this error:

RuntimeError: Tensors must have same number of dimensions: got 2 and 3

what should I do to fix this problem? how can I convert 2d PyTorch tensor into 3d tensor OR how can I convert 3d PyTorch tensor to 2d tensor without losing any data? or any other idea?

Output:
Depending on what you are looking to do with those two tensors, you could consider concatenating on the last axis such that the resulting tensor is shaped (64, 100, 257). This requires you first unsqueeze a singleton dimensions on the first tensor:
>>> x, y = torch.rand(64, 100), torch.rand(64, 100, 256)
>>> z = torch.cat((x[..., None], y), -1)

>>> z.shape
torch.Size([64, 100, 257])


Instruction: Install PyTorch 1.3 via Anaconda

Input:
My current PyTorch version is 1.10. I want to downgrade it to 1.3 but conda suggests the CPU-only version. My GPU is a GTX 1080 TI.
My current setup is
cudatoolkit               10.2.89         hfd86e86_1
cudnn                     7.6.5           cuda10.2_0
pytorch                   1.10.0          py3.7_cuda10.2_cudnn7.6.5_0    pytorch
pytorch-mutex             1.0             cuda    pytorch

When I execute:
conda install pytorch==1.3.0 torchvision -c pytorchCollecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/lorenzp/.conda/envs/detection

  added / updated specs:
    - pytorch==1.3.0
    - torchvision

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    cpuonly-1.0                |                0           2 KB  pytorch
    pytorch-1.3.0              |      py3.7_cpu_0        36.1 MB  pytorch
    torchvision-0.4.1          |         py37_cpu        14.6 MB  pytorch
    ------------------------------------------------------------
                                           Total:        50.7 MB

The following NEW packages will be INSTALLED:

  cpuonly            pytorch/noarch::cpuonly-1.0-0

The following packages will be DOWNGRADED:

  pytorch                1.10.0-py3.7_cuda10.2_cudnn7.6.5_0 --> 1.3.0-py3.7_cpu_0
  torchvision                             0.11.1-py37_cu102 --> 0.4.1-py37_cpu

Proceed ([y]/n)? n

CondaSystemExit: Exiting.

It just suggests the CPU-only version. How to find the GPU version?

Output:
Well, I needed to find the right cudatoolkit. For that
conda search cudatoolkit

Then, I installed I can choose version somewhere between Cuda 9.2 and 10.0. I
took a look at https://pytorch.org/get-started/previous-versions and made a guess of the cuda version.
In my case:
conda install cudatoolkit==10.0.130

This automatically installed PyTorch==1.3.1.


Instruction: how do I optmize the weights of the input layer using backward for this simple neural network in pytorch when .grad is None

Input:
I defined the following simple neural network:
import torch
import torch.nn as nn

X = torch.tensor(([1, 2]), dtype=torch.float)
y = torch.tensor([1.])
learning_rate = 0.001

class Neural_Network(nn.Module):
    def __init__(self, ):
        super(Neural_Network, self).__init__()

        self.W1 = torch.nn.Parameter(torch.tensor(([1, 0], [2, 3]), dtype=torch.float, requires_grad=True))
        self.W2 = torch.nn.Parameter(torch.tensor(([2], [1]), dtype=torch.float, requires_grad=True))

    def forward(self, X):
        self.xW1 = torch.matmul(X, self.W1)
        self.h = torch.tensor([torch.tanh(self.xW1[0]), torch.tanh(self.xW1[1])])
        return torch.sigmoid(torch.matmul(self.h, self.W2))

net = Neural_Network()

for z in range(60):
    loss = (y - net(X))**2
    optim = torch.optim.SGD(net.parameters(), lr=learning_rate, momentum=0.9)
    loss = criterion(net(X), y)
    loss.backward()
    optim.step()

I can run it and print(net.W1) print(net.W2) prints
Parameter containing:
tensor([[1., 0.],
        [2., 3.]], requires_grad=True)
Parameter containing:
tensor([[2.0078],
        [1.0078]], requires_grad=True)

So my problem is that it seems like W1 is not being updated.
When I call print(net.W1.grad) I get None for every iteration which confuses me a lot.

I tried to define the function as one line like so:
loss = (y - torch.sigmoid(math.tanh(x[0] * W_1[0][0] + x[1] * W_1[1][0]) * W_2[0] + math.tanh(x[0] * W_1[0][1] + x[1] * W_1[1][1]) * W_2[1])) ** 2, but it did not help anything.

For sure I could hardcode the derivate and everything but it seems painful and I though .backward() can be used in this case.

How can I optmize W1 with using backward()?

Output:
I suspect that the following line:
self.h = torch.tensor([torch.tanh(self.xW1[0]), torch.tanh(self.xW1[1])])
is the culprit.
The new tensor self.h does not inherit the requires_grad attribute from self.xW1 and, by default, it is set to False.
You can call self.h = self.tanh(self.xW1) and the operation will be then applied point-wise to all the elements of self.xW1.
In addition, I suggest you to inspect your gradients by using PyTorch hooks.


Instruction: Error: Expected all tensors to be on the same device, While all are on same device

Input:
I don't know why I am getting this error, while my all tensors are at the same device "cuda".
Here is code
def train(self):
    self.model.train()
    self.optimizer.zero_grad()  # Clear gradients.
    print("X is",self.XT.get_device())
    print("edge_index_ is", self.edge_index_.get_device())
    print("R_train_mask is", self.R_train_mask.get_device())

    print("datas.y is", self.datas.y.get_device())
    print("edge_index_ is", self.edge_index_.get_device())
    out = self.model(self.XT, self.edge_index_)  # Perform a single forward pass.
    loss = self.criterion(out[self.R_train_mask],
                          self.datas.y[self.R_train_mask])

Here is error
X is 0
edge_index_ is 0
R_train_mask is 0
datas.y is 0
edge_index_ is 0
Traceback (most recent call last):
  File "/home/adnan/GNNPaperCodes/PyGCL/examples/GCAPaperAd.py", line 126, in <module>
    main()
  File "/home/adnan/GNNPaperCodes/PyGCL/examples/GCAPaperAd.py", line 91, in main
    topfeatures=tf.TopFeaturesFind()
  File "/home/adnan/GNNPaperCodes/PyGCL/examples/PatchFinder.py", line 64, in TopFeaturesFind
    loss = self.train()
  File "/home/adnan/GNNPaperCodes/PyGCL/examples/PatchFinder.py", line 154, in train
    out = self.model(self.XT, self.edge_index_)  # Perform a single forward pass.
  File "/home/adnan/.conda/envs/forGit/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/adnan/GNNPaperCodes/PyGCL/examples/PatchFinder.py", line 211, in forward
    x = self.conv1(x, edge_index_)
  File "/home/adnan/.conda/envs/forGit/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/adnan/.conda/envs/forGit/lib/python3.9/site-packages/torch_geometric/nn/conv/gcn_conv.py", line 181, in forward
    x = self.lin(x)
  File "/home/adnan/.conda/envs/forGit/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/adnan/.conda/envs/forGit/lib/python3.9/site-packages/torch_geometric/nn/dense/linear.py", line 103, in forward
    return F.linear(x, self.weight, self.bias)
  File "/home/adnan/.conda/envs/forGit/lib/python3.9/site-packages/torch/nn/functional.py", line 1848, in linear
    return torch._C._nn.linear(input, weight, bias)
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument mat2 in method wrapper_mm)

Obvioulsy, Everything is ok if i change device to 'cpu'

Output:
You forgot to transfer self.model to the device. Indeed, the model has weights that must be transferred to the correct device, since they interact with your input.
You can do it with self.model.to("cuda").


Instruction: How to achieve fast "vector clustering" sort?

Input:
I have n c-dimensional vectors that formed into a matrix A with the shape of (n, c), how can I perform a quick sort such that the vectors with low Euclidean distances are as close as possible, and the vectors with high distances are as far as possible?
For example, I have
A = [[0, 3], [0, 0], [0, 1]],

and the solution can be
A_sorted = [[0, 3], [0, 1], [0, 0]].

Explain: Because the original A has a total weighted distance sum of 3x1+1x1+2x2 = 7, and A_sorted has 2x1+1x1+3x2 = 8.
Mathematically, the goal is to maximize the total weighted distance sum.
For the 1-dimensional case, this can be achieved by some APIs like sort() in Numpy or PyTorch, and my main concern is if there exists a fast implementation when c ≥ 2 with a time complexity of O(nlog(n))?
After a long struggle, I failed. Could you please do me a favor?

Output:
I believe this is what you want:
def d(l):
    return math.sqrt(sum([x**2 for x in l]))

A = [[0, 3], [0, 0], [0, 1]]

dists = [d(el) for el in A]
sorted_vecs = sorted(zip(A, dists), key=lambda x: -x[1])
[x[0] for x in sorted_vecs]

Returns
[[0, 3], [0, 1], [0, 0]]

It's O(nlog(n))


Instruction: What's the difference between torch.mean and torch.nn.avg_pool?

Input:
Taking a tensor with shape [4,8,12] as an example, what's the difference between the two lines:
torch.mean(x, dim=2)
torch.nn.functional.avg_pool1d(x, kernel_size=12)

Output:
With the very example you provided the result is the same, but only because you specified dim=2 and kernel_size equal to the dimensionality of the third (index 2) dimension.
But in principle, you are applying two different functions, that sometimes just happen to collide with specific choices of hyperparameters.
torch.mean is effectively a dimensionality reduction function, meaning that when you average all values across one dimension, you effectively get rid of that dimension.
On the other hand, average 1-dimensional pooling is more powerful in this regard, as it gives you a lot more flexibility in choosing kernel size, padding and stride like you would normally do when using a convolutional layer.
You can see the first function as a specific case of 1-d pooling.


Instruction: Split torch dataset without shuffling

Input:
I'm using Pytorch to run Transformer model. when I want to split data (tokenized data) i'm using this code:
train_dataset, test_dataset = torch.utils.data.random_split(
                                                            tokenized_datasets,
                                                            [train_size, test_size])

torch.utils.data.random_split using shuffling method, but I don't want to shuffle. I want to split it sequentially.
Any advice? thanks

Output:
The random_split method has no parameter that can help you create a non-random sequential split.
The easiest way to achieve a sequential split is by directly passing the indices for the subset you want to create:
# Created using indices from 0 to train_size.
train_dataset = torch.utils.data.Subset(tokenized_datasets, range(train_size))

# Created using indices from train_size to train_size + test_size.
test_dataset = torch.utils.data.Subset(tokenized_datasets, range(train_size, train_size + test_size))

Refer: PyTorch docs.


Instruction: Match pytorch scatter output in tensorflow

Input:
How can I do the same operation in tensorflow?
tensor = np.random.RandomState(42).uniform(size=(2, 4, 2)).astype(np.float32)
tensor = torch.from_numpy(tensor)
index = tensor.max(-1, keepdim=True)[1]
output = torch.zeros_like(tensor).scatter_(-1, index, 1.0)

expected output:
tensor([[[0., 1.],
         [1., 0.],
         [1., 0.],
         [0., 1.]],

        [[0., 1.],
         [0., 1.],
         [1., 0.],
         [0., 1.]]])

Output:
As always, everything is a bit more complicated with Tensorflow:
import tensorflow as tf
import numpy as np

tensor = np.random.RandomState(42).uniform(size=(2, 4, 2)).astype(np.float32)
tensor = tf.constant(tensor)
_, indices = tf.math.top_k(tensor)
zeros = tf.zeros_like(tensor)

ij = tf.stack(tf.meshgrid(
    tf.range(zeros.shape[0], dtype=tf.int32),
    tf.range(zeros.shape[1], dtype=tf.int32),
                              indexing='ij'), axis=-1)
gathered_indices = tf.concat([ij, indices], axis=-1)
indices_shape = tf.shape(indices)
values = tf.ones((indices_shape[0], indices_shape[1]))
output = tf.tensor_scatter_nd_update(zeros, gathered_indices, values)

print(output)

tf.Tensor(
[[[0. 1.]
  [1. 0.]
  [1. 0.]
  [0. 1.]]

 [[0. 1.]
  [0. 1.]
  [1. 0.]
  [0. 1.]]], shape=(2, 4, 2), dtype=float32)


Instruction: BertModel and BertForMaskedLM weights count

Input:
I want understand BertForMaskedLM model, in huggingface github code, BertForMaskedLM is bert model with additional 2 linear layers with shape (input 768, output 768) and (input 768, output 30522). Count of all weights will be weights of BertModel + 768 * 768 + 768 * 30522, but when I check the numbers don't match.
from transformers import BertModel, BertForMaskedLM
import torch

bertmodel = BertModel.from_pretrained('bert-base-uncased')
bertForMaskedLM = BertForMaskedLM.from_pretrained('bert-base-uncased')

def count_parameters(model):
    return sum(p.numel() for p in model.parameters() if p.requires_grad)

count_parameters(bertmodel)
#output 109482240
count_parameters(bertForMaskedLM)
#output 109514298

109482240 + 768 * 768 + 768 * 30522 != 109514298
what am I doing wrong?

Output:
Using numel() along with model.parameters() is not a reliable method for counting the total number of parameters and may fail for recursive configuration of layers. This is exactly what is happening in your case. Instead, try following:
from torchinfo import summary

print(summary(bertmodel))

Output:

print(summary(bertForMaskedLM))

Output:

From the above outputs we can see that total number of trainable params for the two models are:
bertmodel: 109,482,240
bertForMaskedLM: 132,955,194
In order to understand the difference, lets have a look at the last module of both the models (rest of the base model is exactly the same):
bertmodel:
(pooler): BertPooler(
(dense): Linear(in_features=768, out_features=768, bias=True)
(activation): Tanh())

bertForMaskedLM:
(cls): BertOnlyMLMHead((predictions): BertLMPredictionHead(
(transform): BertPredictionHeadTransform(
  (dense): Linear(in_features=768, out_features=768, bias=True)
  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
)
(decoder): Linear(in_features=768, out_features=30522, bias=True)))

Only additions are the LayerNorm layer (2 * 768 params for layer gammas and betas) and the decoder layer (769 * 30522, using the y=A*X + B, where A is of size (nxm) and B of (nx1) with a total params of nx(m+1).
Params for bertForMaskedLM = 109482240 + 2 * 768 + 769 * 30522 = 132955194


Instruction: RuntimeError: expected scalar type Long but found Float (Pytorch)

Input:
I've tried many times to fix, also I've used the example codes from functional.py then I got my  same "loss" value. How can I fix this?
My libraries

    import matplotlib.pyplot as plt
    import torch
    import torch.nn as nn
    import numpy as np
    import matplotlib
    import pandas as pd
    from torch.autograd import Variable
    from torch.utils.data import DataLoader,TensorDataset
    from sklearn.model_selection import train_test_split
    import warnings
    import os
    import torchvision
    import torchvision.datasets as dsets
    import torchvision.transforms as transforms

Data set of Mnis

    train=pd.read_csv("train.csv",dtype=np.float32)

    targets_numpy = train.label.values
    features_numpy = train.loc[:,train.columns != "label"].values/255 # normalization

    features_train, features_test, targets_train, targets_test = train_test_split(features_numpy,
                                                                                targets_numpy,test_size = 0.2,
                                                                                random_state = 42)

    featuresTrain=torch.from_numpy(features_train)
    targetsTrain=torch.from_numpy(targets_train)

    featuresTest=torch.from_numpy(features_test)
    targetsTest=torch.from_numpy(targets_test)

    batch_size=100
    n_iterations=10000
    num_epochs=n_iterations/(len(features_train)/batch_size)
    num_epochs=int(num_epochs)

    train=torch.utils.data.TensorDataset(featuresTrain,targetsTrain)
    test=torch.utils.data.TensorDataset(featuresTest,targetsTest)

    print(type(train))

    train_loader=DataLoader(train,batch_size=batch_size,shuffle=False)
    test_loader=DataLoader(test,batch_size=batch_size,shuffle=False)
    print(type(train_loader))

    plt.imshow(features_numpy[226].reshape(28,28))
    plt.axis("off")
    plt.title(str(targets_numpy[226]))
    plt.show()

Here is my model

    class ANNModel(nn.Module):

      def __init__(self,input_dim,hidden_dim,output_dim):
        super(ANNModel,self).__init__()

        self.fc1=nn.Linear(input_dim,hidden_dim)

        self.relu1=nn.ReLU()

        self.fc2=nn.Linear(hidden_dim,hidden_dim)

        self.tanh2=nn.Tanh()

        self.fc4=nn.Linear(hidden_dim,output_dim)

      def forward (self,x): #forward ile elde edilen layer lar bağlanır

        out=self.fc1(x)

        out=self.relu1(out)

        out=self.fc2(out)

        out=self.tanh2(out)

        out=self.fc4(out)
        return out

    input_dim=28*28
    hidden_dim=150
    output_dim=10

    model=ANNModel(input_dim,hidden_dim,output_dim)

    error=nn.CrossEntropyLoss()

    learning_rate=0.02
    optimizer=torch.optim.SGD(model.parameters(),lr=learning_rate)

where the problem is

    count=0
    loss_list=[]
    iteration_list=[]
    accuracy_list = []
    for epoch in range(num_epochs):
      for i,(images,labels) in enumerate(train_loader):

        train=Variable(images.view(-1,28*28))
        labels=Variable(labels)
        #print(labels)
        #print(outputs)

        optimizer.zero_grad()

        #forward propagation
        outputs=model(train)

        #outputs=torch.randn(784,10,requires_grad=True)
        ##labels=torch.randn(784,10).softmax(dim=1)
        loss=error(outputs,labels)

        loss.backward()

        optimizer.step()

        count+=1

        if count %50 ==0:

          correct=0
          total=0

          for images,labels in test_loader:
            test=Variable(images.view(-1,28*28))

            outputs=model(test)

            predicted=torch.max(outputs.data,1)[1] #mantık???

            total+= len(labels)

            correct+=(predicted==labels).sum()

          accuracy=100  *correct/float(total)

          loss_list.append(loss.data)
          iteration_list.append(count)
          accuracy_list.append(accuracy)
          if  count %500 ==0 :

           print('Iteration: {}  Loss: {}  Accuracy: {} %'.format(count, loss.data, accuracy))

Which gives

    ---------------------------------------------------------------------------
    RuntimeError                              Traceback (most recent call last)
    <ipython-input-9-9e53988ad250> in <module>()
         26     #outputs=torch.randn(784,10,requires_grad=True)
         27     ##labels=torch.randn(784,10).softmax(dim=1)
    ---> 28     loss=error(outputs,labels)
         29
         30

    2 frames
    /usr/local/lib/python3.7/dist-packages/torch/nn/functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction, label_smoothing)
       2844     if size_average is not None or reduce is not None:
       2845         reduction = _Reduction.legacy_get_string(size_average, reduce)
    -> 2846     return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)
       2847
       2848

    RuntimeError: expected scalar type Long but found Float

Output:
it seems that the dtype of the tensor "labels" is FloatTensor. However, nn.CrossEntropyLoss expects a target of type LongTensor. This means that you should check the type of "labels". if its the case then you should use the following code to convert the dtype of "labels" from FloatTensor to LongTensor:
loss=error(outputs,labels.long())


Instruction: How to Run Pytorch Bert with AMD

Input:
github code: https://github.com/bellowman/Deep-Learning-Practice/blob/main/BioBert%20for%20Multi%20Label%20AMD.ipynb
Hello everyone,
I am a beginner with pytorch, tensorflow, and BERT. I have a machine at home with an AMD Ryzen 7 1800x and a Radeon RX 6600 video card.
I am trying to run a bioBERT model at home. I have trouble leveraging my model to use my AMD card. I posted my github notebook. I have troubles in cell 3 and 9.

First Question: In cell 3,I am trying to convert the bioBERT weight to PyTorch with transformmer-cli. I get the warning of "Could not load dynamic library 'cudart64_110.dll'". Does this affect performance later?
Second Question: In cell 9, My model load is really slow because it is using just the CPU. How can I get the model to run on my AMD GPU

Output:
Thank you to chrispresso
AMD ROCm seems to be the way to go, but it requires one to run under linux


Instruction: Issue with loading the Roberta-base model

Input:
I am trying to use the Roberta-base model using AutoTokenizer.from_pretrained('roberta-base') but I get the following error:
RuntimeError: Failed to import transformers.modeling_tf_utils because of the following error (look up to see its traceback):
No module named 'tensorflow.python.keras.engine.keras_tensor'

I have tried to install tensorflow but still the same error. Any idea what is going on?

Output:
You need to install with GPU, try this:
pip install --ignore-installed --upgrade tensorflow-gpu

See here for more details.


Instruction: How to get the output gradient w.r.t input

Input:
I have some problem with getting the output gradient of input.
It is simple mnist model.
for num,(sample_img, sample_label) in enumerate(mnist_test):
    if num == 1:
        break

    sample_img = sample_img.to(device)
    sample_img.requires_grad = True
    prediction = model(sample_img.unsqueeze(dim=0))
    cost = criterion(prediction, torch.tensor([sample_label]).to(device))
    optimizer.zero_grad()
    cost.backward()
    print(sample_label)
    print(sample_img.shape)

    plt.imshow(sample_img.detach().cpu().squeeze(),cmap='gray')
    plt.show()

print(sample_img.grad)

sample_img.grad is None

Output:
If you need to compute the gradient with respect to the input you can do so by calling sample_img.requires_grad_(), or by setting sample_img.requires_grad = True, as suggested in your comments.
Here is a small example:
import torch
import torch.nn as nn
import torch.optim as optim
import matplotlib.pyplot as plt

model = nn.Sequential(  # a dummy model
    nn.Conv2d(1, 1, 3),
    nn.ReLU(),
    nn.MaxPool2d(2),
    nn.Flatten()
)

sample_img = torch.rand(1, 5, 5)  # a dummy input
sample_label = 0

criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(model.parameters(), lr=1e-3)
device = "cpu"

sample_img = sample_img.to(device)
sample_img.requires_grad = True

prediction = model(sample_img.unsqueeze(dim=0))
cost = criterion(prediction, torch.tensor([sample_label]).to(device))
optimizer.zero_grad()
cost.backward()
print(sample_label)
print(sample_img.shape)

plt.imshow(sample_img.detach().cpu().squeeze(), cmap='gray')
plt.show()

print(sample_img.grad.shape)
print(sample_img.grad)

Additionally, if you don't need the gradients of the model, you can set their gradient requirements off:
for param in model.parameters():
    param.requires_grad = False


Instruction: How to disable automatic checkpoint loading

Input:
Im trying to run a loop over a set of parameters and I wan't to make a new network for each parameter and let it learn a few epochs.
Currently my code looks like this:
def optimize_scale(self, epochs=5, comp_scale=100, scale_list=[1, 100]):
    trainer = pyli.Trainer(gpus=1, max_epochs=epochs)

    for scale in scale_list:
        test_model = CustomNN(num_layers=1, scale=scale, lr=1, pad=True, batch_size=1)
        trainer.fit(test_model)
        trainer.test(verbose=True)

        del test_model

Everything works fine for the first element of scale_list, the network learns 5 epochs and completes the test. All this can be seen in the console. However for all following elements of scale_list it doesn't work as the old network is not overwritten, but instead an old checkpoint is loaded automatically when trainer.fit(model) is called. In the console this is indicated through:
C:\Users\XXXX\AppData\Roaming\Python\Python39\site-packages\pytorch_lightning\callbacks\model_checkpoint.py:623: UserWarning:
Checkpoint directory D:\XXXX\src\lightning_logs\version_0\checkpoints exists and is not empty.
rank_zero_warn(f"Checkpoint directory {dirpath} exists and is not empty.")
train_size = 8   val_size = 1    test_size = 1
Restoring states from the checkpoint path at D:\XXXX\src\lightning_logs\version_0\checkpoints\epoch=4-step=39.ckpt
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
Loaded model weights from checkpoint at D:\XXXX\src\lightning_logs\version_0\checkpoints\epoch=4-step=39.ckpt

The consequence is that the second test outputs the same result, as the the checkpoint from the old network was loaded which already finished all 5 epochs. I though that adding the del test_model might help in dropping the model completely, but that did not work.
On my search I found a few Issues closely related, for example: https://github.com/PyTorchLightning/pytorch-lightning/issues/368. However I did not manage to fix my problem. I assume it has something to with the fact that the new network which should overwrite the old one has the same name/version and therefore looks for the same checkpoints.
If anyone has an idea or knows how to circumvent this I would be very grateful.

Output:
I think, in your settings, you want to disable automatic checkpointing:
trainer = pyli.Trainer(gpus=1, max_epochs=epochs,enable_checkpointing=False)

You may need to explicitly save a checkpoint (with a different name) for each training session you are running.
You can manually save a checkpoint via:
trainer.save_checkpoint(f'checkpoint_for_scale_{scale}.pth')


Instruction: How does torch.flatten() order the elements from the flattened dimensions

Input: I have a 4D tensor of shape [32,64,64,3] which corresponds to [batch, timeframes, frequency_bins, features] and I do tensor.flatten(start_dim=2) (in PyTorch). I understand the shape will then transform to [32,64,64*3] --> [batch,timeframes,frequency_bins*features] - but in terms of the actual ordering of the elements within that new flattened dimension of 64*3 are the first 64 indexes relating to what would have been [:,:,:,0] the second 64 [:,:,:,1] and the final 64 [:,:,:,2]?

Output:
For the sake of understanding, let us first take the simplest case where we have a tensor of rank 2, i.e., a regular matrix. PyTorch performs flattening in what is called row-major order, traversing from the "innermost" axis to the "outermost" axis.
Taking a simple 3x3 array of rank 2, let us call it A[3, 3]:
[[a, b, c],
 [d, e, f],
 [g, h, i]]

Flattening this from innermost to outermost axes would give you [a, b, c, d, e, f, g, h, i]. Let us call this flattened array B[3].
The relation between corresponding elements in A (at index [i, j]) and B (at index k) can easily be derived as:
k = A.size[1] * i + j

This is because to reach the element at [i, j], we first move i rows down, counting A.size[1] (i.e., the width of the array) elements for each row. Once we reach row i, we need to get to column j, thus we add j to obtain the index in the flattened array.
For example, element e is at index [1, 1] in A. In B, it would occupy the index 3 * 1 + 1 = 4, as expected.
Let us extend that same idea to a tensor of rank of rank 4, as in your case, where we are flattening only the last two axes.
Again, taking a simple rank 4 tensor A of shape (2, 2, 2, 2) as below:
A =
[[[[ 1,  2],
   [ 3,  4]],

   [[ 5,  6],
   [ 7,  8]]],

   [[[ 9, 10],
   [11, 12]],

   [[13, 14],
   [15, 16]]]]

Let us find a relation between the indices of A and torch.flatten(A, start_dim=2) (let's call the flattened version B).
B =
[[[ 0,  1,  2,  3],
  [ 4,  5,  6,  7]],

  [[ 8,  9, 10, 11],
  [12, 13, 14, 15]]]

Element 12 is at index [1, 1, 0, 0] in A and index [1, 1, 0] in B. Note that the indices at axes 0 and 1, i.e., [1, 1] remain intact even after partial flattening. This is because those axes are not flattened and thus not impacted.
This is fantastic! Thus, we can represent the transformation from A to B as
B[i, j, _] = A[i, j, _, _]

Our task now reduces to finding a relation between the last axis of B and the last 2 axes of A. But A[i, j, _, _] is a 2x2 array, for which we have already derived the relation k = A.size[1] * i + j,
A.size[1] would now change to A.size[3] as 3 is now the last axis. But the general relation remains.
Filling in the blanks, we get the relation between corresponding elements in A and B as:
B[i, j, k] = A[i, j, m, n]

where k = A.size[3] * m + n.
We can verify that this is correct. Element 14 is at [1, 1, 1, 0] in A. and moves to [1, 1, 2 * 1 + 0] = [1, 1, 2] in B.
EDIT: Added example
Taking @Molem7b5's example of array A with shape (1, 4, 4, 3), from the comments:
Iterating from inner (dim=3) to outer axes (dim=2) of A gives consecutive elements of B. What I mean by this is:
// Using relation: A[:, :, i, j] == B[:, :, 3 * i + j]

// i = 0, all j
A[:, :, 0, 0] == B[:, :, 0]
A[:, :, 0, 1] == B[:, :, 1]
A[:, :, 0, 2] == B[:, :, 2]
// (Note the consecutive order in B.)

// i = 1, all j
A[:, :, 1, 0] == B[:, :, 3]
A[:, :, 1, 1] == B[:, :, 4]

// and so on until

A[:, :, 3, 2] == B[:, :, 11]

This should give you a better picture as to how the flattening occurs. When in doubt, extrapolate from the relation.


Instruction: How to get a probability distribution over tokens in a huggingface model?

Input:
I'm following this tutorial on getting predictions over masked words. The reason I'm using this one is because it seems to be working with several masked word simultaneously while other approaches I tried could only take 1 masked word at a time.
The code:
from transformers import RobertaTokenizer, RobertaForMaskedLM
import torch

tokenizer = RobertaTokenizer.from_pretrained('roberta-base')
model = RobertaForMaskedLM.from_pretrained('roberta-base')

sentence = "Tom has fully ___ ___ ___ illness."

def get_prediction (sent):

    token_ids = tokenizer.encode(sent, return_tensors='pt')
    masked_position = (token_ids.squeeze() == tokenizer.mask_token_id).nonzero()
    masked_pos = [mask.item() for mask in masked_position ]

    with torch.no_grad():
        output = model(token_ids)

    last_hidden_state = output[0].squeeze()

    list_of_list =[]
    for index,mask_index in enumerate(masked_pos):
        mask_hidden_state = last_hidden_state[mask_index]
        idx = torch.topk(mask_hidden_state, k=5, dim=0)[1]
        words = [tokenizer.decode(i.item()).strip() for i in idx]
        list_of_list.append(words)
        print ("Mask ",index+1,"Guesses : ",words)

    best_guess = ""
    for j in list_of_list:
        best_guess = best_guess+" "+j[0]

    return best_guess

print ("Original Sentence: ",sentence)
sentence = sentence.replace("___","<mask>")
print ("Original Sentence replaced with mask: ",sentence)
print ("\n")

predicted_blanks = get_prediction(sentence)
print ("\nBest guess for fill in the blank :::",predicted_blanks)

How can I get the probability distribution over the 5 tokens instead of the indices of them? That is, similarly to how this approach (that I used before but once I change to multiple masked tokens I get an error) gets the score as an output:
from transformers import pipeline

# Initialize MLM pipeline
mlm = pipeline('fill-mask')

# Get mask token
mask = mlm.tokenizer.mask_token

# Get result for particular masked phrase
phrase = f'Read the rest of this {mask} to understand things in more detail'
result = mlm(phrase)

# Print result
print(result)

[{
    'sequence': 'Read the rest of this article to understand things in more detail',
    'score': 0.35419148206710815,
    'token': 1566,
    'token_str': ' article'
},...

Output:
The variable last_hidden_state[mask_index] is the logits for the prediction of the masked token. So to get token probabilities you can use a softmax over this, i.e.
probs = torch.nn.functional.softmax(last_hidden_state[mask_index])

You can then get the probabilities of the topk using
word_probs = [probs[i] for i in idx]

PS I assume you're aware that you should use <mask> rather then ___, i.e. sent = "Tom has fully <mask> <mask> <mask> illness.", I get the following:

Mask  1 Guesses :  ['recovered', 'returned', 'cleared', 'recover', 'healed']

[tensor(0.9970), tensor(0.0007), tensor(0.0003), tensor(0.0003), tensor(0.0002)]

Mask  2 Guesses :  ['from', 'his', 'with', 'to', 'the']

[tensor(0.5066), tensor(0.2048), tensor(0.0684), tensor(0.0513), tensor(0.0399)]

Mask  3 Guesses :  ['his', 'the','mental', 'serious', 'this']

[tensor(0.5152), tensor(0.2371), tensor(0.0407), tensor(0.0257), tensor(0.0199)]


Instruction: How to access loss at each epoch from Pytorch Lighting?

Input:
I'm using Pytorch Lighting and Tensorboard as PyTorch Forecasting library is build using them. I want to create my own loss curves via matplotlib and don't want to use Tensorboard.
It is possible to access metrics at each epoch via a method? Validation Loss, Training Loss etc?
My code is below:
logger = TensorBoardLogger("logs", name = "model")
trainer = pl.Trainer(#Some params)

Does logger or trainer have any method to access this information?
PL documentation isn't clear and there are many methods associated with logger and trainer.

Output:
My recommendation is that you:

Create a csv logger:

from pytorch_lightning.loggers import CSVLogger

csv_logger = CSVLogger(
    save_dir=str'./',
    name='csv_file'
)

Pass it to your trainer

# Initialize a trainer
trainer = Trainer(
    accelerator="auto",
    max_epochs=1,
    log_every_n_steps=10,
    logger=[csv_logger],
)

Have your model log your epoch results. This will trigger a write action into a txt file by the CSVlogger

class MNISTModel(LightningModule):
    def __init__(self):
        super().__init__()
        self.l1 = torch.nn.Linear(28 * 28, 10)

    def forward(self, x):
        return torch.relu(self.l1(x.view(x.size(0), -1)))

    def training_step(self, batch, batch_nb):
        x, y = batch
        loss = F.cross_entropy(self(x), y)
        self.log('loss_epoch', loss, on_step=False, on_epoch=True)
        return loss

    def configure_optimizers(self):
        return torch.optim.Adam(self.parameters(), lr=0.02)

Use the logged values into the CSV file for plotting you results. In this way, if you are unhappy with your plot you would be able to just re-run everything with your plot script modifications without having to wait for the training to end again.


Instruction: How to create torch.tensor with shape (1,1,32) with default value?

Input:
I want to create torch.tensor variable with shape (1,1,32) with default value (None).
How can I do it ?

Output:
I don't believe you can assign None to a torch.Tensor. What is more appropriate however is to instead assign NaN. You can do so using the builtin torch.full:
>>> torch.full((1, 1, 32), torch.nan)
tensor([[[nan, nan, nan, nan, nan, nan, nan, nan,
          nan, nan, nan, nan, nan, nan, nan, nan,
          nan, nan, nan, nan, nan, nan, nan, nan,
          nan, nan, nan, nan, nan, nan, nan, nan]]])


Instruction: How to implement a diagonal data for a linear layer in pytorch

Input:
I would like to have a network in pytorch that only scale the data.
The mathematical notations for my request is:

which means that if my input is [1, 2] and my output is [2, 6].
then the linear layer will look like this:
[ [ 2, 0],
  [ 0, 3] ].

I have this network written in pytorch:
class ScalingNetwork(nn.Module):
    def __init__(self, input_shape, output_shape):
        super().__init__()
        self.linear_layer = nn.Linear(in_features=input_shape, out_features=output_shape)
        self.mask = torch.diag(torch.ones(input_shape))
        self.linear_layer.weight.data = self.linear_layer.weight * self.mask
        self.linear_layer.weight.requires_grad = True

    def get_tranformation_matrix(self):
        return self.linear_layer.weight

    def forward(self, X):
        X = self.linear_layer(X)
        return X

But at the end of the training, my self.linear is not diagonal.
What am I doing wrong?

Output:
It seems like an apparent constraint here is the fact that self.linear_layer needs to be a squared matrix. You can use the diagonal matrix self.mask to zero out all non-diagonal elements in the forward pass:
class ScalingNetwork(nn.Module):
    def __init__(self, in_features):
        super().__init__()
        self.linear = nn.Linear(in_features, in_features, bias=False)
        self.mask = torch.eye(in_features, dtype=bool)

    def forward(self, x):
        self.linear.weight.data *= self.mask
        print(self.linear.weight)
        x = self.linear(x)
        return x

For instance:
>>> m = ScalingNetwork(5)

>>> m(torch.rand(1,5))
Parameter containing:
tensor([[-0.2987, -0.0000, -0.0000, -0.0000, -0.0000],
        [ 0.0000, -0.1042, -0.0000, -0.0000, -0.0000],
        [-0.0000,  0.0000, -0.4267,  0.0000, -0.0000],
        [ 0.0000, -0.0000, -0.0000,  0.1758,  0.0000],
        [ 0.0000,  0.0000,  0.0000, -0.0000, -0.3208]], requires_grad=True)
tensor([[-0.1032, -0.0087, -0.1709,  0.0035, -0.1496]], grad_fn=<MmBackward0>)


Instruction: How to resolve the error that says "shapes cannot be multiplied"

Input:
I tried the code mentioned in the article in google colab.
https://theaisummer.com/spiking-neural-networks/
I got the error that looks like this...
Test loss:  8.86368179321289
Test loss:  5.338221073150635
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-9-646cb112ccb7> in <module>()
     15         # forward pass
     16         net.train()
---> 17         spk_rec, mem_rec = net(data.view(batch_size, -1))
     18
     19         # initialize the loss & sum over time

4 frames
/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py in linear(input, weight, bias)
   1846     if has_torch_function_variadic(input, weight, bias):
   1847         return handle_torch_function(linear, (input, weight, bias), input, weight, bias=bias)
-> 1848     return torch._C._nn.linear(input, weight, bias)
   1849
   1850

RuntimeError: mat1 and mat2 shapes cannot be multiplied (128x588 and 784x1000)

I am not sure how to fix it.

Output:
I just ran their Colab notebook and ran into the same error. It occurs because the final iteration does not have 128 samples of data, as the total dataset size (60000 and 10000 for training and test set) is not evenly divisible by 128. So there is some left over, and reshaping it to 128 x ... leads to a mismatch of dimensions between input data and the number of neurons in the input layer.
There are two possible fixes.

Just drop the final batch:

train_loader = DataLoader(mnist_train, batch_size=batch_size, shuffle=True, drop_last=True)
test_loader = DataLoader(mnist_test, batch_size=batch_size, shuffle=True, drop_last=True)

Don't drop the final batch. But flatten the tensor in a way that preserves the original batch_size, instead of forcing it to 128:

spk_rec, mem_rec = net(data.flatten(1))


Instruction: question of neural network training：the gradient of the same module which is used multiple times in one iteration

Input:
When training a neural network, if the same module is used multiple times in one iteration, does the gradient of the module need special processing during backpropagation?
for example:

One Deformable Compensation is used three times in this model, which means they share the same weights.
What will happen when I use loss.backward()?
Will loss.backward() work correctly？

Output:
The nice thing about autograd and backward passes is that the underlying framework is not "algorithmic", but rather a mathematic one: it implements the chain rule of derivatives. Therefore, there are no "algorithmic" considerations of "shared weights" or "weighting different layers", it's pure math. The backward pass provides the derivative of your loss function w.r.t the weights in a purely mathematical way.
Sharing weights can be done globally (e.g., when training Saimese networks), on a "layer level" (as in your example), but also within a layer. When you think about it Convolution layers and Reccurent layers are a fancy way of locally sharing weights.
Naturally, pytorch (as well as all other DL frameworks) can trivially handle these cases.
As long as your "deformable compensation" layer is correctly implemented -- pytorch will take care of the gradients for you, in a mathematically correct manner, thanks to the chain rule.


Instruction: How to make conda use its own gcc version?

Input:
I am trying to run the training of stylegan2-pytorch on a remote system. The remote system has gcc (9.3.0) installed on it. I'm using conda env that has the following installed (cudatoolkit=10.2, torch=1.5.0+, and ninja=1.8.2, gcc_linux-64=7.5.0).
I encounter the following error:
RuntimeError: Error building extension 'fused': [1/2]
/home/envs/segmentation_base/bin/nvcc -DTORCH_EXTENSION_NAME=fused -DTORCH_API_INCLUDE_EXTENSION_H -isystem /home/envs/segmentation_base/lib/python3.6/site-packages/torch/include -isystem /home/envs/segmentation_base/lib/python3.6/site-packages/torch/include/torch/csrc/api/include -isystem /home/envs/segmentation_base/lib/python3.6/site-packages/torch/include/TH -isystem /home/envs/segmentation_base/lib/python3.6/site-packages/torch/include/THC -isystem /home/envs/segmentation_base/include -isystem /home/envs/segmentation_base/include/python3.6m -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_70,code=sm_70 --compiler-options '-fPIC' -std=c++14 -c /home/code/semanticGAN_code/models/op/fused_bias_act_kernel.cu -o fused_bias_act_kernel.cuda.o
FAILED: fused_bias_act_kernel.cuda.o
/home/envs/segmentation_base/bin/nvcc -DTORCH_EXTENSION_NAME=fused -DTORCH_API_INCLUDE_EXTENSION_H -isystem /home/envs/segmentation_base/lib/python3.6/site-packages/torch/include -isystem /home/envs/segmentation_base/lib/python3.6/site-packages/torch/include/torch/csrc/api/include -isystem /home/envs/segmentation_base/lib/python3.6/site-packages/torch/include/TH -isystem /home/envs/segmentation_base/lib/python3.6/site-packages/torch/include/THC -isystem /home/envs/segmentation_base/include -isystem /home/envs/segmentation_base/include/python3.6m -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_70,code=sm_70 --compiler-options '-fPIC' -std=c++14 -c /home/code/semanticGAN_code/models/op/fused_bias_act_kernel.cu -o fused_bias_act_kernel.cuda.o
In file included from /home/envs/segmentation_base/include/cuda_runtime.h:83,
                 from <command-line>:
/home/envs/segmentation_base/include/crt/host_config.h:138:2: error: #error -- unsupported GNU version! gcc versions later than 8 are not supported!
  138 | #error -- unsupported GNU version! gcc versions later than 8 are not supported!
      |  ^~~~~
ninja: build stopped: subcommand failed.

I would like to use the gcc of my conda env (gcc_linux-64=7.5.0) to build cuda. When I run gcc --version in my conda env, I get the system's gcc:
gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0

which gcc when my conda env is active returns:
usr/bin/gcc
I'd expect it to return gcc version 7.5.0 (the one installed in the environment). I understand that conda has different names for gcc, but the environment variables should point to the installed gcc.
Running echo $CC returns
/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-cc.
Following suggested solution here, I get the following upon activating my environment, but the same issue stand:
INFO: activate-binutils_linux-64.sh made the following environmental changes:
+ADDR2LINE=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-addr2line
+AR=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-ar
+AS=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-as
+CXXFILT=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-c++filt
+ELFEDIT=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-elfedit
+GPROF=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-gprof
+LD_GOLD=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-ld.gold
+LD=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-ld
+NM=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-nm
+OBJCOPY=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-objcopy
+OBJDUMP=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-objdump
+RANLIB=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-ranlib
+READELF=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-readelf
+SIZE=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-size
+STRINGS=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-strings
+STRIP=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-strip
INFO: activate-gcc_linux-64.sh made the following environmental changes:
+build_alias=x86_64-conda-linux-gnu
+BUILD=x86_64-conda-linux-gnu
+CC_FOR_BUILD=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-cc
+CC=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-cc
+CFLAGS=-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /include -fdebug-prefix-map==/usr/local/src/conda/- -fdebug-prefix-map==/usr/local/src/conda-prefix
+CMAKE_ARGS=-DCMAKE_LINKER=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-ld -DCMAKE_STRIP=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-strip -DCMAKE_FIND_ROOT_PATH_MODE_PROGRAM=NEVER -DCMAKE_FIND_ROOT_PATH_MODE_LIBRARY=ONLY -DCMAKE_FIND_ROOT_PATH_MODE_INCLUDE=ONLY -DCMAKE_FIND_ROOT_PATH=;/x86_64-conda-linux-gnu/sysroot -DCMAKE_INSTALL_PREFIX= -DCMAKE_INSTALL_LIBDIR=lib
+CMAKE_PREFIX_PATH=:/home/envs/segmentation_base/x86_64-conda-linux-gnu/sysroot/usr
+CONDA_BUILD_SYSROOT=/home/envs/segmentation_base/x86_64-conda-linux-gnu/sysroot
+_CONDA_PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_x86_64_conda_linux_gnu
+CPPFLAGS=-DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /include
+CPP=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-cpp
+DEBUG_CFLAGS=-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-all -fno-plt -Og -g -Wall -Wextra -fvar-tracking-assignments -ffunction-sections -pipe -isystem /include -fdebug-prefix-map==/usr/local/src/conda/- -fdebug-prefix-map==/usr/local/src/conda-prefix
+DEBUG_CPPFLAGS=-D_DEBUG -D_FORTIFY_SOURCE=2 -Og -isystem /include
+GCC_AR=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-gcc-ar
+GCC_NM=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-gcc-nm
+GCC=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-gcc
+GCC_RANLIB=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-gcc-ranlib
+host_alias=x86_64-conda-linux-gnu
+HOST=x86_64-conda-linux-gnu
+LDFLAGS=-Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,/lib -Wl,-rpath-link,/lib -L/lib
INFO: activate-gxx_linux-64.sh made the following environmental changes:
+CXXFLAGS=-fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /include -fdebug-prefix-map==/usr/local/src/conda/- -fdebug-prefix-map==/usr/local/src/conda-prefix
+CXX_FOR_BUILD=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-c++
+CXX=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-c++
+DEBUG_CXXFLAGS=-fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-all -fno-plt -Og -g -Wall -Wextra -fvar-tracking-assignments -ffunction-sections -pipe -isystem /include -fdebug-prefix-map==/usr/local/src/conda/- -fdebug-prefix-map==/usr/local/src/conda-prefix
+GXX=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-g++

How could one set gcc to conda gcc instead of system gcc? I understand that should be done automatically, when activating the environment through bash scripts in activate.d.
Most of the open issues (regarding unsupported GNU version!) either require sudo permission to adjust gcc version (which I don't have) or aren't accepted in the case of conda environments. I have yet to find a clear solution to this :/
TLDR: How to force conda to use own installed gcc version instead of host system gcc?
Edit 1: Added conda list output
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main
_openmp_mutex             4.5                       1_gnu
_sysroot_linux-64_curr_repodata_hack 3                   haa98f57_10
absl-py                   1.0.0                    pypi_0    pypi
albumentations            0.5.2                    pypi_0    pypi
binutils_impl_linux-64    2.35.1               h27ae35d_9
binutils_linux-64         2.35.1              h454624a_30
blas                      1.0                         mkl
ca-certificates           2021.10.26           h06a4308_2
cachetools                4.2.4                    pypi_0    pypi
certifi                   2021.5.30        py36h06a4308_0
charset-normalizer        2.0.9                    pypi_0    pypi
cudatoolkit               10.2.89                       3    hcc
cycler                    0.11.0                   pypi_0    pypi
decorator                 4.4.2                    pypi_0    pypi
freetype                  2.11.0               h70c0345_0
gcc_impl_linux-64         7.5.0               h7105cf2_17
gcc_linux-64              7.5.0               h8f34230_30
google-auth               2.3.3                    pypi_0    pypi
google-auth-oauthlib      0.4.6                    pypi_0    pypi
grpcio                    1.42.0                   pypi_0    pypi
gxx_impl_linux-64         7.5.0               h0a5bf11_17
gxx_linux-64              7.5.0               hffc177d_30
idna                      3.3                      pypi_0    pypi
imageio                   2.8.0                    pypi_0    pypi
imageio-ffmpeg            0.4.2                    pypi_0    pypi
imgaug                    0.4.0                    pypi_0    pypi
importlib-metadata        4.8.2                    pypi_0    pypi
intel-openmp              2021.4.0          h06a4308_3561
jpeg                      9d                   h7f8727e_0
kernel-headers_linux-64   3.10.0              h57e8cba_10
kiwisolver                1.3.1                    pypi_0    pypi
lcms2                     2.12                 h3be6417_0
ld_impl_linux-64          2.35.1               h7274673_9
libffi                    3.3                  he6710b0_2
libgcc-devel_linux-64     7.5.0               hbbeae57_17
libgcc-ng                 9.3.0               h5101ec6_17
libgomp                   9.3.0               h5101ec6_17
libpng                    1.6.37               hbc83047_0
libstdcxx-devel_linux-64  7.5.0               hf0c5c8d_17
libstdcxx-ng              9.3.0               hd4cf53a_17
libtiff                   4.2.0                h85742a9_0
libwebp-base              1.2.0                h27cfd23_0
lmdb                      0.98                     pypi_0    pypi
lz4-c                     1.9.3                h295c915_1
markdown                  3.3.6                    pypi_0    pypi
matplotlib                3.3.4                    pypi_0    pypi
mkl                       2020.2                      256
mkl-service               2.3.0            py36he8ac12f_0
mkl_fft                   1.3.0            py36h54f3939_0
mkl_random                1.1.1            py36h0573a6f_0
ncurses                   6.3                  h7f8727e_2
networkx                  2.5.1                    pypi_0    pypi
ninja                     1.8.2                    pypi_0    pypi
numpy                     1.19.5                   pypi_0    pypi
numpy-base                1.19.2           py36hfa32c7d_0
oauthlib                  3.1.1                    pypi_0    pypi
olefile                   0.46                     py36_0
opencv-python             4.5.4.60                 pypi_0    pypi
opencv-python-headless    4.5.4.60                 pypi_0    pypi
openjpeg                  2.4.0                h3ad879b_0
openssl                   1.1.1l               h7f8727e_0
pillow                    8.4.0                    pypi_0    pypi
pip                       21.2.2           py36h06a4308_0
protobuf                  3.19.1                   pypi_0    pypi
pyasn1                    0.4.8                    pypi_0    pypi
pyasn1-modules            0.2.8                    pypi_0    pypi
pyparsing                 3.0.6                    pypi_0    pypi
python                    3.6.13               h12debd9_1
python-dateutil           2.8.2                    pypi_0    pypi
pytorch                   1.5.0           py3.6_cuda10.2.89_cudnn7.6.5_0    pytorch
pywavelets                1.1.1                    pypi_0    pypi
pyyaml                    6.0                      pypi_0    pypi
readline                  8.1                  h27cfd23_0
requests                  2.26.0                   pypi_0    pypi
requests-oauthlib         1.3.0                    pypi_0    pypi
rsa                       4.8                      pypi_0    pypi
scikit-image              0.17.2                   pypi_0    pypi
scipy                     1.5.0                    pypi_0    pypi
setuptools                58.0.4           py36h06a4308_0
shapely                   1.8.0                    pypi_0    pypi
six                       1.16.0             pyhd3eb1b0_0
sqlite                    3.36.0               hc218d9a_0
sysroot_linux-64          2.17                h57e8cba_10
tensorboard               2.7.0                    pypi_0    pypi
tensorboard-data-server   0.6.1                    pypi_0    pypi
tensorboard-plugin-wit    1.8.0                    pypi_0    pypi
tifffile                  2020.9.3                 pypi_0    pypi
tk                        8.6.11               h1ccaba5_0
torchvision               0.6.0                py36_cu102    pytorch
typing-extensions         4.0.1                    pypi_0    pypi
urllib3                   1.26.7                   pypi_0    pypi
werkzeug                  2.0.2                    pypi_0    pypi
wheel                     0.37.0             pyhd3eb1b0_1
xz                        5.2.5                h7b6447c_0
zipp                      3.6.0                    pypi_0    pypi
zlib                      1.2.11               h7b6447c_3
zstd                      1.4.9                haebb681_0

Output:
In addition to the solution posted in this issue. I added symbolic-links that point to the conda installed gcc, which I was missing.
ln -s /home/envs/segmentation_base/bin/x86_64-conda_cos6-linux-gnu-cc gcc
ln -s /home/envs/segmentation_base/bin/x86_64-conda_cos6-linux-gnu-cpp g++


Instruction: ToPILImage : TypeError: Input type int64 is not supported

Input:
I'm trying to develop a GAN using FastAi. When converting the Tensor to an Image I get this error.
Traceback (most recent call last):
  File "/Users/DevDog/Documents/vsc/pokemon/dementad.py", line 44, in <module>
    im =transforms.ToPILImage()(img[0]).convert('RGBA')
  File "/Users/DevDog/miniforge3/envs/python386/lib/python3.8/site-packages/torchvision/transforms/transforms.py", line 179, in __call__
    return F.to_pil_image(pic, self.mode)
  File "/Users/DevDog/miniforge3/envs/python386/lib/python3.8/site-packages/torchvision/transforms/functional.py", line 290, in to_pil_image
    raise TypeError('Input type {} is not supported'.format(npimg.dtype))
TypeError: Input type int64 is not supported

Here's the full code
import fastai
from fastai.data import transforms
from fastai.data.block import DataBlock, TransformBlock
from fastai.data.transforms import get_image_files
from fastai.optimizer import RMSProp
from fastai.vision.data import ImageBlock, ImageDataLoaders
from fastcore.imports import noop
from numpy import negative
import torch
import cv2
import PIL
from torchvision import transforms
from PIL import Image
from torch import nn
from fastai.vision import *
from fastai.vision.augment import *
from fastai.imports import *
from fastai.vision.gan import *
from fastai.data.block import *
from fastai.data.transforms import *
from fastai.callback.all import *
path = Path('pokeman')

bs=100
size=64
dblock = DataBlock(blocks = (TransformBlock, ImageBlock),
                   get_x = generate_noise,
                   get_items = get_image_files,
                   splitter = IndexSplitter([]),
                   item_tfms=Resize(size, method=ResizeMethod.Crop),
                   batch_tfms = Normalize.from_stats(torch.tensor([0.5,0.5,0.5]), torch.tensor([0.5,0.5,0.5])))
dls = dblock.dataloaders(path,path=path,bs=bs)
generator = basic_generator(64,3,n_extra_layers=1)
critic = basic_critic(64, 3, n_extra_layers=1,act_cls=partial(nn.LeakyReLU))
student = GANLearner.wgan(dls,generator,critic,opt_func = RMSProp)
student.recorder.train_metrics=True
student.recorder.valid_metrics=False
student.fit(1,2e-4,wd=0.)
#cv2.waitKey(0)
student.show_results(max_n=9,ds_idx=0)
student.gan_trainer.switch(gen_mode=True)
img = student.predict(generate_noise('pocheman',size=100))
print(img[0].size())
im =transforms.ToPILImage()(img[0]).convert('RGB')

The point of the Code is to generate pokemon images. But whenever I predict and convert it to a PIL Image the code fails with the aforementioned error.

Output:
I suggest for you to use this code to convert the output of your model from a tensor to a PIL image:
img = Image.fromarray((255*imgs[0]).numpy().astype(np.uint8).transpose(1, 2, 0))


Instruction: loss function with pytorch

Input:
I'm new to pytorch, when I see tutorials with MNIST dataset the target is a scalar (a digit from 0 to 9) and the output of the model is a layer is a vector (the code of the last layer is nn.Linear(32,10)) and they calculte the loss with (loss=nn.CrossEntropyLoss()   loss = loss(output,target) )
are they compareing digit with a vector ?

Output: I think that according to the PyTorch documentation torch.nn.functional.cross_entropy() that the output is like you mentioned a tensor with shape (N,C) (N is batch size) and C is the number of classes, where the target is either shape (N) when containing only class indices and shape (N,C) when containing also class probabilities. Details about how they compute the actual cross entropy is mentioned here (pytorch docs). So yes they are comparing a digit to a tensor in the sense that the digit tries to show which index should be 1 and therefore the other ones 0.


Instruction: Pytorch ValueError: either size or scale_factor should be defined

Input:
Background Information
I am trying to create a model (a beginner so please excuse my ignorance).
The architecture I am trying to convert is given below as a link as well.
![The Architecture to convert to a model][0]

This is the code I came up with. I am using Binder to run the code.
import os
import torch
import torchvision
import tarfile
from torchvision.datasets.utils import download_url
from torch.utils.data import random_split
from torchsummary import summary

# Implementation of CNN/ConvNet Model

class build_unet(torch.nn.Module):

    def __init__(self):
        super(build_unet, self).__init__()

        keep_prob = 0.5
        self.layer1 = torch.nn.Sequential(
            torch.nn.Conv2d(3, 32, kernel_size=3),
            torch.nn.ReLU(),
            torch.nn.MaxPool2d(kernel_size=2, padding=1))

        self.layer2 = torch.nn.Sequential(
            torch.nn.Conv2d(32, 64, kernel_size=3),
            torch.nn.ReLU(),
            torch.nn.MaxPool2d(kernel_size=2, padding=1))

        self.layer3 = torch.nn.Sequential(
            torch.nn.Conv2d(64, 128, kernel_size=3),
            torch.nn.ReLU(),
            torch.nn.MaxPool2d(kernel_size=2, padding=1))

        self.dense = torch.nn.Linear(64, 128, bias=True)
        torch.nn.init.xavier_uniform_(self.dense.weight)
        self.layer4 = torch.nn.Sequential(
            self.dense,
            torch.nn.ReLU(),
            torch.nn.Upsample()
        )

        self.layer5 = torch.nn.Sequential(
            torch.nn.Conv2d(128, 128, kernel_size=3),
            torch.nn.Sigmoid(),
            torch.nn.Upsample()
        )

        self.layer6 = torch.nn.Sequential(
            torch.nn.Conv2d(128, 64, kernel_size=3),
            torch.nn.Sigmoid(),
            torch.nn.Upsample()
        )

        self.layer7 = torch.nn.Sequential(
            torch.nn.Conv2d(64, 1, kernel_size=3),
            torch.nn.Sigmoid()
        )

    def forward(self, x):
        out = self.layer1(x)
        out = self.layer2(out)
        out = self.layer3(out)
        out = self.layer4(out)
        out = self.layer5(out)
        out = self.layer6(out)
        out = self.layer7(out)

        return out

if __name__ == "__main__":
    x = torch.randn((2, 3, 512, 512))
    f = build_unet()
    y = f(x)
    print(y.shape)

How would I resolve this error?
ERROR MESSAGE
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
/tmp/ipykernel_36/1438699785.py in <module>
     87     x = torch.randn((2, 3, 512, 512))
     88     f = build_unet()
---> 89     y = f(x)
     90     print(y.shape)

/opt/conda/lib/python3.9/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1049         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1050                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1051             return forward_call(*input, **kwargs)
   1052         # Do not call functions when jit is used
   1053         full_backward_hooks, non_full_backward_hooks = [], []

/tmp/ipykernel_36/1438699785.py in forward(self, x)
     72         out = self.layer3(out)
     73
---> 74         out = self.layer4(out)
     75         out = self.layer5(out)
     76         out = self.layer6(out)

/opt/conda/lib/python3.9/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1049         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1050                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1051             return forward_call(*input, **kwargs)
   1052         # Do not call functions when jit is used
   1053         full_backward_hooks, non_full_backward_hooks = [], []

/opt/conda/lib/python3.9/site-packages/torch/nn/modules/container.py in forward(self, input)
    137     def forward(self, input):
    138         for module in self:
--> 139             input = module(input)
    140         return input
    141

/opt/conda/lib/python3.9/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1049         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1050                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1051             return forward_call(*input, **kwargs)
   1052         # Do not call functions when jit is used
   1053         full_backward_hooks, non_full_backward_hooks = [], []

/opt/conda/lib/python3.9/site-packages/torch/nn/modules/upsampling.py in forward(self, input)
    139
    140     def forward(self, input: Tensor) -> Tensor:
--> 141         return F.interpolate(input, self.size, self.scale_factor, self.mode, self.align_corners)
    142
    143     def extra_repr(self) -> str:

/opt/conda/lib/python3.9/site-packages/torch/nn/functional.py in interpolate(input, size, scale_factor, mode, align_corners, recompute_scale_factor)
   3647             scale_factors = [scale_factor for _ in range(dim)]
   3648     else:
-> 3649         raise ValueError("either size or scale_factor should be defined")
   3650
   3651     if recompute_scale_factor is None:

ValueError: either size or scale_factor should be defined

Output:
nn.Upsample() has following parameters: size, scale_factor, mode, align_corners. By default size=None, mode=nearest and align_corners=None.

torch.nn.Upsample(size=None, scale_factor=None, mode='nearest', align_corners=None)

When you set scale_factor=2 you will get following result:
import torch
import torch.nn as nn

class Net(torch.nn.Module):

    def __init__(self):
        super(Net, self).__init__()

        keep_prob = 0.5
        self.layer1 = nn.Sequential(
            nn.Conv2d(3, 32, kernel_size=3),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, padding=1))

        self.layer2 = nn.Sequential(
            nn.Conv2d(32, 64, kernel_size=3),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, padding=1))

        self.layer3 = nn.Sequential(
            nn.Conv2d(64, 128, kernel_size=3),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, padding=1))

        self.dense = nn.Linear(64, 128, bias=True)
        nn.init.xavier_uniform_(self.dense.weight)
        self.layer4 = nn.Sequential(
            self.dense,
            nn.ReLU(),
            nn.Upsample(scale_factor=2)
        )

        self.layer5 = nn.Sequential(
            nn.Conv2d(128, 128, kernel_size=3),
            nn.Sigmoid(),
            nn.Upsample(scale_factor=2)
        )

        self.layer6 = nn.Sequential(
            nn.Conv2d(128, 64, kernel_size=3),
            nn.Sigmoid(),
            nn.Upsample(scale_factor=2)
        )

        self.layer7 = nn.Sequential(
            nn.Conv2d(64, 1, kernel_size=3),
            nn.Sigmoid()
        )

    def forward(self, x):
        out = self.layer1(x)
        out = self.layer2(out)
        out = self.layer3(out)
        out = self.layer4(out)
        out = self.layer5(out)
        out = self.layer6(out)
        out = self.layer7(out)

        return out

if __name__ == "__main__":
    x = torch.randn((2, 3, 512, 512))
    f = Net()
    y = f(x)
    print(y.shape)

Result:
torch.Size([2, 1, 498, 1010])


Instruction: Change shape of pytorch tensor

Input:
I have a pytorch tensor with a shape: torch.size([6000, 30, 30, 9]) and I want to convert it into the shape: torch.size([6000, 8100]) such that I go from 6000 elements that contain 30 elements that in turn contain 30 elements that in turn contain 9 elements TO
6000 elements that contain 8100 elements. How do I achieve it?

Output:
let's say you have a tensor x with the shape torch.size([6000, 30, 30, 9]). In Pytorch, To change the shape of it to torch.size([6000, 8100]), you can use the function view or reshape to keep the first dimension of the tensor (6000) and flatten the rest of dimensions (30,30,9) as follows:
import torch

x= torch.rand(6000, 30, 30, 9)
print(x.shape) #torch.Size([6000, 30, 30, 9])
x=x.view(6000,-1) # or x= x.view(x.size(0),-1)
print(x.shape) #torch.Size([6000, 8100])

x= torch.rand(6000, 30, 30, 9)
print(x.shape) #torch.Size([6000, 30, 30, 9])
x=x.reshape(6000,-1) # or x= x.reshape(x.size(0),-1)
print(x.shape) #torch.Size([6000, 8100])


Instruction: Can I install pytorch cpu + any specified version of cudatoolkit?

Input:
My remote has cuda==11.0 and I want to install pytorch on it.
I use the command conda install pytorch cudatoolkit=11.0 -c pytorch -c conda-forge but in the installation list:
cudatoolkit        conda-forge/linux-64::cudatoolkit-11.0.3-h15472ef_8
pytorch            pytorch/linux-64::pytorch-1.10.0-py3.8_cpu_0

I found that pytorch is a cpu one.
Alternatively, I substitute 11.0 with 11.1 and the installation list appears to be:
cudatoolkit        conda-forge/linux-64::cudatoolkit-11.1.1-h6406543_8
pytorch            pytorch/linux-64::pytorch-1.10.0-py3.8_cuda11.1_cudnn8.0.5_0

where pytorch is a gpu one.
My question is: are the above two installation essentially same? If not, how can I install pytorch=1.10.0 with cuda==11.0?
I'd also like to know how does the cuda compatibility work? Is a cudatoolkit==11.1 compatible with programs compiled with cudatoolkit==11.0?

Output:
It all depends on whether the pytorch channel has built a version against the particular cudatoolkit version. I don't know a specific way to search this, but one can browse what builds are available on the pytorch channel. For PyTorch 1.10 on linux-64 platform it appears only CUDA versions 10.2, 11.1, and 11.3 are available.
As mentioned in the comments, one can try forcing a CUDA build of PyTorch with
conda create -n foo -c pytorch -c conda-forge cudatoolkit=11.0 'pytorch=*=*cuda*'

which would fail in this combination.
As for compatibility, no, the pytorch package builds lock in the minor version of cudatoolkit. For example,


Instruction: separately save the model weight in pytorch

Input:
I am using PyTorch to train a deep learning model. I wonder if it is possible for me to separately save the model weight. For example:
class my_model(nn.Module):
def __init__(self):
    super(my_model, self).__init__()
    self.bert = transformers.AutoModel.from_pretrained(BERT_PATH)
    self.out = nn.Linear(768,1)

def forward(self, ids, mask, token_type):
    x = self.bert(ids, mask, token_type)[1]
    x = self.out(x)

    return x

I have the BERT model as the base model and an additional linear layer on the top. After I train this model, can I save the weight for the BERT model and this linear layer separately?

Output:
Alternatively to the previous answer, You can create two separated class of nn.module. One for the BERT model and another one for the linear layer:
class bert_model(nn.Module):
  def __init__(self):
  super(bert_model, self).__init__()
  self.bert = transformers.AutoModel.from_pretrained(BERT_PATH)

  def forward(self, ids, mask, token_type):
    x = self.bert(ids, mask, token_type)[1]

    return x

class linear_layer(nn.Module):
  def __init__(self):
  super(linear_layer, self).__init__()
  self.out = nn.Linear(768,1)

  def forward(self, x):
    x = self.out(x)

    return x

Then you can save the two part of the model separately with:
bert_model = bert_model()
linear_layer = linear_layer()
#train
torch.save(bert_model.state_dict(), PATH)
torch.save(linear_layer.state_dict(), PATH)


Instruction: Learnable LeakyReLU activation function with Pytorch

Input:
I'm trying to write a class for Invertible trainable LeakyReLu in which the model modifies the negative_slope in each iteration,
class InvertibleLeakyReLU(nn.Module):
  def __init__(self, negative_slope):
    super(InvertibleLeakyReLU, self).__init__()
    self.negative_slope = torch.tensor(negative_slope, requires_grad=True)
  def forward(self, input, logdet = 0, reverse = False):
    if reverse == True:
      input = torch.where(input>=0.0, input, input *(1/self.negative_slope))

      log = - torch.where(input >= 0.0, torch.zeros_like(input), torch.ones_like(input) * math.log(self.negative_slope))
      logdet = (sum(log, dim=[1, 2, 3]) +logdet).mean()
      return input, logdet
    else:
      input = torch.where(input>=0.0, input, input *(self.negative_slope))

      log = torch.where(input >= 0.0, torch.zeros_like(input), torch.ones_like(input) * math.log(self.negative_slope))
      logdet = (sum(log, dim=[1, 2, 3]) +logdet).mean()
      return input, logdet

However I set requires_grad=True, the negative slope wouldn't update. Are there any other points that I must modify?

Output:
Does your optimizer know it should update InvertibleLeakyReLU.negative_slope?
My guess is - no:
self.negative_slope is not defined as nn.Parameter, and therefore, by default, when you initialize your optimizer with model.parameters() negative_slope is not one of the optimization parameters.
You can either define negative_slope as a nn.Parameter:
self.negative_slope = nn.Parameter(data=torch.tensor(negative_slope), requires_grad=True)

Or, explicitly pass negative_slope from all InvertibleLeakyReLU in your model to the optimizer.


Instruction: Missing keys when loading the model weight in pytorch

Input:
I plan to load weight from a pth file, e.g.,
model = my_model()
model.load_state_dict(torch.load("../input/checkpoint/checkpoint.pth")

However, here is an error, saying:
RuntimeError: Error(s) in loading state_dict for my_model:
Missing key(s) in state_dict: "att.in_proj_weight", "att.in_proj_bias", "att.out_proj.weight", "att.out_proj.bias".
Unexpected key(s) in state_dict: "in_proj_weight", "in_proj_bias", "out_proj.weight", "out_proj.bias".

seems that the parameter name of my model is different from the one that stored in the state_dict. In this case, how am I supposed to make them consistent?

Output:
You can create new dictionary and modify keys without att. prefix and you can load the new dictionary to your model as following:

state_dict = torch.load('path\to\checkpoint.pth')

from collections import OrderedDict
new_state_dict = OrderedDict()

for key, value in state_dict.items():
    key = key[4:] # remove `att.`
    new_state_dict[key] = value

# load params
model = my_model()
model.load_state_dict(new_state_dict)


Instruction: Pytorch, retrieving values from a tensor using several indices. Most computationally efficient solution

Input:
If I have an example 3d tensor
a = [[4, 2, 1, 6],[1, 2, 3, 8], [92, 4, 23, 54]]
tensor_a = torch.tensor(a)

I can get 2 of the 1D tensors along the first dimension using
tensor_a[[0, 1]]

tensor([[4, 2, 1, 6],
        [1, 2, 3, 8]])

But how about using several indices?
So I have something like this
list_indices = [[0,  0], [0,2], [1, 2]]
I could do something like
combos = []
for indi in list_indices:
    combos.append(tensor_a[indi])

But I'm wondering if since there's a for loop, if there's a more computationally way to do this, perhaps also using pytorch

Output:
It is more computationally effecient to use the predefined Pytorch function "torch.index_select" to select tensor elements using a list of indices:
a = [[4, 2, 1, 6],[1, 2, 3, 8], [92, 4, 23, 54]]
tensor_a = torch.tensor(a)
list_indices = [[0,  0], [0,2], [1, 2]]
#convert list_indices to Tensor
indices = torch.tensor(list_indices)
#get elements from tensor_a using indices.
tensor_a=torch.index_select(tensor_a, 0, indices.view(-1))
print(tensor_a)

if you want the result to be a list not a tensors, you can convert tensor_a to a list:
tensor_a_list = tensor_a.tolist()

To test the computational efficiency I created 1000000 indices and I compared the execution time. Using the loop takes more time then using my suggested pytorch approach:
import time
import torch
start_time = time.time()
a = [[4, 2, 1, 6],[1, 2, 3, 8], [92, 4, 23, 54]]
tensor_a = torch.tensor(a)
indices = torch.randint(0, 2, (1000000,)).tolist()
for indi in indices:
   combos.append(tensor_a[indi])
print("--- %s seconds ---" % (time.time() - start_time))
--- 3.3966853618621826 seconds ---

start_time = time.time()
indices = torch.tensor(indices)
tensor_a=torch.index_select(tensor_a, 0, indices)
print("--- %s seconds ---" % (time.time() - start_time))
--- 0.10641193389892578 seconds ---


Instruction: Create CNN model for video resolution recognition from split frames

Input:
I'm working on a student project involved with resolution recognition from videos
My job is to prepare a training dataset from videos (I'm downloading these movies from YT) and does it in the following steps

Downloading pre-selected videos in every quality (2160p, 1440p,
1080p,720p...)
Extracting frames from every downloaded video (something about 20-30
frames)
Upscaling every frame to the same resolution (in my case I upscale
all frames to 4K)
Extracted frames have different dimensions so I need to expand them all to the same resolution
Splitting these upscaled frames to 100x100 blocks

After completing this process, he gets a great deal of sorted data
Below is a picture of what it looks like

On the left, you can see sorted directories by resolutions
In the middle directories for randomly downloaded videos
On the right mentioned in the fourth point 100x100 blocks from each video for each quality

The result I would like to achieve is that model from the same prepared dataset as for training would be able to properly recognize the quality (e.g. for a full had video output would be 1080p)
Now I'm wondering about the choice of a ready-made model using CNN.
My questions:

What solution do you think I should use here?
With the current set of data, how should I label it or do a different set of data?

Thank you very much in advance for your answers

Output:
It seems like you are actually trying to solve an easier problem than the discriminator of KernelGAN:
Sefi Bell-Kligler, Assaf Shocher, Michal Irani Blind Super-Resolution Kernel Estimation using an Internal-GAN (NeurIPS 2019).
In their work, they tried to estimate an arbitrary downsampling kernel relating HR and LR images. Your work is much simpler: you only try to select between several known upsampling kernels. Since your upscaling method is known, you only need to recover the amount of upscaling.
I suggest you start with a CNN that has an architecture similar to the discriminator of KernelGAN. However, I would consider increasing significantly the receptive field so it can reason about upscaling from 144p to 4K.
Side notes:

Do not change the aspect ratio of the frames when you upscale them! this will make your problem much more difficult: you will need to estimate two upscaling parameters (horizontal/vertical) instead of only one.
Do not crop 100x100 regions in advance - let your Dataset's transformations do it for you as random augmentations.


Instruction: How does pytorch L1-norm pruning works?

Input:
Lets see the result that I got first. This is one of a convolution layer of my model, and im only showing 11 filter's weight of it (11 3x3 filter with channel=1)
Left side is original weight Right side is Pruned weight
So I was wondering how does the "TORCH.NN.UTILS.PRUNE.L1_UNSTRUCTURED" works because by the pytorch website said, it prune the lowest L1-norm unit, but as far as I know, L1-norm pruning is a filter pruning method which prune the whole filter which use this equation to fine the lowest filter value instead of pruning single weight. So I'm a bit curious about how does this function actually works?
The following is my pruning code
parameters_to_prune = (
    (model.input_layer[0], 'weight'),
    (model.hidden_layer1[0], 'weight'),
    (model.hidden_layer2[0], 'weight'),
    (model.output_layer[0], 'weight')
)

prune.global_unstructured(
    parameters_to_prune,
    pruning_method=prune.L1Unstructured,
    amount = (pruned_percentage/100),
)

Output:
The nn.utils.prune.l1_unstructured utility does not prune the whole filter, it prunes individual parameter components as you observed in your sheet. That is components with the lower norm get masked.

Here is a minimal example as discussed in the comments below:
>>> m = nn.Linear(10,1,bias=False)
>>> m.weight = nn.Parameter(torch.arange(10).float())
>>> prune.l1_unstructured(m, 'weight', .3)
>>> m.weight
tensor([0., 0., 0., 3., 4., 5., 6., 7., 8., 9.], grad_fn=<MulBackward0>)


Instruction: Tensor Operation and gradient

Input:
I was going some tutorials on youtube where below code sample was used to explain derivatives
Create tensors.
    x = torch.tensor(3.)

    w = torch.tensor(4., requires_grad=True)

    b = torch.tensor(5., requires_grad=True)

    x, w, b

Arithmetic operations
    y = w * x + b

    y

Compute derivatives
    y.backward()

Display gradients
    print('dy/dx:', x.grad)

    print('dy/dw:', w.grad)

    print('dy/db:', b.grad)

OUTPUT
dy/dx: None
dy/dw: tensor(3.)
dy/db: tensor(1.)
Could anyone please explain me how we are getting tensor(3.) & tensor(1.) as an output of gradient. I need to understand how pytorch is performing this operation behind the scene.
Any help would be appreciated.

Output:
You have y = w*x + b, then
dy/dx = w
dy/dw = x
dy/db = 1

Since you've not set requires_grad=True for x, PyTorch won't calculate derivative w.r.t. it.
Hence, dy/dx = None
Rest are the values of corresponding tensors. Thus, the final output is
dy/dx: None
dy/dw: tensor(3.)
dy/db: tensor(1.)


Instruction: What is the difference between x.view(x.size(0), -1) and torch.nn.Flatten() layer and torch.flatten(x)? pytorch question

Input:
I'm quite curious on what's the difference between using view(,-1) and flatten like the simple code here:
Since I found that the size and data all flatten to one dimension.
import torch
from torch import nn
from torch.utils.data import DataLoader
from torchvision import datasets
from torchvision.transforms import ToTensor, Lambda, Compose
import matplotlib.pyplot as plt

x = torch.rand(3,256,256)
x.size()

a = x.view(x.size(0), -1)
print('after view:',a.size())

m = nn.Sequential(nn.Flatten())
y = m(x)
print('after nn flatten:',y.size())
z = torch.flatten(x)
print('after torch flatten:',y.size())

And also, it seems there is no difference between = and =.contiguous which said :Returns a contiguous in memory tensor containing the same data as self tensor. but for me, it seems just return the self tensor but not copy or new tensor with same data.
c = y
print(c)
b = y.contiguous()
print(b)

# change original data
y[0][0]=1
print(b)
print(c)
print(y)

Output:
A view is a way to modify the way you look at your data without modifying the data itself:

torch.view returns a view on the data: the data is not copied, only the "window" which you look through on the data changes
torch.flatten returns a one-dimensional output from a multi-dimensional input. It may not copy the data if

[the] input can be viewed as the flattened shape (source)

torch.nn.Flatten is just a wrapper for convenience around torch.flatten

Contiugous data just means that the data is linearly adressable in memory, e.g. for two dimension data this would mean that element [i][j] is at position i * num_columns + j. If this is already the case then .contiguous will not change your data or copy anything.


Instruction: 1. Weighted Loss in CrossEntropyLoss() 2. Combination of WeightedRandomSampler and subsampler

Input:
I wanted to implement class weights to my 3 class classification problem.
Tried by just directly adding the weights, which gives me an error when passing my model output and the labels to my loss
criterion = nn.CrossEntropyLoss(weight=torch.tensor([1,2,2]))

The error:
loss = criterion(out, labels)
expected scalar type Float but found Long

So I print dtypes and change them to float but it still gives me the same error
labels = labels.float()
print("Labels Training", labels, labels.dtype)
print("Out Training ", out, out.dtype)
loss = criterion(out, labels)

>>Labels Training tensor([2.]) torch.float32
>>Out Training  tensor([[ 0.0540, -0.1439, -0.0070]], grad_fn=<AddmmBackward0>) torch.float32
>>expected scalar type Float but found Long

I also tried to change it to float64(), but it tells me that tensor Object has no attribute float64

Problem: I Havent tried this one out but I have seen that the more used approach would be the RandomWeightedSampler. My problem is that I use CV with K-Fold and use a SubSampler for that. Is it possible to use both? Havent foudn anything related to that.

Output:
For the first Problem, nn.CrossEntropyLoss requires the output to be of type float, label of type long, and weight of type float. Therefore, you should change the optional parameter of nn.CrossEntropyLoss "weight" to be float by:
criterion = nn.CrossEntropyLoss(weight=torch.tensor([1.0,2.0,2.0]))
loss = criterion(out, labels.long())


Instruction: Using BatchNorm1d layer with Embedding and Linear layers for NLP text-classification problem throws RuntimeError

Input:
I am trying to create a neural network and train my own Embeddings. The network has the following structure (PyTorch):
import torch.nn as nn

class MultiClassClassifer(nn.Module):
  #define all the layers used in model
  def __init__(self, vocab_size, embedding_dim, hidden_dim, output_dim):

    #Constructor
    super(MultiClassClassifer, self).__init__()

    #embedding layer
    self.embedding = nn.Embedding(vocab_size, embedding_dim)

    #dense layer
    self.hiddenLayer = nn.Linear(embedding_dim, hidden_dim)

    #Batch normalization layer
    self.batchnorm = nn.BatchNorm1d(hidden_dim)

    #output layer
    self.output = nn.Linear(hidden_dim, output_dim)

    #activation layer
    self.act = nn.Softmax(dim=1) #2d-tensor

    #initialize weights of embedding layer
    self.init_weights()

  def init_weights(self):

    initrange = 1.0

    self.embedding.weight.data.uniform_(-initrange, initrange)

  def forward(self, text):

    embedded = self.embedding(text)

    hidden_1 = self.batchnorm(self.hiddenLayer(embedded))

    return self.act(self.output(hidden_1))

My training_iterator object looks like:
 batch = next(iter(train_iterator))
 batch.text_normalized_tweet[0]

tensor([[ 240,  538,  305,   73,    9,  780, 2038,   13,   48,    1,    1,    1,
            1,    1,    1,    1,    1,    1,    1,    1,    1],
        [ 853,   57,    2,   70, 1875,  176,  466,    1,    1,    1,    1,    1,
            1,    1,    1,    1,    1,    1,    1,    1,    1],
        ...])

with shape: torch.Size([32, 25]). 32= batch_size I used to create the training iterator with data.BucketIterator and 25 = the sequences in the batch.
When I create a model instance:
INPUT_DIM = len(TEXT.vocab) #~5,000 tokens
EMBEDDING_DIM = 100
HIDDEN_DIM = 64
OUTPUT_DIM = 3 #target has 3 classes

model = MultiClassClassifer(INPUT_DIM, EMBEDDING_DIM, HIDDEN_DIM, OUTPUT_DIM)

and execute
model(batch.text_normalized_tweet[0]).squeeze(1)

I get back the following RuntimeError

RuntimeError: running_mean should contain 15 elements not 64

You may also find my Golab Notebook here.

Output:
I found a workaround based on the example given by @jhso (above).
INPUT_DIM = len(TEXT.vocab) #~5,000 tokens
EMBEDDING_DIM = 100
HIDDEN_DIM = 64

e = nn.Embedding(INPUT_DIM, EMBEDDING_DIM)
l = nn.Linear(EMBEDDING_DIM, HIDDEN_DIM)
b = nn.BatchNorm1d(HIDDEN_DIM)
soft = nn.Softmax(dim=1)
out = nn.Linear(HIDDEN_DIM, 3)

text, text_lengths = batch.text_normalized_tweet
y = e(text)
tensor, batch_size = nn.utils.rnn.pack_padded_sequence(y,text_lengths, batch_first=True)[0], nn.utils.rnn.pack_padded_sequence(y,text_lengths, batch_first=True)[1] #added rnn.pack_padded_sequence
y = b(l(tensor))

Added pack_padded_sequence() method from utils.rnn package which will take the embeddings as input. I also had to calculate both the text and the text_lengths since the way I created the training_iteror it returns 2 outputs (text, text_lenght).


Instruction: Why do we need to pre-process image datasets?

Input:
Refer to this Complete guide on How to use Autoencoders in Python
Notice the author add:
x_train = x_train.astype('float32') / 255.
x_test = x_test.astype('float32') / 255.
x_train = x_train.reshape((len(x_train), np.prod(x_train.shape[1:])))
x_test = x_test.reshape((len(x_test), np.prod(x_test.shape[1:])))

after they loaded the MNIST data.
Why do they divide the image data by 255? And why 255? After that why do they reshape a 2d matrix into 1d?
Thank you so much!

Output:
Why dividing by 255:
The RGB value is of values up to 255 and you want to standardize your colors between 0 and 1.

Then why the transformation to a 1D vector is to easily send the whole vector into a model. If you have a 2D vector you will have to use other forms of input layers or different kinds of models which are built especially for this. In many cases a 2D vector can be indicative of timeseries datasets which I actually do not know if there are CNN implementations which may use 2D inputs for images.


Instruction: Pytorch, retrieving values from a 3D tensor using several indices. Most computationally efficient solution

Input:
Related:
Pytorch, retrieving values from a tensor using several indices. Most computationally efficient solution
This is another question about retrieving values from a 3D tensor, using a list of indices.
In this case, I have a 3d tensor, for example
b = [[[4, 20], [1, -1]], [[1, 2], [8, -1]], [[92, 4], [23, -1]]]
tensor_b = torch.tensor(b)
tensor_b

tensor([[[ 4, 20],
         [ 1, -1]],

        [[ 1,  2],
         [ 8, -1]],

        [[92,  4],
         [23, -1]]])

In this case, I have a list of 3D indices. So
indices = [
           [[1, 0, 1], [2, 0, 1]],
           [[1, 1, 1], [0, 0, 0]],
           [[2, 1, 0], [0, 1, 0]]
]

Each triple is an index for tensor-b. The desired result is
[[2, 4], [-1, 4], [23, 1]]

Potential Approach
Like in the last question, the first solution that comes to mind is a nested for loop, but there is probably a more computationally efficient solution using pytorch function.
And like in the last question, perhaps reshape would be needed to get the desired shape for the last solution.
So a desired solution could be [2, 4, -1, 4, 23, 1], which can come from a flattened list of indices
[ [1, 0, 1], [2, 0, 1], [1, 1, 1], [0, 0, 0], [2, 1, 0], [0, 1, 0] ]
But I am not aware of any pytorch functions so far which allow for a list of 3D indices. I have been looking at gather and index_select.

Output:
You can use advanced indexing specifically integer array indexing
tensor_b = torch.tensor([[[4, 20], [1, -1]], [[1, 2], [8, -1]], [[92, 4], [23, -1]]])
indices = torch.tensor([
           [[1, 0, 1], [2, 0, 1]],
           [[1, 1, 1], [0, 0, 0]],
           [[2, 1, 0], [0, 1, 0]]
])

result = tensor_b[indices[:, :, 0], indices[:, :, 1], indices[:, :, 2]]

results in
tensor([[ 2,  4],
        [-1,  4],
        [23,  1]])


Instruction: PyTorch multi-class: ValueError: Expected input batch_size (416) to match target batch_size (32)

Input:
I have created a mutli-class classification neural network. Training, and validation iterators where created with BigBucketIterator method with fields {'text_normalized_tweet':TEXT, 'label': LABEL}
TEXT = a tweet
LABEL = a float number (with 3 values: 0,1,2)
Below I execute a dummy example of my neural network:
import torch.nn as nn

class MultiClassClassifer(nn.Module):
  #define all the layers used in model
  def __init__(self, vocab_size, embedding_dim, hidden_dim, output_dim):

    #Constructor
    super(MultiClassClassifer, self).__init__()

    #embedding layer
    self.embedding = nn.Embedding(vocab_size, embedding_dim)

    #dense layer
    self.hiddenLayer = nn.Linear(embedding_dim, hidden_dim)

    #Batch normalization layer
    self.batchnorm = nn.BatchNorm1d(hidden_dim)

    #output layer
    self.output = nn.Linear(hidden_dim, output_dim)

    #activation layer
    self.act = nn.Softmax(dim=1) #2d-tensor

    #initialize weights of embedding layer
    self.init_weights()

  def init_weights(self):

    initrange = 1.0

    self.embedding.weight.data.uniform_(-initrange, initrange)

  def forward(self, text, text_lengths):

    embedded = self.embedding(text)

    #packed sequence
    packed_embedded = nn.utils.rnn.pack_padded_sequence(embedded, text_lengths, batch_first=True)

    tensor, batch_size = packed_embedded[0], packed_embedded[1]

    hidden_1 = self.batchnorm(self.hiddenLayer(tensor))

    return self.act(self.output(hidden_1))

Instantiate the model
INPUT_DIM = len(TEXT.vocab)
EMBEDDING_DIM = 100
HIDDEN_DIM = 64
OUTPUT_DIM = 3

model = MultiClassClassifer(INPUT_DIM, EMBEDDING_DIM, HIDDEN_DIM, OUTPUT_DIM)

When I call
text, text_lengths = batch.text_normalized_tweet

predictions = model(text, text_lengths).squeeze()

loss = criterion(predictions, batch.label)

it returns,

ValueError: Expected input batch_size (416) to match target batch_size (32).

model(text, text_lengths).squeeze() = torch.Size([416, 3])
batch.label = torch.Size([32])

I can see that the two objects have different sizes, but I have no clue how to fix this?
You may find the Google Colab notebook here
Shapes of each in, out tensor of my forward() method:
torch.Size([32, 10, 100]) #self.embedding(text)
torch.Size([320, 100]) #nn.utils.rnn.pack_padded_sequence(embedded, text_lengths, batch_first=True)
torch.Size([320, 64]) #self.batchnorm(self.hiddenLayer(tensor))
torch.Size([320, 3]) #self.act(self.output(hidden_1))

Output:
You shouldn't be using the squeeze function after the forward pass, that doesn't make sense.
After removing the squeeze function, as you see, the shape of your final output is [320,3] whereas it is expecting [32,3]. One way to fix this is to average out the embeddings you obtain for each word after the self.Embedding function like shown below:
def forward(self, text, text_lengths):

    embedded = self.embedding(text)
    embedded = torch.mean(embedded, dim=1, keepdim=True)

    packed_embedded = nn.utils.rnn.pack_padded_sequence(embedded, text_lengths, batch_first=True)
    tensor, batch_size = packed_embedded[0], packed_embedded[1]

    hidden_1 = self.batchnorm(self.hiddenLayer(tensor))
    return self.act(self.output(hidden_1))