Instruction: Cannot import "BasicBlock" from torchvision.models.resnet

Input:
I am trying to import the class BasicBlock from torchvision.models.resnet by doing this
from torchvision.models.resnet import *

It is giving no error but when I am trying to use BasicBlock class in my code (which is supposed to be imported already), I am getting the error that
NameError: name 'BasicBlock' is not defined
even though BasicBlock is present in torchvision.models.resnet
But it gives no error when I import like this
from torchvision.models.resnet import BasicBlock
and then use it in my code
Why am I getting this error?

Output: BasicBlock is indeed defined, however it is not exported by the module: see here the definition of __all__. So torchvision/models/resnet.py only exports these: ResNet, resnet18, resnet34, resnet50, resnet101, resnet152, resnext50_32x4d, resnext101_32x8d, wide_resnet50_2, and wide_resnet101_2.


Instruction: Why does the output of the neural network is the same for all samples

Input:
I'm using Pytorch to make a regression model (neural network). Just train and test it on the same sample (because i'm just learning how to build a neural network).
The data is the fish market dataset from kaggle:
https://www.kaggle.com/aungpyaeap/fish-market
Small sample from the training Tensor:
tensor([[ 0.0000, 23.2000, 25.4000, 30.0000, 11.5200,  4.0200],
        [ 0.0000, 24.0000, 26.3000, 31.2000, 12.4800,  4.3056],
        [ 0.0000, 23.9000, 26.5000, 31.1000, 12.3778,  4.6961],
        [ 0.0000, 26.3000, 29.0000, 33.5000, 12.7300,  4.4555],
        [ 0.0000, 26.5000, 29.0000, 34.0000, 12.4440,  5.1340],
        [ 0.0000, 26.8000, 29.7000, 34.7000, 13.6024,  4.9274],
        [ 0.0000, 26.8000, 29.7000, 34.5000, 14.1795,  5.2785],
        [ 0.0000, 27.6000, 30.0000, 35.0000, 12.6700,  4.6900],
        [ 0.0000, 27.6000, 30.0000, 35.1000, 14.0049,  4.8438],
        [ 0.0000, 28.5000, 30.7000, 36.2000, 14.2266,  4.9594],
        [ 0.0000, 28.4000, 31.0000, 36.2000, 14.2628,  5.1042],
        [ 0.0000, 28.7000, 31.0000, 36.2000, 14.3714,  4.8146],
        [ 0.0000, 29.1000, 31.5000, 36.4000, 13.7592,  4.3680],

Small Sample from the target tensor:
tensor([[ 242.0000],
        [ 290.0000],
        [ 340.0000],
        [ 363.0000],
        [ 430.0000],
        [ 450.0000],
        [ 500.0000],
        [ 390.0000],
        [ 450.0000],
        [ 500.0000],
        [ 475.0000],
        [ 500.0000],
        [ 500.0000],
        [ 340.0000],
        [ 600.0000],
        [ 600.0000],
        [ 700.0000],
        [ 700.0000],

The neural network:
class NeuralNetFish(nn.Module):

    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(6, 10)
        self.tanh1 = nn.Tanh()
        self.fc2 = nn.Linear(10, 10)
        self.tanh2 = nn.Tanh()
        self.fc3 = nn.Linear(10, 1)

    def forward(self, x):
        out = self.fc1(x)
        out = self.tanh1(out)
        out = self.fc2(out)
        out = self.tanh2(out)
        out = self.fc3(out)
        return out

model = NeuralNetFish()
criteron = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.02)

for i in range(10000):
    output = model(fish_tensor)
    loss = criteron(output, weight_tensor)

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    if i % 1000 == 0:
        print("Epoch:", i, " ---  Loss:", loss.item())

OUTPUT:
Epoch: 0  ---  Loss: 286071.78125
Epoch: 1000  ---  Loss: 127342.515625
Epoch: 2000  ---  Loss: 127342.515625
Epoch: 3000  ---  Loss: 127342.515625
Epoch: 4000  ---  Loss: 127342.515625
Epoch: 5000  ---  Loss: 127342.515625
Epoch: 6000  ---  Loss: 127342.515625
Epoch: 7000  ---  Loss: 127342.515625
Epoch: 8000  ---  Loss: 127342.515625
Epoch: 9000  ---  Loss: 127342.515625

And i get the same output for all the samples.

Output:
Scale your inputs.
Your inputs are roughly in range [0, 40], you take linear combinations of these numbers, and then apply the tanh function. Most likely your outputs will be at the "saturated" region of the tanh thus the gradients are roughly zero. Your neurons are dead.
To overcome this issue:

Scale your inputs: compute the mean and std over the training set (mean and std are 6-vectors in your case). Subtract the mean and divide by the std to have your inputs distributed roughly as N(0, 1).
Replace the tanh activations with nn.PReLU.
You should tune your learning-rate to be in accordance with the batch size (in your case batch size = entire dataset).


Instruction: How to convert dataset of images to tensor?

Input:
I've got a dataset of images that looks like this:
array([[[[0.35980392, 0.26078431, 0.14313725],
     [0.38137255, 0.26470588, 0.15196078],
     [0.51960784, 0.3745098 , 0.26176471],
     ...,
     [0.34313725, 0.22352941, 0.15      ],
     [0.30784314, 0.2254902 , 0.15686275],
     [0.28823529, 0.22843137, 0.16862745]],

    [[0.38627451, 0.28235294, 0.16764706],
     [0.45098039, 0.32843137, 0.21666667],
     [0.62254902, 0.47254902, 0.36470588],
     ...,
     [0.34607843, 0.22745098, 0.15490196],
     [0.30686275, 0.2245098 , 0.15588235],
     [0.27843137, 0.21960784, 0.16176471]],

    [[0.41568627, 0.30098039, 0.18431373],
     [0.51862745, 0.38529412, 0.27352941],
     [0.67745098, 0.52058824, 0.40980392],
     ...,
     [0.34901961, 0.22941176, 0.15588235],
     [0.29901961, 0.21666667, 0.14901961],
     [0.26078431, 0.20098039, 0.14313725]],

    ...,

What I need is convert it to tensor so that I could pass it to CNN.
I'm trying to do it like that:
from torchvision import transforms as transforms
transform = transforms.Compose([
            transforms.ToTensor(),
            transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
        ])

How can I apply this transform to my dataset? Thanks for any help.

Output:
you probably want to create a dataloader. You will need a class which iterates over your dataset, you can do that like this:
import torch
import torchvision.transforms

class YourDataset(torch.utils.data.Dataset):
    def __init__(self):

       # load your dataset (how every you want, this example has the dataset stored in a json file
        with open(<dataset-path>, "r") as f:
            self.dataset = json.load(f)

    def __getitem__(self, idx):
        sample = self.dataset[idx]
        data, label = sample[0], sample[1]

        transform = transforms.Compose([
            transforms.ToTensor(),
            transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
        ])
        return transform(data), torch.tensor(label)

    def __len__(self):
        return len(self.dataset)

Now you can create a dataloader:

train_set = YourDataset()

train_dataloader = torch.utils.data.DataLoader(
            train_set,
            batch_size=64,
            num_workers=1,
            shuffle=True,
        )

And now you can iterator over the dataloader in your train-loop:
for samples, labels in self.train_set:
    . . .
    # samples will hold N samples of your dataset where N is the batchsize

If you need more explanation, take a look a pytorchs documentation on this topic.


Instruction: How can I install Pytorch version 1.1 on Windows x64?

Input:
I need to install an older version of Pytorch, version 1.1, on my Windows 10 x64 machine.
The instructions here tell me to download the wheel and install it. However, pip refuses to install this wheel, claiming it is not supported on my platform. I suppose that is because, judging by the the name of the wheel (torch-1.1.0-cp37-cp37m-win_amd64.whl) it is meant for AMD, and I'm on an Intel i7. I found no better matching wheel on the Pytorch site.
What is the easiest way to install Pytorch 1.1 on my machine?

Output:
You can use conda
conda install pytorch=1.1.0

I've checked the exisiting versions with conda search -f pytorch and 1.1.0 (and many others too) is available.


Instruction: How to calculate KL Divergence between two batches of distributions in Pytroch?

Input:
Given a batch of distributions, which represented as a pytorch tensor:
A = torch.tensor([[0., 0., 0., 0., 1., 5., 1., 2.],
        [0., 0., 1., 0., 4., 2., 1., 1.],
        [0., 0., 1., 1., 0., 5., 1., 1.],
        [0., 1., 1., 0., 2., 3., 1., 1.],
        [0., 0., 2., 1., 3., 1., 1., 0.],
        [0., 0., 2., 0., 5., 0., 1., 0.],
        [0., 2., 1., 4., 0., 0., 1., 0.],
        [0., 0., 2., 4., 1., 0., 1., 0.]], device='cuda:0')

A is a batch of distributions, which consists of eight distributions.
Now, given another batch of distributions B:
B = torch.tensor([[0., 0., 1., 4., 2., 1., 1., 0.],
        [0., 0., 0., 5., 1., 2., 1., 0.],
        [0., 0., 0., 4., 2., 3., 0., 0.],
        [0., 0., 1., 7., 0., 0., 1., 0.],
        [0., 0., 1., 2., 4., 0., 1., 1.],
        [0., 0., 1., 3., 1., 3., 0., 0.],
        [0., 0., 1., 4., 1., 0., 2., 0.],
        [1., 0., 1., 5., 0., 1., 0., 0.],
        [0., 1., 5., 1., 0., 0., 1., 0.],
        [0., 0., 3., 2., 2., 0., 1., 0.],
        [0., 2., 4., 0., 1., 0., 1., 0.],
        [1., 0., 4., 1., 1., 1., 0., 0.]], device='cuda:0')

B has 12 distributions. I want to calculate the KL Divergence between each distribution in A and each distribution in B, and then obtain a KL Distance Matrix, of which shape is 12*8. I know to use loop structure and torch.nn.functional.kl_div() to reach it. Is there any other methods in pytorch to implement it without using for-loop?
Here is my implementation using for-loop:
p_1 = F.softmax(A, dim = -1)
p_2 = F.softmax(B, dim = -1)
C = torch.empty(size = (A.shape[0], B.shape[0]), dtype = torch.float)

for i,a in enumerate(p_1):
    for j,b in enumerate(p_2):
        C[i][j] =  torch.nn.functional.kl_div(a.log(), b)
print(C)

Output is :
tensor([[0.4704, 0.5431, 0.3422, 0.6284, 0.3985, 0.2003, 0.4925, 0.5739, 0.5793,
         0.3992, 0.5007, 0.4934],
        [0.3416, 0.4518, 0.2950, 0.5263, 0.0218, 0.2254, 0.3786, 0.4747, 0.3626,
         0.1823, 0.2960, 0.2937],
        [0.3845, 0.4306, 0.2722, 0.5022, 0.4769, 0.1500, 0.3964, 0.4556, 0.4609,
         0.3396, 0.4076, 0.3933],
        [0.2862, 0.3752, 0.2116, 0.4520, 0.1307, 0.1116, 0.3102, 0.3990, 0.2869,
         0.1464, 0.2164, 0.2225],
        [0.1829, 0.2674, 0.1763, 0.3227, 0.0244, 0.1481, 0.2067, 0.2809, 0.1675,
         0.0482, 0.1271, 0.1210],
        [0.4359, 0.5615, 0.4427, 0.6268, 0.0325, 0.4160, 0.4749, 0.5774, 0.3492,
         0.2093, 0.3015, 0.3014],
        [0.0235, 0.0184, 0.0772, 0.0286, 0.3462, 0.1461, 0.0142, 0.0162, 0.3524,
         0.1824, 0.2844, 0.2988],
        [0.0097, 0.0171, 0.0680, 0.0284, 0.2517, 0.1374, 0.0082, 0.0148, 0.2403,
         0.1058, 0.2100, 0.1978]], device='cuda:0')

Output:
Looking at nn.KLDivLoss, the formula for computing the KL divergence is
kl = torch.mean(b * (torch.log(b) - a))

We can use broadcasting to compute the KL efficiently:
# avoid NaNs from log(0)
lB = B.clone()
lB[B==0] = 1.

# do the computation efficiently
C = (B[None, ...] * (torch.log(lB[None, ...]) - A[:, None, :])).mean(dim=-1)

Coming to think of it, I'm not sure what you are asking makes much sense. Your A and B tensors are filled with numbers, but they do not represent distributions (they do not sum to 1). Please consider carefully what you are trying to do here.


Instruction: LSTM for time-series prediction failing to learn (PyTorch)

Input:
I'm currently working on building an LSTM network to forecast time-series data using PyTorch. I tried to share all the code pieces that I thought would be helpful, but please feel free to let me know if there's anything further I can provide. I added some comments at the end of the post regarding what the underlying issue might be.
From the univariate time-series data indexed by date, I created 3 date features and split the data into training and validation sets as below.
# X_train
             weekday    monthday    hour
timestamp
2015-01-08 17:00:00 3   8   17
2015-01-12 19:30:00 0   12  19
2014-12-01 15:30:00 0   1   15
2014-07-26 09:00:00 5   26  9
2014-10-17 20:30:00 4   17  20
... ... ... ...
2014-08-29 06:30:00 4   29  6
2014-10-13 14:30:00 0   13  14
2015-01-03 02:00:00 5   3   2
2014-12-06 16:00:00 5   6   16
2015-01-06 20:30:00 1   6   20
8256 rows × 3 columns

# y_train
                    value
timestamp
2015-01-08 17:00:00 17871
2015-01-12 19:30:00 20321
2014-12-01 15:30:00 16870
2014-07-26 09:00:00 11209
2014-10-17 20:30:00 26144
... ...
2014-08-29 06:30:00 9008
2014-10-13 14:30:00 17698
2015-01-03 02:00:00 12850
2014-12-06 16:00:00 18277
2015-01-06 20:30:00 19640
8256 rows × 1 columns

# X_val
             weekday    monthday    hour
timestamp
2015-01-08 07:00:00 3   8   7
2014-10-13 22:00:00 0   13  22
2014-12-07 01:30:00 6   7   1
2014-10-14 17:30:00 1   14  17
2014-10-25 09:30:00 5   25  9
... ... ... ...
2014-09-26 12:30:00 4   26  12
2014-10-08 16:00:00 2   8   16
2014-12-03 01:30:00 2   3   1
2014-09-11 08:00:00 3   11  8
2015-01-15 10:00:00 3   15  10
2064 rows × 3 columns

# y_val
                    value
timestamp
2014-09-13 13:00:00 21345
2014-10-28 20:30:00 23210
2015-01-21 17:00:00 17001
2014-07-20 10:30:00 13936
2015-01-29 02:00:00 3604
... ...
2014-11-17 11:00:00 15247
2015-01-14 00:00:00 10584
2014-09-02 13:00:00 17698
2014-08-31 13:00:00 16652
2014-08-30 12:30:00 15775
2064 rows × 1 columns

Then, I transformed the values in the datasets by using MinMaxScaler from the sklearn library.
scaler = MinMaxScaler()
X_train_arr = scaler.fit_transform(X_train)
X_val_arr = scaler.transform(X_val)
y_train_arr = scaler.fit_transform(y_train)
y_val_arr = scaler.transform(y_val)

After converting these NumPy arrays into PyTorch Tensors, I created iterable datasets using TensorDataset and DataLoader classes provided by PyTorch.
from torch.utils.data import TensorDataset, DataLoader
from torch.autograd import Variable

train_features = torch.Tensor(X_train_arr)
train_targets = torch.Tensor(y_train_arr)

val_features = torch.Tensor(X_val_arr)
val_targets = torch.Tensor(y_val_arr)

train = TensorDataset(train_features, train_targets)
train_loader = DataLoader(train, batch_size=64, shuffle=False)

val = TensorDataset(val_features, val_targets)
val_loader = DataLoader(train, batch_size=64, shuffle=False)

Then, I defined my LSTM Model and train_step functions as follows:
class LSTMModel(nn.Module):
    def __init__(self, input_dim, hidden_dim, layer_dim, output_dim):
        super(LSTMModel, self).__init__()
        # Hidden dimensions
        self.hidden_dim = hidden_dim

        # Number of hidden layers
        self.layer_dim = layer_dim

        # Building your LSTM
        # batch_first=True causes input/output tensors to be of shape
        # (batch_dim, seq_dim, feature_dim)
        self.lstm = nn.LSTM(input_dim, hidden_dim, layer_dim, batch_first=True)

        # Readout layer
        self.fc = nn.Linear(hidden_dim, output_dim)

    def forward(self, x):
        # Initialize hidden state with zeros
        h0 = torch.zeros(self.layer_dim, x.size(0), self.hidden_dim).requires_grad_()

        # Initialize cell state
        c0 = torch.zeros(self.layer_dim, x.size(0), self.hidden_dim).requires_grad_()

        # We need to detach as we are doing truncated backpropagation through time (BPTT)
        # If we don't, we'll backprop all the way to the start even after going through another batch
        out, (hn, cn) = self.lstm(x, (h0.detach(), c0.detach()))

        # Index hidden state of last time step
        out = self.fc(out[:, -1, :])
        return out

def make_train_step(model, loss_fn, optimizer):
    # Builds function that performs a step in the train loop
    def train_step(x, y):
        # Sets model to TRAIN mode
        model.train()
        # Makes predictions
        yhat = model(x)
        # Computes loss
        loss = loss_fn(y, yhat)
        # Computes gradients
        loss.backward()
        # Updates parameters and zeroes gradients
        optimizer.step()
        optimizer.zero_grad()
        # Returns the loss
        return loss.item()

    # Returns the function that will be called inside the train loop
    return train_step

Finally, I start training my LSTM model in mini-batches with AdamOptimizer for 20 epochs, which is already long enough to see the model is not learning.
import torch.optim as optim

input_dim = n_features
hidden_dim = 64
layer_dim = 3
output_dim = 1

model = LSTMModel(input_dim, hidden_dim, layer_dim, output_dim)

criterion = nn.MSELoss(reduction='mean')
optimizer = optim.Adam(model.parameters(), lr=1e-2)

train_losses = []
val_losses = []
train_step = make_train_step(model, criterion, optimizer)
n_epochs = 20
device = 'cuda' if torch.cuda.is_available() else 'cpu'

for epoch in range(n_epochs):
    batch_losses = []
    for x_batch, y_batch in train_loader:
        x_batch = x_batch.unsqueeze(dim=0).to(device)
        y_batch = y_batch.to(device)
        loss = train_step(x_batch, y_batch)
        batch_losses.append(loss)
    training_loss = np.mean(batch_losses)
    train_losses.append(training_loss)
    with torch.no_grad():
        batch_val_losses = []
        for x_val, y_val in val_loader:
            x_val = x_val.unsqueeze(dim=0).to(device)
            y_val = y_val.to(device)
            model.eval()
            yhat = model(x_val)
            val_loss = criterion(y_val, yhat).item()
            batch_val_losses.append(val_loss)
        validation_loss = np.mean(batch_val_losses)
        val_losses.append(validation_loss)

    print(f"[{epoch+1}] Training loss: {training_loss:.4f}\t Validation loss: {validation_loss:.4f}")

And this is the output:
C:\Users\VS32XI\Anaconda3\lib\site-packages\torch\nn\modules\loss.py:446: UserWarning: Using a target size (torch.Size([1, 1])) that is different to the input size (torch.Size([64, 1])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
  return F.mse_loss(input, target, reduction=self.reduction)
[1] Training loss: 0.0505    Validation loss: 0.0315
[2] Training loss: 0.0317    Validation loss: 0.0315
[3] Training loss: 0.0317    Validation loss: 0.0315
[4] Training loss: 0.0317    Validation loss: 0.0315
[5] Training loss: 0.0317    Validation loss: 0.0315
[6] Training loss: 0.0317    Validation loss: 0.0315
[7] Training loss: 0.0317    Validation loss: 0.0315
[8] Training loss: 0.0317    Validation loss: 0.0315
[9] Training loss: 0.0317    Validation loss: 0.0315
[10] Training loss: 0.0317   Validation loss: 0.0315
[11] Training loss: 0.0317   Validation loss: 0.0315
[12] Training loss: 0.0317   Validation loss: 0.0315
[13] Training loss: 0.0317   Validation loss: 0.0315
[14] Training loss: 0.0317   Validation loss: 0.0315
[15] Training loss: 0.0317   Validation loss: 0.0315
[16] Training loss: 0.0317   Validation loss: 0.0315
[17] Training loss: 0.0317   Validation loss: 0.0315
[18] Training loss: 0.0317   Validation loss: 0.0315
[19] Training loss: 0.0317   Validation loss: 0.0315
[20] Training loss: 0.0317   Validation loss: 0.0315

Note 1: Looking at the warning given, I'm not sure if that's the real reason why the model is not learning well. After all, I'm trying to predict the future values in the time-series data; therefore, 1 would be a plausible output dimension.
Note 2: To train the model in mini-batches, I relied on the class DataLoader. When iterating over the X and Y batches in both train and validation DataLoaders, the dimensions of x_batches were 2, while the model expected 3. So, I used PyTorch's unsqueeze function to match the expected dimension as in x_batch.unsqueeze(dim=0) . I'm not sure if this is how I should have gone about it, which could also be the issue.

Output:
Once I used Tensor View to reshape the mini-batches for the features in training and the validation set, the issue was resolved. As a side note, view() enable fast and memory-efficient reshaping, slicing, and element-wise operations, by avoiding an explicit data copy.
It turned out that in the earlier implementation torch.unsqueeze() did not reshape the batches into tensors with the dimensions (batch size, timesteps, number of features). Instead, the function unsqueeze(dim=0) returns a new tensor with a singleton dimension inserted at the Oth index.
So, the mini batches for the feature sets is shaped as follows x_batch = x_batch.view([batch_size, -1, n_features]).to(device)
Then, the new training loop becomes:
for epoch in range(n_epochs):
    batch_losses = []
    for x_batch, y_batch in train_loader:
        x_batch = x_batch.view([batch_size, -1, n_features]).to(device) # <---
        y_batch = y_batch.to(device)
        loss = train_step(x_batch, y_batch)
        batch_losses.append(loss)
    training_loss = np.mean(batch_losses)
    train_losses.append(training_loss)
    with torch.no_grad():
        batch_val_losses = []
        for x_val, y_val in val_loader:
            x_val = x_val.view([batch_size, -1, n_features]).to(device) # <---
            y_val = y_val.to(device)
            model.eval()
            yhat = model(x_val)
            val_loss = criterion(y_val, yhat).item()
            batch_val_losses.append(val_loss)
        validation_loss = np.mean(batch_val_losses)
        val_losses.append(validation_loss)

    print(f"[{epoch+1}] Training loss: {training_loss:.4f}\t Validation loss: {validation_loss:.4f}")

Here's the output:
[1] Training loss: 0.0235    Validation loss: 0.0173
[2] Training loss: 0.0149    Validation loss: 0.0086
[3] Training loss: 0.0083    Validation loss: 0.0074
[4] Training loss: 0.0079    Validation loss: 0.0069
[5] Training loss: 0.0076    Validation loss: 0.0069

                          ...

[96] Training loss: 0.0025   Validation loss: 0.0028
[97] Training loss: 0.0024   Validation loss: 0.0027
[98] Training loss: 0.0027   Validation loss: 0.0033
[99] Training loss: 0.0027   Validation loss: 0.0030
[100] Training loss: 0.0023  Validation loss: 0.0028

Recently, I've decided to put together the things I had learned and the things I would have liked to know earlier. If you'd like to have a look, you can find the links down below. I hope you'll find it useful. Feel free to comment or reach out to me if you agree or disagree with any of the remarks I made above.

Building RNN, LSTM, and GRU for time series using PyTorch
Predicting future values with RNN, LSTM, and GRU using PyTorch


Instruction: mask list/tensor with multiple conditions?

Input:
The following code masks fine
mask = targets >= 0
targets = targets[mask]

However, when I try masking with two conditions, it gives an error of RuntimeError: Boolean value of Tensor with more than one value is ambiguous
mask = (targets >= 0 and targets <= 5)
targets = targets[mask]

is there a way to do this?

Output:
You are making a mistake while using brackets. Bracket around each of the conditions so that NumPy considers them as individual arrays.
targets = np.random.randint(0,10,(10,))

mask = (targets>=0) & (targets<=5) #<----------

print(mask)
targets[mask]

[ True False  True False False  True  True  True False  True]
array([4, 1, 3, 1, 5, 3])

You can create some complex logic using multiple masks and then directly index an array with them. Example - XNOR can be written as ~(mask1 ^ mask2)


Instruction: Is there a way to force any function to not be verbose in Python?

Input:
While loading a pretrained model with pytorch via
model.load_state_dict(torch.load(MODEL_PATH))

the console is flooded with output containing information about the model (which is annoying). Afaik there is no verbosity option, neither in model.load_state_dict nor in torch.load. Please let me know if I overlooked this parameter.
However, this led me to the question if there is any general way to force a function to be non verbose. Maybe something like:
with os.nonverbose():
   model.load_state_dict(torch.load(MODEL_PATH))

Any ideas?

Output:
As @triplee commented, most libraries would use Python logging, which can be modified extensively. I haven't worked with PyTorch before but from https://github.com/pytorch/vision/issues/330 it looks like it's actaully using print (which is horrible if it is but okay).
In general, you can, however, suppress the stdout output of anything by redirecting stdout. Here's a good link https://wrongsideofmemphis.com/2010/03/01/store-standard-output-on-a-variable-in-python/
In your question you ask if this can be done with a context manager. I don't see why not and it seems appropriate given that you want to reset stdout after the function call. Something like this:
from io import StringIO  # Python3

import sys

class SilencedStdOut:
    def __enter__(self):
        self.old_stdout = sys.stdout
        self.result = StringIO()
        sys.stdout = self.result

    def __exit__(self, *args, **kwargs):

        sys.stdout = self.old_stdout
        result_string = self.result.getvalue() # use if you want or discard.

If you only ever want to supress a single function and never a block of code, however, then a decorator should work fine too.


Instruction: Overriding stem method in video classification model to change filter channel

Input:
I am trying to use torchvision’s video classification models (R3D, R(2+1)D, MC18) but my data is single channel (grey scale video), and these model uses 3 channel input, in that case I am trying to override the stem class , can someone please confirm if what I am doing is correct?
For R3D18 and MC18  stem=BasicStem
class BasicStemModified(nn.Sequential):

    def __init__(self):
        super(BasicStemModified, self).__init__(
            nn.Conv3d(1, 45, kernel_size=(7, 7, 1),  #changing filter to 1 channel input
                      stride=(2, 2, 1), padding=(3, 3, 0),
                      bias=False),
            nn.BatchNorm3d(45),
            nn.ReLU(inplace=True),

            nn.Conv3d(45, 64, kernel_size=(1, 1, 3),
                      stride=(1, 1, 1), padding=(0, 0, 1),
                      bias=False),
            nn.BatchNorm3d(64),
            nn.ReLU(inplace=True))

model = torchvision.models.video.mc3_18(pretrained=False)

model.stem = BasicStemModified() #here assigning the modified stem

model.fc = nn.Sequential(
    nn.Dropout(0.3),
    nn.Linear(model.fc.in_features, num_classes)
)

model.to('cuda:0')

For R(2+1)D:
#For R(2+1)D model `stem=R2Plus1dStem`

class R2Plus1dStemModified(nn.Sequential):
    """R(2+1)D stem is different than the default one as it uses separated 3D convolution
    """
    def __init__(self):
        super(R2Plus1dStemModified, self).__init__(
            nn.Conv3d(3, 45, kernel_size=(1, 7, 7),   #changing filter to 1 channel input
                      stride=(1, 2, 2), padding=(0, 3, 3),
                      bias=False),
            nn.BatchNorm3d(45),
            nn.ReLU(inplace=True),
            nn.Conv3d(45, 64, kernel_size=(3, 1, 1),
                      stride=(1, 1, 1), padding=(1, 0, 0),
                      bias=False),
            nn.BatchNorm3d(64),
            nn.ReLU(inplace=True))

model = torchvision.models.video.mc3_18(pretrained=False)

model.stem = R2Plus1dStemModified() #here assigning the modified stem

model.fc = nn.Sequential(
    nn.Dropout(0.3),
    nn.Linear(model.fc.in_features, num_classes)
)

model.to('cuda:0')

Output:
When switching from RGB to gray, the most simple way to go is to change the DATA and not the model:
If you have an input frame with only one channel (gray), you can simply expand the singleton channel dimension to span three channels. This is trivial and allows you to use  pre-trained models as-is.

If you insist on modifying the model - you can do so while preserving most of the pre-trained weights:
model = torchvision.models.video.mc3_18(pretrained=True)  # get the pretrained
# modify only the first conv layer
origc = model.stem[0]  # the orig conv layer
# build a new layer only with one input channel
c1 = torch.nn.Conv3d(1, origc.out_channels, kernel_size=origc.kernel_size, stride=origc.stride, padding=origc.padding, bias=origc.bias)

# this is the nice part - init the new weights using the original ones
with torch.no_grad():
  c1.weight.data = origc.weight.data.sum(dim=1, keepdim=True)


Instruction: Why does torchvision.utils.make_grid() return copies of the wanted grid?

Input:
In the below coding example I can not understand why the output tensor , grid has a shape of
3,28,280. I understand why its 28 in height and 280 in width, but not the 3. It seems from running plt.imshow() on all 3 28x280 arrays along axis 0 that they are identical copies since printing any 1 of these gives me the image I want.
Also I do not understand why I can pass grid as an argument to plt.imshow() given that it is supposed to take in a 2D array, not a 3D one as grid clearly is.
import torch
import torchvision
import torchvision.transforms as transforms
import matplotlib.pyplot as plt
import numpy as np

train_set = torchvision.datasets.FashionMNIST(
    root = './pytorch_obj_classifier/data/FashionMNIST',
    train = True,
    download = True,
    transform = transforms.Compose([
            transforms.ToTensor()
    ])
)
sample = next(iter(train_loader))
image,label = sample
print(image.shape)

grid = torchvision.utils.make_grid(image,padding=0, nrow=10)
print(grid.shape)

plt.figure(figsize=(15,15))
grid = np.transpose(grid,(1,2,0))
grid1 = grid[:,:,0]
grid2 = grid[:,:,1]
grid3 = grid[:,:,2]
plt.imshow(grid1,cmap = 'gray')
plt.imshow(grid2,cmap = 'gray')
plt.imshow(grid3,cmap = 'gray')
plt.imshow(grid,cmap = 'gray')

Output:
The MNIST dataset consists of grascale images. If you look at the implementation detail of torchvision.utils.make_grid, single-channel images get their channel copied three times:
if tensor.dim() == 4 and tensor.size(1) == 1:  # single-channel images
    tensor = torch.cat((tensor, tensor, tensor), 1)

As for matplotlib.pyplot.imshow it can take 2D, 3D or 4D inputs:

The image data. Supported array shapes are:

(M, N): an image with scalar data. The data is visualized using a colormap.
(M, N, 3): an image with RGB values (0-1 float or 0-255 int).
(M, N, 4): an image with RGBA values (0-1 float or 0-255 int), i.e. including transparency.

Generally speaking, we wouldn't refer to dimensions but rather describe tensors by their shape (the size on each of their axes). In PyTorch, images always have three axes, and have a shape that follows: (channel, height, width). Even for single-channel images: considering it as a 3D tensor (1, height, width) instead of a 2D tensor (height, width). This is to be consistant with cases where you have more than one channel, which is very often (cf. convolution neural networks).


Instruction: How to create a tensor of given shape and interval?

Input:
I am going through Pytorch and want to create a random tensor of shape 5X3 in the interval [3,7)
torch.rand(5,3) will return a random tensor of shape 5 X 3, however, I could not figure to set the given interval.
Please guide.

Output:
You can map U ~ [0, 1] to U ~ [a, b] with u -> (a - b)*u + b:
(a - b)*torch.rand(5, 3) + b


Instruction: How to vectorize indexing and computation when indexed tensors are different dimensions?

Input:
I'm trying to vectorize the following for-loop in Pytorch. I'd be happy with just vectorizing the inner for-loop, but doing the whole batch would also be awesome.
# B: the batch size
# N: the number of training examples
# dim: the dimension of each feature vector
# K: the number of discrete labels. each vector has a single label
# delta: margin for hinge loss

batch_data = torch.tensor(...)  # Tensor of shape [B x N x d]
batch_labels = torch.tensor(...)  # Tensor of shape [B x N x 1], each element is one of K labels (ints)

batch_losses = []  # Ultimately should be [B x 1]
batch_centroids = []  # Ultimately should be [B x K_i x dim]
for i in range(B):
    centroids = []  # Keep track of the means for each class.
    classes = torch.unique(labels)  # Get the unique labels for the classes.

    # NOTE: The number of classes K for each item in the batch might actually
    # be different. This may complicate batch-level operations.

    total_loss = 0

    # For each class independently. This is the part I want to vectorize.
    for cl in classes:
        # Take the subset of training examples with that label.
        subset = data[torch.where(labels == cl)]

        # Find the centroid of that subset.
        centroid = subset.mean(dim=0)
        centroids.append(centroid)

        # Get the distance between each point in the subset and the centroid.
        dists = subset - centroid
        norm = torch.linalg.norm(dists, dim=1)

        # The loss is the mean of the hinge loss across the subset.
        margin = norm - delta
        hinge = torch.clamp(margin, min=0.0) ** 2

        total_loss += hinge.mean()

    # Keep track of everything. If it's too hard to keep track of centroids, that's also OK.
    loss = total_loss.mean()
    batch_losses.append(loss)
    batch_centroids.append(centroids)

I've been scratching my head on how to deal with the irregularly sized tensors. The number of classes in each batch K_i is different, and the size of each subset is different.

Output:
It turns out it actually is possible to vectorize across ragged arrays. I'll use numpy, but code should be directly translatable to torch. The key technique is to:

Sort by ragged array membership
Perform an accumulation
Find boundary indices, compute adjacent differences

For a single (non-batch) input of an n x d matrix X and an n-length array label, the following returns the k x d centroids and n-length distances to respective centroids:
def vcentroids(X, label):
    """
    Vectorized version of centroids.
    """
    # order points by cluster label
    ix = np.argsort(label)
    label = label[ix]
    Xz = X[ix]

    # compute pos where pos[i]:pos[i+1] is span of cluster i
    d = np.diff(label, prepend=0) # binary mask where labels change
    pos = np.flatnonzero(d) # indices where labels change
    pos = np.repeat(pos, d[pos]) # repeat for 0-length clusters
    pos = np.append(np.insert(pos, 0, 0), len(X))

    Xz = np.concatenate((np.zeros_like(Xz[0:1]), Xz), axis=0)
    Xsums = np.cumsum(Xz, axis=0)
    Xsums = np.diff(Xsums[pos], axis=0)
    counts = np.diff(pos)
    c = Xsums / np.maximum(counts, 1)[:, np.newaxis]

    repeated_centroids = np.repeat(c, counts, axis=0)
    aligned_centroids = repeated_centroids[inverse_permutation(ix)]
    dist = np.sum((X - aligned_centroids) ** 2, axis=1)

    return c, dist

Batching requires little special handling. For an input B x n x d array batch_X, with B x n batch labels batch_labels, create unique labels for each batch:
batch_k = batch_labels.max(axis=1) + 1
batch_k[1:] = batch_k[:-1]
batch_k[0] = 0
base = np.cumsum(batch_k)
batch_labels += base.expand_dims(1)

So now each batch element has a unique contiguous range of labels. I.e., the first batch element will have n labels in some range [0, k0) where k0 = batch_k[0], the second will have range [k0, k0 + k1) where k1 = batch_k[1], etc.
Then just flatten the n x B x d input to n*B x d and call the same vectorized method. Your loss function is derivable using the final distances and same position-array based reduction technique.
For a detailed explanation of how the vectorization works, see my blog post.


Instruction: PyTorch convolutional block - CIFAR10 - RuntimeError

Input:
I am using PyTorch 1.7 and Python 3.8 with CIFAR-10 dataset. I am trying to create a block with: conv -> conv -> pool -> fc. Fully connected layer (fc) has 256 neurons. The code for this is as follows:
# Testing-
conv1 = nn.Conv2d(
    in_channels = 3, out_channels = 64,
    kernel_size = 3, stride = 1,
    padding = 1, bias = True
    )
conv2 = nn.Conv2d(
    in_channels = 64, out_channels = 64,
    kernel_size = 3, stride = 1,
    padding = 1, bias = True
    )
pool = nn.MaxPool2d(
    kernel_size = 2, stride = 2
    )
fc1 = nn.Linear(
    in_features = 64 * 16 * 16, out_features = 256
    bias = True
)

images.shape
# torch.Size([32, 3, 32, 32])

x = conv1(images)
x.shape
# torch.Size([32, 64, 32, 32])

x = conv2(x)
x.shape
# torch.Size([32, 64, 32, 32])

x = pool(x)
x.shape
# torch.Size([32, 64, 16, 16])

# This line of code gives error-
x = fc1(x)

RuntimeError: mat1 and mat2 shapes cannot be multiplied (32768x16 and
16384x256)

What is going wrong?

Output:
You are nearly there! As you will have noticed nn.MaxPool returns a shape (32, 64, 16, 16) which is incompatible with a nn.Linear's input: a 2D dimensional tensor (batch, in_features). You need to broadcast to (batch, 64*16*16).
I would recommend using a nn.Flatten layer rather than broadcasting yourself. It will act as x.view(x.size(0), -1) but is clearer. By default it preserves the first dimension:
conv1 = nn.Conv2d(in_channels=3, out_channels=64, kernel_size=3, stride=1, padding=1)
conv2 = nn.Conv2d(in_channels=64, out_channels=64, kernel_size=3, stride=1, padding=1)
pool = nn.MaxPool2d(kernel_size=2, stride=2)
flatten = nn.Flatten()
fc1 = nn.Linear(in_features=64*16*16, out_features=256)

x = conv1(images)
x = conv2(x)
x = pool(x)
x = flatten(x)
x = fc1(x)

Alternatively, you could use the functional alternative torch.flatten, where you will have to provide the start_dim as 1: x = torch.flatten(x, start_dim=1).

When you're done debugging, you could assemble your layers with nn.Sequential:
model = nn.Sequential(
    nn.Conv2d(in_channels=3, out_channels=64, kernel_size=3, stride=1, padding=1),
    nn.Conv2d(in_channels=64, out_channels=64, kernel_size=3, stride=1, padding=1),
    nn.MaxPool2d(kernel_size=2, stride=2),
    nn.Flatten(),
    nn.Linear(in_features=64*16*16, out_features=256)
)

x = model(images)


Instruction: How to find the (Most important) responsible Words/ Tokens/ embeddings responsible for the label result of a text classification model in PyTorch

Input:
Let us suppose I have a model like:
class BERT_Subject_Classifier(nn.Module):

    def __init__(self,out_classes,hidden1=128,hidden2=32,dropout_val=0.2):
      super(BERT_Subject_Classifier, self).__init__()

      self.hidden1 = hidden1
      self.hidden2 = hidden2
      self.dropout_val = dropout_val
      self.logits = logit
      self.bert = AutoModel.from_pretrained('bert-base-uncased')
      self.out_classes = out_classes
      self.unfreeze_n = unfreeze_n # make the last n layers trainable

      self.dropout = nn.Dropout(self.dropout_val)
      self.relu =  nn.ReLU()
      self.fc1 = nn.Linear(768,self.hidden1)
      self.fc2 = nn.Linear(self.hidden1,self.hidden2)
      self.fc3 = nn.Linear(self.hidden2,self.out_classes)

    def forward(self, sent_id, mask):
      _, cls_hs = self.bert(sent_id, attention_mask=mask)
      x = self.fc1(cls_hs)
      x = self.relu(x)
      x = self.dropout(x)
      x = self.fc2(x)
      x = self.dropout(x)
      return self.fc3(x)

I train my model and for a new data point x = ['My Name is Slim Shady'], I get my label result as  3.
My Question is that how can I check which of the words in the sentence were responsible for the the classification? I mean it could be any collection of words. Is there a library or way to check the functionality? Just like shown in the paper and Tensorflow Implementation of show Attend and Tell, you can get the areas of images where the model is paying attention to. How can I do it for the Text?

Output:
Absolutely. One way to demonstrate which words have the greatest impact is through integrated gradients methods. For PyTorch, one package you can use is Captum. I would check out this page for a good example: https://captum.ai/tutorials/IMDB_TorchText_Interpret
For Tensorflow, one package that you can use is Seldon. I would check out this page for a good example:
https://docs.seldon.io/projects/alibi/en/stable/examples/integrated_gradients_imdb.html


Instruction: TypeError: hook() takes 3 positional arguments but 4 were given

Input:
I am trying to extract features from an intermediate of ResNet18 in PyTorch using forward hooks
class CCLModel(nn.Module):
    def __init__(self,output_layer,*args):
        self.output_layer = output_layer
        super().__init__(*args)

        self.output_layer = output_layer
        #PRETRAINED MODEL
        self.pretrained = models.resnet18(pretrained=True)

        #TAKING OUTPUT FROM AN INTERMEDIATE LAYER

        #self._layers = []
        for l in list(self.pretrained._modules.keys()):
            #self._layers.append(l)
            if l == self.output_layer:
                handle = getattr(self.pretrained,l).register_forward_hook(self.hook)

    def hook(self,input,output):
        return output

    def _forward_impl(self, x):
        x = self.pretrained(x)
        return x

    def forward(self, x):
        return self._forward_impl(x)

I also want the predictions alongside the feature outputs from layer 4
But I am getting the TypeError: hook() takes 3 positional arguments but 4 were given
The full error message is this
TypeError                                 Traceback (most recent call last)
<ipython-input-66-18c4a0f917f2> in <module>()
----> 1 out = model(x.to('cuda:0').float())

6 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

<ipython-input-61-71fe0d1420a6> in forward(self, x)
     78
     79     def forward(self, x):
---> 80         return self._forward_impl(x)
     81
     82     '''def forward(self,x):

<ipython-input-61-71fe0d1420a6> in _forward_impl(self, x)
     73         #x = torch.flatten(x, 1)
     74         #x = self.fc(x)
---> 75         x = self.pretrained(x)
     76
     77         return x

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

/usr/local/lib/python3.6/dist-packages/torchvision/models/resnet.py in forward(self, x)
    218
    219     def forward(self, x):
--> 220         return self._forward_impl(x)
    221
    222

/usr/local/lib/python3.6/dist-packages/torchvision/models/resnet.py in _forward_impl(self, x)
    209         x = self.layer2(x)
    210         x = self.layer3(x)
--> 211         x = self.layer4(x)
    212
    213         x = self.avgpool(x)

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    729                 _global_forward_hooks.values(),
    730                 self._forward_hooks.values()):
--> 731             hook_result = hook(self, input, result)
    732             if hook_result is not None:
    733                 result = hook_result

TypeError: hook() takes 3 positional arguments but 4 were given

Why is the hook not working, although in various forums I see that this is the way to do it?

Output:
Here's a simple example of a forward hook, it must have three parameters model, input, and output:
m = models.resnet18(pretrained=False)

def hook(module, input, output):
    print(output.detach().shape)

m.fc.register_forward_hook(hook)

Try it with dummy data:
>>> m(torch.rand(1, 3, 224, 224))
torch.Size([1, 1000])
<<< tensor(...)

To combine it with your nn.Module, you need to implement hook with an extra argument self:
class CCLModel(nn.Module):
    def __init__(self, output_layer, *args):
        super(CCLModel, self).__init__()

        self.pretrained = models.resnet18(pretrained=True)
        self.output_layer = output_layer

        self.output_layer.register_forward_hook(self.hook)

    def hook(self, module, input, output):
        return print(output.shape)

    def forward(self, x):
        x = self.pretrained(x)
        x = self.output_layer(x)
        return x

Note - self corresponds to the CCLModel instance while model is the layer we're hooked on i.e. nn.Linear
Here's an example:
>>> m = CCLModel(nn.Linear(1000, 100))
>>> m(torch.rand(1, 3, 224, 224))
torch.Size([1, 100])
<<< tensor(...)


Instruction: Custom small CNN has better accuracy than the pretrained classifiers

Input:
I have a dataset of laser welding images of size 300*300 which contains two class of bad and good weld seam. I have followed Pytorch fine-tuning tutorial for an inception-v3 classifier.
on the other hand, I also build a custom CNN with 3 conv layer and 3 fc. What I observed is that the fine tuning showed lots of variation on validation accuracy. basically, I see different maximum accuracy every time I train my model. Plus, my accuracy in fine-tuning is much less than my custom CNN!! for example the accuracy for my synthetic images from a GAN is 86% with inception-v3, while it is 94% with my custom CNN. The real data for both network shows almost similar behaviour and accuracy, however accuracy in custom CNN is about 2% more.
I trained with different training scales of 200, 500 and 1000 train-set images (half of them for each class like for 200 images we have 100 good and 100 bad). I also include a resize transform of 224 in my train_loader; in fine tuning tutorial, this resize is automatically done to 299 for inception-v3. for each trial, the validation-size and its content is constant.
Do you know what cause this behavior? Is it because my dataset is so different from the pretrained model classes? am I not supposed to get better results with fine-tuning?
My custom CNN:
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv3 = nn.Conv2d(16, 24, 5)
        self.fc1 = nn.Linear(13824, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 2)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = self.pool(F.relu(self.conv3(x)))
        #x = x.view(-1, 16 * 5 * 5)
        x = x.view(x.size(0),-1)
        #print(x.shape)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        #x = F.softmax(x, dim=1)
        return x

model = Net()
criterion = nn.CrossEntropyLoss()
#optimizer = optim.Adam(model.parameters(), lr=0.001)
optimizer = optim.SGD(model.parameters(), lr=0.001, momentum=0.9, weight_decay=5e-4)
model.to(device)

with training loop of:
epochs = 15
steps = 0
running_loss = 0
print_every = 10
train_losses, test_losses = [], []
train_acc, test_acc = [], []
for epoch in range(epochs):
    for inputs, labels in trainloader:
        steps += 1
        inputs, labels = inputs.to(device), labels.to(device)
        optimizer.zero_grad()
        logps = model.forward(inputs)
        loss = criterion(logps, labels)
        loss.backward()
        optimizer.step()
        running_loss += loss.item()

        if steps % print_every == 0:
            test_loss = 0
            accuracy = 0
            model.eval()
            with torch.no_grad():
                for inputs, labels in testloader:
                    inputs, labels = inputs.to(device), labels.to(device)
                    logps = model.forward(inputs)
                    batch_loss = criterion(logps, labels)
                    test_loss += batch_loss.item()

                    ps = torch.exp(logps)
                    top_p, top_class = ps.topk(1, dim=1)
                    equals = top_class == labels.view(*top_class.shape)
                    accuracy += torch.mean(equals.type(torch.FloatTensor)).item()

            train_losses.append(running_loss/len(trainloader))
            test_losses.append(test_loss/len(testloader))

            #train_acc.append(running_loss/len(trainloader))
            test_acc.append(accuracy/len(testloader))
            print(f"Epoch {epoch+1}/{epochs}.. "
                  f"Train loss: {running_loss/print_every:.3f}.. "
                  f"Test loss: {test_loss/len(testloader):.3f}.. "
                  f"Test accuracy: {accuracy/len(testloader):.3f}")
            running_loss = 0
            model.train()

Output:
Here is my theory :
Pre-training is useful when you want to leverage already existing data to help the model train on similar data, for which you have few instances. At least this was the reasoning behind the Unet architecture in medical image segmentation.
Now, to me the key is in the notion of "similar". If your network have been pre-trained on cats, dogs and you want to extrapolate to weld seam there's a chance your pre-training is not helping or even getting in the way of the model training properly.
Why ?
When training your CNN you get randomly initialized weights, whereas using a pre-trained network you get pre-trainned weights. If the features your are extracting are similar across dataset then you get a head start by having the network already attuned to this features.
For example, Cats and Dogs share similar spatial features visually (eye position, nose, ears...). So there's chance that you converge to a local minima faster during training since your are already starting from a good base that just need to adapt to the new specific of your data.
Conclusions:
If the similarity assumptions does not hold it means your model would have to "unlearn" what he already learned to adapt to the new specifics of your dataset and I guess that would be the reason why training is more difficult and does not give as good result as a blank slate CNN. (especially if you don't have that much data).
PS : I'd be curious to see if your pre trained model end up catching up with your CNN if you give it more epochs to train.


Instruction: Does adding a forward hook to a layer of ensure that the gradient of the loss calculated using the layer's output will be calculated automatically?

Input:
I have a model
class NewModel(nn.Module):
    def __init__(self,output_layer,*args):
        self.output_layer = output_layer
        super().__init__(*args)

        self.output_layer = output_layer
        self.selected_out = None
        #PRETRAINED MODEL
        self.pretrained = models.resnet18(pretrained=True)

        #TAKING OUTPUT FROM AN INTERMEDIATE LAYER

        #self._layers = []
        for l in list(self.pretrained._modules.keys()):
            #self._layers.append(l)
            if l == self.output_layer:
                handle = getattr(self.pretrained,l).register_forward_hook(self.hook)

    def hook(self,module, input,output):
        self.selected_out = output

    def forward(self, x):
        return x = self.pretrained(x)

I have two target outputs, one which is same as any label of an image and the second one is the same dimensions as the output obtained from self.output_layer, called target_feature
out = model(img)
layerout = model.selected_out

Now, if I want to calculate the loss of layerout with the target feature map, can it be done like the line written below?
loss = criterion(y_true, out) + feature_criterion(layerout, target_feature)

Or do I need to add backward_hooks?
In this Kaggle notebook
https://www.kaggle.com/sironghuang/understanding-pytorch-hooks
it is written that loss.backward() cannot be used when using backward_hooks.
Quoting the author
# backprop once to get the backward hook results
out.backward(torch.tensor([1,1],dtype=torch.float),retain_graph=True)
#! loss.backward(retain_graph=True)  # doesn't work with backward hooks,
#! since it's not a network layer but an aggregated result from the outputs of last layer vs target

Then how can be gradient be calculated based on the loss function?

Output:
If I understand you correctly, you want to get two outputs from your model, calculate two losses, then combine them and backpropagate. I imagine you come from Tensorflow & Keras from the way you tried implementing it. In Pytorch, it's actually fairly straight foward, you can do this very easily because of its purely functional aspect.
This is just an example:
class NewModel(nn.Module):
    def __init__(self, output_layer, *args):
        super(MyModel, self).__init__()

        self.pretrained = models.resnet18(pretrained=True)
        self.output_layer = output_layer

    def forward(self, x):
        out = self.pretrained(x)
        features = self.output_layer(out)
        return out, features

On inference, you will get two results per call:
>>> m = NewModel(nn.Linear(1000, 10))
>>> x = torch.rand(16, 3, 224, 224)
>>> y_pred, y_feature = m(x)

Call you loss functions:
>>> loss = criterion(y_pred, y_true) + feature_criterion(y_feature, target_feature)

Then, backpropagate with loss.backward().
So no need for hooks, nor complicated gradient on your .backward call!

Edit - If you wish to extract an intermediate layer output, keep the hook, that's good. And just modify the forward definition.
def forward(self, x):
    out = self.pretrained(x)
    return out, self.selected_out

For example:
>>> m = NewModel(output_layer='layer1')
>>> x = torch.rand(16, 3, 224, 224)
>>> y_pred, y_feature = m(x)

>>> y_pred.shape, y_feature.shape
(torch.Size([16, 1000]), torch.Size([16, 64, 56, 56]))

Also, what I said above about the loss stills stands. Compute your loss, then call loss.backward().


Instruction: Torchtext TabularDataset() reads in Datafields incorrectly

Input:
Goal: I want to create a text classifier based upon my custom Dataset, simillar (and following) This (now deleted)  Tutorial from mlexplained.
What happened
I sucessfully formatted my data, created a training, validation and test dataset, and formatted it, so that it so that it equals the "toxic tweet" dataset they are using (with a column for each tag, with 1/0 for True/not True). Most of the other parts also worked just as intended, but when it came to iterating i got an Error.
The `device` argument should be set by using `torch.device` or passing a string as an argument.

This behavior will be deprecated soon and currently defaults to cpu.
The `device` argument should be set by using `torch.device` or passing a string as an argument. This behavior will be deprecated soon and currently defaults to cpu.
The `device` argument should be set by using `torch.device` or passing a string as an argument. This behavior will be deprecated soon and currently defaults to cpu.
The `device` argument should be set by using `torch.device` or passing a string as an argument. This behavior will be deprecated soon and currently defaults to cpu.
  0%|          | 0/25517 [00:01<?, ?it/s]
Traceback (most recent call last):
... (trace back messages)
AttributeError: 'Example' object has no attribute 'text'

The lines the Traceback indicated:
opt = optim.Adam(model.parameters(), lr=1e-2)
loss_func = nn.BCEWithLogitsLoss()

epochs = 2

for epoch in range(1, epochs + 1):
    running_loss = 0.0
    running_corrects = 0
    model.train() # turn on training mode
    for x, y in tqdm.tqdm(train_dl): # **THIS LINE CONTAINS THE ERROR**
        opt.zero_grad()

        preds = model(x)
        loss = loss_func(y, preds)
        loss.backward()
        opt.step()

        running_loss += loss.data[0] * x.size(0)

    epoch_loss = running_loss / len(trn)

    # calculate the validation loss for this epoch
    val_loss = 0.0
    model.eval() # turn on evaluation mode
    for x, y in valid_dl:
        preds = model(x)
        loss = loss_func(y, preds)
        val_loss += loss.data[0] * x.size(0)

    val_loss /= len(vld)
    print('Epoch: {}, Training Loss: {:.4f}, Validation Loss: {:.4f}'.format(epoch, epoch_loss, val_loss))

Attempts to solve the problems already made, and what i think was the Reson:
I know that this Problem occured to others, there are even 2 Questions to it on here, bot both had the problem of skipping either Columns or Rows in the dataset (i checked for empty lines/Cokumns, and found none). Another Solution was that the Parameters given the model had to be in the same order (with none Missing) than the parameters in the .csv File.
However, the relevant code (the loading and creating of the tst, trn and vld sets)
def createTestTrain():
    # Create a Tokenizer
    tokenize = lambda x: x.split()

    # Defining Tag and Text
    TEXT = Field(sequential=True, tokenize=tokenize, lower=True)
    LABEL = Field(sequential=False, use_vocab=False)

    # Our Datafield
    tv_datafields = [("ID", None),
                     ("text", TEXT)]

    # Loading our Additional columns we added earlier
    with open(PATH + 'columnList.pickle', 'rb') as handle:
        addColumns = pickle.load(handle)

    # Adding the extra columns, no way we are defining 1000 tags by hand
    for column in addColumns:
        tv_datafields.append((column, LABEL))

    #tv_datafields.append(("split", None))
    # Loading Train/Test Split we created
    trn = TabularDataset(
                   path=PATH+'train.csv',
                   format='csv',
                   skip_header=True,
                   fields=tv_datafields)

    vld = TabularDataset(
            path=PATH+'train.csv',
            format='csv',
            skip_header=True,
            fields=tv_datafields)

    # Creating Test Datafield
    tst_datafields = [("id", None),
              ("text", TEXT)]
    # Using TabularDataset, as we want to Analyse Text on it
    tst = TabularDataset(
               path=PATH+"test.csv", # the file path
               format='csv',
               skip_header=True,
               fields=tst_datafields)

    return(trn, vld, tst)

Has uses the same list and order, like my csv does. tv_datafields is structured exactly like the file. Furthermore, as Datafield objects are just Dicts with Datapoints, i read out the Keys of the dictionary, like the tutorial also did, via:
trn[0].dict_keys()

What Should have happened:
The behaviour of the example was like this
trn[0]
torchtext.data.example.Example at 0x10d3ed3c8
trn[0].__dict__.keys()
dict_keys(['comment_text', 'toxic', 'severe_toxic', 'threat', 'obscene', 'insult', 'identity_hate'])

My result:
trn[0].__dict__.keys()
Out[19]: dict_keys([])

trn[1].__dict__.keys()
Out[20]: dict_keys([])

trn[2].__dict__.keys()
Out[21]: dict_keys([])

trn[3].__dict__.keys()
Out[22]: dict_keys(['text'])

While trn[0] does contain nothing, it is instead spread from 3 to 15, the amount of columns that should normally be there should be way more than that.
Now i am at a loss, as to where i went wrong. The Data fits, the function obviously works, but TabularDataset() seems to read in my columns the wrong way (if at all). Did i classify
# Defining Tag and Text
TEXT = Field(sequential=True, tokenize=tokenize, lower=True)
LABEL = Field(sequential=False, use_vocab=False)

the wrong way? At least that is what my Debuggin seems to indicate.
With the meager Documentation on Torchtext i have problems finding that out, but when im looking at the definitions of Data or Fields i cant see anything wrong with it.
Thank you for your help.

Output: I found out where my Problem was, apparently Torchtext only accepts Data in Quotes and only with "," as separator. My Data was not within quotes and has ";" as separator.


Instruction: AttributeError: type object 'TFLiteConverterV2' has no attribute 'from_frozen_gragh'

Input:
I was trying to convert my PyTorch model into TensorFlow lite for mobile. My model was pre-trained DenseNet 169 so I did this:-
import sys
import os
import torch
import torch.nn as nn
import torch.nn.functional as F
import onnx
from collections import OrderedDict
import tensorflow as tf

from torch.autograd import Variable
from onnx_tf.backend import prepare
dummy_input = Variable(torch.randn(32, 3, 224, 224))
torch.onnx.export(trained_model, dummy_input, "mymodel.onnx")

model = onnx.load("mymodel.onnx")
tf_rep = prepare(model)

print('inputs:', tf_rep.inputs)

# Output nodes from the model
print('outputs:', tf_rep.outputs)

# All nodes in the model
print('tensor_dict:')
print(tf_rep.tensor_dict)

tf_rep.export_graph("mymodel.pb")
converter = tf.lite.TFLiteConverter.from_frozen_gragh("mymodel.pb/saved_model.pb",
tf_rep.inputs, tf_rep.outputs) # **ERROR HERE**

tflite_model = converter.convert()
open("mymodel.tflite", "wb").write(tflite_model)

HERE IS MY ERROR
AttributeError                            Traceback (most recent call last)
<ipython-input-37-0abbde392f91> in <module>()
----> 1 converter = tf.lite.TFLiteConverter.from_frozen_gragh("flowers.pb/saved_model.pb", tf_rep.inputs, tf_rep.outputs)

      2 tflite_model = converter.convert()
      3 open("flowers.tflite", "wb").write(tflite_model)

AttributeError: type object 'TFLiteConverterV2' has no attribute 'from_frozen_gragh'

When I tried with compat.v1 I got the same error but instead of TFLiteConverterV2 I got TFLiteConverter
Thanks, in Advance.
EDIT
So I tried with compat.v1 and fixed the typo in 'from_frozen_gragh' and got this ugly error
---------------------------------------------------------------------------
DecodeError                               Traceback (most recent call last)
/usr/local/lib/python3.6/dist-packages/tensorflow/lite/python/lite.py in from_frozen_graph(cls, graph_def_file, input_arrays, output_arrays, input_shapes)
   1804           graph_def = _graph_pb2.GraphDef()
-> 1805           graph_def.ParseFromString(file_content)
   1806         except (_text_format.ParseError, DecodeError):

DecodeError: Error parsing message

During handling of the above exception, another exception occurred:

UnicodeDecodeError                        Traceback (most recent call last)
2 frames
<ipython-input-32-46dac4006b0d> in <module>()
----> 1 tflitconverter = tf.compat.v1.lite.TFLiteConverter.from_frozen_graph("flowers.pb/saved_model.pb", tf_rep.inputs, tf_rep.outputs)
      2 e_model = converter.convert()
      3 open("flowers.tflite", "wb").write(tflite_model)

/usr/local/lib/python3.6/dist-packages/tensorflow/lite/python/lite.py in from_frozen_graph(cls, graph_def_file, input_arrays, output_arrays, input_shapes)
   1812                 file_content = six.ensure_binary(file_content, "utf-8")
   1813               else:
-> 1814                 file_content = six.ensure_text(file_content, "utf-8")
   1815             graph_def = _graph_pb2.GraphDef()
   1816             _text_format.Merge(file_content, graph_def)

/usr/local/lib/python3.6/dist-packages/six.py in ensure_text(s, encoding, errors)
    933     """
    934     if isinstance(s, binary_type):
--> 935         return s.decode(encoding, errors)
    936     elif isinstance(s, text_type):
    937         return s

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb7 in position 3: invalid start byte

PLEASE HELP

Output:
it is "from_frozen_graph" not "from_frozen_gragh"
You need to use compat.v1 since from_frozen_graph is not available in TF 2.x


Instruction: Segmentation to one-hot encoding

Input:
I have a batch of segmented images of size
seg --> [batch, channels, imsize, imgsize] --> [16, 6, 50, 50]
each scalar in this tensor specifies one of the segmentation classes.
We have 2000 total segmentation classes.
Now the goal is to convert
[16, 6, 50, 50] --> [16, 2000, 50, 50]
where each class is encoded in one hot fashion.
How do I do it with pytorch api?
I can only think of ridiculously inefficient looping construction.
Example
Here we will have only 2 initial channels (instead of 6), 4 labels (instead of 2000), size batch 1 (instead of 16) and 4x4 image instead of 50x50.
0, 0, 1, 1
0, 0, 0, 1
1, 1, 1, 1
1, 1, 1, 1

3, 3, 2, 2
3, 3, 2, 2
3, 3, 2, 2
3, 3, 2, 2

Now this turns into 4 channel output
1, 1, 0, 0
1, 1, 1, 0
0, 0, 0, 0
0, 0, 0, 0

0, 0, 1, 1
0, 0, 0, 1
1, 1, 1, 1
1, 1, 1, 1

1, 1, 0, 0
1, 1, 0, 0
1, 1, 0, 0
1, 1, 0, 0

0, 0, 1, 1
0, 0, 1, 1
0, 0, 1, 1
0, 0, 1, 1

The key observation is that a particular label appears only on a single input channel.

Output:
I think you can achieve this without too much trouble. Construct as many masks as there are labels, then stack those masks together, sum on the channel layer and convert to floats:
>>> x
tensor([[[0, 0, 1, 1],
         [0, 0, 0, 1],
         [1, 1, 1, 1],
         [1, 1, 1, 1]],

        [[3, 3, 2, 2],
         [3, 3, 2, 2],
         [3, 3, 2, 2],
         [3, 3, 2, 2]]])

>>> y = torch.stack([x==i for i in range(x.max()+1)], dim=1).sum(dim=2)
tensor([[[1., 1., 0., 0.],
         [1., 1., 1., 0.],
         [0., 0., 0., 0.],
         [0., 0., 0., 0.]],

        [[0., 0., 1., 1.],
         [0., 0., 0., 1.],
         [1., 1., 1., 1.],
         [1., 1., 1., 1.]],

        [[0., 0., 1., 1.],
         [0., 0., 1., 1.],
         [0., 0., 1., 1.],
         [0., 0., 1., 1.]],

        [[1., 1., 0., 0.],
         [1., 1., 0., 0.],
         [1., 1., 0., 0.],
         [1., 1., 0., 0.]]])


Instruction: RuntimeError: Subtraction, the `-` operator, with a bool tensor is not supported

Input:
I am using this github repo: https://github.com/vchoutas/smplify-x and I am running a script. I get the following error. How can I fix it? I understand I might have to convert - to ~ however not sure where it exactly is. I don't want to downgrade my PyTorch.
$ cat fit.sh
export CUDA_VISIBLE_DEVICES=0
python smplifyx/main.py --config cfg_files/fit_smplx.yaml --data_folder ../../data/smplify-x/DATA_FOLDER --output_folder ../../data/smplify-x/RESULTS --visualize="False" --model_folder ../../data/smplify-x/models_smplx_v1_1/models/smplx/SMPLX_NEUTRAL.npz --vposer_ckpt ../../data/smplify-x/vposer_v1_0 --part_segm_fn ../../data/smplify-x/smplx_parts_segm.pkl

--------------------------------------

(smplifyx) mona@ubuntu:~/mona/code/smplify-x$ ./fit.sh
Processing: ../../data/smplify-x/DATA_FOLDER/images/woman_bike.jpg
Found Trained Model: ../../data/smplify-x/vposer_v1_0/snapshots/TR00_E096.pt
Traceback (most recent call last):
  File "smplifyx/main.py", line 272, in <module>
    main(**args)
  File "smplifyx/main.py", line 262, in main
    **args)
  File "~/mona/code/smplify-x/smplifyx/fit_single_frame.py", line 274, in fit_single_frame
    focal_length=focal_length, dtype=dtype)
  File "/home/mona/venv/smplifyx/lib/python3.6/site-packages/torch/autograd/grad_mode.py", line 26, in decorate_context
    return func(*args, **kwargs)
  File "~/mona/code/smplify-x/smplifyx/fitting.py", line 75, in guess_init
    pose_embedding, output_type='aa').view(1, -1) if use_vposer else None
  File "../../data/smplify-x/vposer_v1_0/vposer_smpl.py", line 114, in decode
    if output_type == 'aa': return VPoser.matrot2aa(Xout)
  File "../../data/smplify-x/vposer_v1_0/vposer_smpl.py", line 152, in matrot2aa
    pose = tgm.rotation_matrix_to_angle_axis(homogen_matrot).view(batch_size, 1, -1, 3).contiguous()
  File "/home/mona/venv/smplifyx/lib/python3.6/site-packages/torchgeometry/core/conversions.py", line 233, in rotation_matrix_to_angle_axis
    quaternion = rotation_matrix_to_quaternion(rotation_matrix)
  File "/home/mona/venv/smplifyx/lib/python3.6/site-packages/torchgeometry/core/conversions.py", line 302, in rotation_matrix_to_quaternion
    mask_c1 = mask_d2 * (1 - mask_d0_d1)
  File "/home/mona/venv/smplifyx/lib/python3.6/site-packages/torch/tensor.py", line 511, in __rsub__
    return _C._VariableFunctions.rsub(self, other)
RuntimeError: Subtraction, the `-` operator, with a bool tensor is not supported. If you are trying to invert a mask, use the `~` or `logical_not()` operator instead.

I also have:
(smplifyx) mona@ubuntu:~/mona/code/smplify-x$ python
Python 3.6.9 (default, Oct  8 2020, 12:12:24)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.__version__
'1.7.1'
>>> import torchgeometry
>>> torchgeometry.__version__
'0.1.2'

Output:
$ vi /home/mona/venv/smplifyx/lib/python3.6/site-packages/torchgeometry/core/conversions.py

and comment and add lines similarly for mask conversion compatibility in PyTorch 1.7.1
mask_c0 = mask_d2 * mask_d0_d1
#mask_c1 = mask_d2 * (1 - mask_d0_d1)
mask_c1 = mask_d2 * ~(mask_d0_d1)
#mask_c2 = (1 - mask_d2) * mask_d0_nd1
mask_c2 = ~(mask_d2) * mask_d0_nd1
#mask_c3 = (1 - mask_d2) * (1 - mask_d0_nd1)


Instruction: create a ones tensor according to another lengths tensor

Input:
I have one tensor of the size of [batch_size,1] where each number for sample indicates an integer that is smaller than 5000. I'd like to create a new tensor of the size of [batch_size,5000] where the first numbers for each sample are ones, according to the first tensor, and the rest are zeros. For example:
t1=[[3],[5]]

so we will result with
t2=[[1,1,1,0,0,...],[1,1,1,1,1,0,0,...]]

Output:
This is quite tricky since you want to fill values based on indices and not on the value itself...
Yet you can still manage it, but you have to get creative. We need some way for indices to be reflected on the values themselves. We will keep batch_size=2 and a vector size of 10:
>>> t1 = torch.LongTensor([[3], [5]])
>>> t2 = torch.zeros(len(t1), 10)
tensor([[0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]])

Here is the interesting part: arange a tensor with t2's shape and substract t1.
>>> torch.arange(10).repeat(2, 1) - t1
>>> tensor([[-3, -2, -1,  0,  1,  2,  3,  4,  5,  6],
            [-5, -4, -3, -2, -1,  0,  1,  2,  3,  4]])

Notice how values are negative before the breakpoint, and positive after. That's our mask:
>>> mask = torch.arange(10).repeat(2, 1) - t1 < 0
tensor([[ True,  True,  True, False, False, False, False, False, False, False],
        [ True,  True,  True,  True,  True, False, False, False, False, False]])

To finish it off:
>>> t2[mask] = 1
tensor([[1., 1., 1., 0., 0., 0., 0., 0., 0., 0.],
        [1., 1., 1., 1., 1., 0., 0., 0., 0., 0.]])


Instruction: How do I make Input type and weight type same?

Input:
I am getting a runtime error that says inputs and weights must be on same. However I made sure that my model and input are on the same device yet I cannot get rid of the error. As far as I read, I know that my input data is not on GPU . Since, In this case image is an input so I tried img = torch.from_numpy(img).to(device) and pred = model(img)[0].to(device but no luck. Please let me know what can be done.
Here's the code:
source = '0'
webcam = source == '0'
image_size = 640
imgsz = check_img_size(image_size)
# Load the model
filepath = 'weights/mask.pt'
# device = torch.device('cpu')
device = select_device()
# half = device.type != 'cpu'
model = attempt_load(filepath, map_location = device)
model.to(device).eval()
# if half:
#     model.half()

# Second stage classifier
classify = False
if classify:
    modelc = torch_utils.load_classifier(name = 'resnet101', n = 2)
    modelc.load_state_dict(torch.load('weights/resnet101.pt', map_location = device)['modelc']) ###########
    modelc.to(device).eval()

vid_path, vid_writer = None, None
if webcam:
    view_img = True
    cudnn.benchmark = True
    dataset = LoadStreams(source, img_size = imgsz)

names = model.module.names if hasattr(model, 'module') else model.names
print(names)

def process_image(image):
    h, w = image.shape[:2]
    desired_size = 416
    ratio = desired_size/w
    print("Ratio",ratio)
    img = cv2.resize(image, (0, 0), fx = ratio, fy = ratio)
    h, w = img.shape[:2]
    img = cv2.copyMakeBorder(img, int((416-h)/2), int((416-h)/2), 0, 0, cv2.BORDER_CONSTANT)
    img = img[:, :, ::-1].transpose(2, 0, 1)
    img = np.ascontiguousarray(img)
    img = torch.from_numpy(img).to(device)
    img = img.float()
    img /=255.0
    if img.ndimension() == 3:
        img = img.unsqueeze(0)

    return img

def classify(image):
    # device = torch.device("cpu")

    #image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
    #im = Image.fromarray(image)
    img = process_image(image)
    print('Image processed')
    # img = image.unsqueeze_(0)
    # img = image.float()

    pred = model(img)[0]

    # Apply NMS
    pred = non_max_suppression(pred, 0.4, 0.5, classes = [0, 1, 2], agnostic = None )
    if classify:
        pred = apply_classifier(pred, modelc, img, im0s)
    print("1 ", pred)

    model.eval()
    model.cpu()

    classification = torch.cat(pred)[:, -1]
    if len(classification) == 0:
        return None

    index = int(classification[0])
    print(names[index])
    return names[index]

def detect(frame):
    source = '0'
    webcam = source == '0'

    image_size = 640
    imgsz = check_img_size(image_size)

    # Load model
    file_path = 'weights/yolov5s.pt'
    #device = torch.device('cpu')
    device = select_device()
    # half = device.type != 'cpu'
    model = attempt_load(file_path, map_location = device)
    model.to(device).eval()
    # if half:
    #     model.half()
    names = model.module.names if hasattr(model, 'module') else model.names
    colors = [[75, 125, 2]]
    img = process_image(frame)
    pred = model(img)[0]
    pred = non_max_suppression(pred, 0.4, 0.5, classes = [0], agnostic = None)
    if classify:
        pred = apply_classifier(pred, modelc, img, im0s)
    gn = torch.tensor(frame.shape)[[1,0,1,0]]
    for i, det in enumerate(pred):
        det[:,:4] = scale_coords(img.shape[2:], det[:,:4], frame.shape).round()

        for *xyxy, conf, cls in reversed(det):
            xywh = (xyxy2xywh(torch.tensor(xyxy).view(1, 4)) / gn).view(-1).tolist()  # normalized xywh
            label = '%s %.2f' % (names[int(cls)], conf)
            if label is not None:
                if (label.split())[0] == 'person':
                    plot_one_box(xyxy, frame, label = label, color = colors[0], line_thickness = 1)  # utils.general

Here's the main code:
with tf.Graph().as_default():
    gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=0.6)
    sess = tf.Session(config=tf.ConfigProto(gpu_options=gpu_options, log_device_placement=False))
    with sess.as_default():
        pnet, rnet, onet = detect_face.create_mtcnn(sess, './models/')

        minsize = 20  # minimum size of face
        threshold = [0.6, 0.7, 0.7]  # three steps's threshold
        factor = 0.709  # scale factor
        margin = 44
        frame_interval = 3
        batch_size = 1000
        image_size = 182
        input_image_size = 160

        print('Loading feature extraction model')
        modeldir = './models/'
        facenet.load_model(modeldir)

        images_placeholder = tf.get_default_graph().get_tensor_by_name("input:0")
        embeddings = tf.get_default_graph().get_tensor_by_name("embeddings:0")
        phase_train_placeholder = tf.get_default_graph().get_tensor_by_name("phase_train:0")
        embedding_size = embeddings.get_shape()[1]

        classifier_filename = './myclassifier/my_classifier.pkl'
        classifier_filename_exp = os.path.expanduser(classifier_filename)
        with open(classifier_filename_exp, 'rb') as infile:
            (model, class_names) = pickle.load(infile)
            print('load classifier file-> %s' % type(class_names))
        HumanNames = class_names
        video_capture = cv2.VideoCapture(0)
        c = 0

        print('Start!')
        prevTime = 0
        while True:
            ret, frame = video_capture.read()

            # frame = cv2.resize(frame, (0,0), fx=0.5, fy=0.5)    #resize frame (optional)

            curTime = time.time()    # calc fps
            timeF = frame_interval

            if (c % timeF == 0):
                find_results = []

                if frame.ndim == 2:
                    frame = facenet.to_rgb(frame)
                frame = frame[:, :, 0:3]
                bounding_boxes, _ = detect_face.detect_face(frame, minsize, pnet, rnet, onet, threshold, factor)
                nrof_faces = bounding_boxes.shape[0]
                # print('Bounding Boxes: ', bounding_boxes, 'Shape: ', bounding_boxes.shape, 'nrof_faces:: ', nrof_faces)
                # print('Detected_FaceNum: %d' % nrof_faces)

                if nrof_faces > 0:
                    detect(frame)
                    label = classify(frame)
                    if label == "a":
                        det = bounding_boxes[:, 0:4]
                        img_size = np.asarray(frame.shape)[0:2]

                        cropped = []
                        scaled = []
                        scaled_reshape = []
                        bb = np.zeros((nrof_faces,4), dtype=np.int32)

                        for i in range(nrof_faces):
                            emb_array = np.zeros((1, embedding_size))
                            # print("Embeddinigs:::::")
                            # print(emb_array)
                            # print("Embeddinigs:::::")
                            bb[i][0] = det[i][0]
                            bb[i][1] = det[i][1]
                            bb[i][2] = det[i][2]
                            bb[i][3] = det[i][3]

                            if bb[i][0] <= 0 or bb[i][1] <= 0 or bb[i][2] >= len(frame[0]) or bb[i][3] >= len(frame):
                                print('face is inner of range!')
                                continue

                            cropped.append(frame[bb[i][1]:bb[i][3], bb[i][0]:bb[i][2], :])
                            cropped[0] = facenet.flip(cropped[0], False)
                            scaled.append(misc.imresize(cropped[0], (image_size, image_size), interp='bilinear'))
                            scaled[0] = cv2.resize(scaled[0], (input_image_size,input_image_size),
                                                   interpolation=cv2.INTER_CUBIC)
                            scaled[0] = facenet.prewhiten(scaled[0])
                            scaled_reshape.append(scaled[0].reshape(-1,input_image_size,input_image_size,3))
                            feed_dict = {images_placeholder: scaled_reshape[0], phase_train_placeholder: False}
                            emb_array[0, :] = sess.run(embeddings, feed_dict=feed_dict)

                            predictions = model.predict_proba(emb_array)
                            best_class_indices = np.argmax(predictions, axis=1)
                            best_class_probabilities = predictions[np.arange(len(best_class_indices)), best_class_indices]
                            cv2.rectangle(frame, (bb[i][0], bb[i][1]), (bb[i][2], bb[i][3]), (0, 255, 0), 2)
                            text_x = bb[i][0]
                            text_y = bb[i][3] + 20

(Edit) Error:
Traceback (most recent call last):
  File "realtime.py", line 105, in <module>
    label = classify(frame)
  File "yolov5-master\myutils.py", line 117, in classify
    pred = model(img)[0]
  File "C:\Users\Anuj\anaconda3\envs\py36\lib\site-packages\torch\nn\modules\module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "yolov5-master\models\yolo.py", line 122, in forward
    return self.forward_once(x, profile)  # single-scale inference, train
  File "yolov5-master\models\yolo.py", line 138, in forward_once
    x = m(x)  # run
  File "C:\Users\Anuj\anaconda3\envs\py36\lib\site-packages\torch\nn\modules\module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "yolov5-master\models\common.py", line 94, in forward
    return self.conv(torch.cat([x[..., ::2, ::2], x[..., 1::2, ::2], x[..., ::2, 1::2], x[..., 1::2, 1::2]], 1))
  File "C:\Users\Anuj\anaconda3\envs\py36\lib\site-packages\torch\nn\modules\module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "yolov5-master\models\common.py", line 38, in fuseforward
    return self.act(self.conv(x))
  File "C:\Users\Anuj\anaconda3\envs\py36\lib\site-packages\torch\nn\modules\module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\Users\Anuj\anaconda3\envs\py36\lib\site-packages\torch\nn\modules\conv.py", line 419, in forward
    return self._conv_forward(input, self.weight)
  File "C:\Users\Anuj\anaconda3\envs\py36\lib\site-packages\torch\nn\modules\conv.py", line 416, in _conv_forward
    self.padding, self.dilation, self.groups)
RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same

Output:
You need to send the input tensor to your device, not its result:
pred = model(img.to(device))[0]

As a side note, I would point out that doing x.to(device) as an expression has no effect on the location of the tensor. Instead reassign the tensor with x = x.to(device):
>>> x = torch.ones(1)
>>> x.to(device)
tensor([1], device='cuda:0')
>>> x.is_cuda
False

This is not true for nn.Module, calling model.to(device) will suffice.

Edit - in classify you are sending the model back to the cpu you call it on img. Since you're calling classify in a loop, the first forward will work, while the following calls won't.


Instruction: Inputs to the nn.MultiheadAttention?

Input: I have n-vectors which need to be influenced by each other and output n vectors with same dimensionality d. I believe this is what torch.nn.MultiheadAttention does. But the forward function expects query, key and value as inputs. According to this blog, I need to initialize a random weight matrix of shape (d x d) for each of q, k and v and multiply each of my vectors with these weight matrices and get 3 (n x d) matrices. Now are the q, k and v expected by torch.nn.MultiheadAttention just these three matrices or do I have it mistaken?

Output:
When you want to use self attention, just pass your input vector into torch.nn.MultiheadAttention for the query, key and value.

attention  = torch.nn.MultiheadAttention(<input-size>, <num-heads>)

x, _ = attention(x, x, x)

The pytorch class returns the output states (same shape as input) and the weights used in the attention process.


Instruction: PyTorch - Shapes Don't Match

Input:
Unfortunately, I get the following RuntimeError:

This error hapens at epoch 1 during the last batch (so all other batches run through),
and I don't know what causes the error in my code. Here is a code snippet of my function
def gradient_penalty(critic, real, fake, device):

    BATCH_SIZE, C, H, W = real.shape
    epsilon = torch.rand(size = (BATCH_SIZE, 1, 1, 1)).repeat(1, C, H, W).to(device)

    # generate tensor filles only with ones
    x = torch.ones(size = (BATCH_SIZE, C, H, W), dtype = int)

    # interpolate images
    interpolated_images = real * epsilon + fake * (x - epsilon)

The variable real stands for images and has the shape (128, 3, 64, 64).
I need to admit that I don't find the error message concretely, i. e. where don't the shapes of the tensors coincide?
Any help would be appreciated!

Output:
You can discard incomplete batches when instantiating a DataLoader with the drop_last argument:
torch.utils.data.DataLoader(trainset, batch_size=128, discard_last=True)

However, this seems a bit drastic measure since 128 elements from your dataset will go to waste.


Instruction: ResNet object has no attribute 'predict'

Input:
I have trained a CNN model in PyTorch to detect skin diseases in 6 different classes. My model came out with an accuracy of 92% and I saved it in a .pth file. I wish to use this model for predictions but I don't know how to do so. If anyone can aid me in the necessary steps, I will be grateful.
I have tried just taking the image input straight from the folder, resizing it, and then running it through the model for predictions. The error I face is a ModuleAttributeAError which says there is no attribute named predict. Now I do not understand where I went wrong and I know this is a simple task for most but I was hoping for some guidance in this regard. The dataset I used is the Skin Cancer MNIST: HAM10000 dataset from Kaggle and trained it on ResNet18. If anyone has any pointers on fine-tuning the model, I would greatly appreciate it.
TLDR: I get an error called ModuleAttributeError that says the 'ResNet' module has no attribute 'predict'.
The image is preprocessed here as follows:
import os, cv2,itertools
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import pickle
from tqdm import tqdm
from glob import glob
from PIL import Image

# pytorch libraries
import torch
from torch import optim,nn
from torch.autograd import Variable
from torch.utils.data import DataLoader,Dataset
from torchvision import models,transforms

# sklearn libraries
from sklearn.metrics import confusion_matrix
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report

np.random.seed(10)
torch.manual_seed(10)
torch.cuda.manual_seed(10)

print(os.listdir("/content/drive/My Drive/input"))

from google.colab import drive
drive.mount('/content/drive')

"""**Data analysis and preprocessing**"""

data_dir = '/content/drive/My Drive/input'
all_image_path = glob(os.path.join(data_dir, '*', '*.jpg'))
imageid_path_dict = {os.path.splitext(os.path.basename(x))[0]: x for x in all_image_path}
lesion_type_dict = {
    'nv': 'Melanocytic nevi',
    'mel': 'Melanoma',
    'bkl': 'Benign keratosis-like lesions ',
    'bcc': 'Basal cell carcinoma',
    'akiec': 'Actinic keratoses',
    'vasc': 'Vascular lesions',
    'df': 'Dermatofibroma'
}

def compute_img_mean_std(image_paths):
    """
        computing the mean and std of three channel on the whole dataset,
        first we should normalize the image from 0-255 to 0-1
    """

    img_h, img_w = 224, 224
    imgs = []
    means, stdevs = [], []

    for i in tqdm(range(len(image_paths))):
        img = cv2.imread(image_paths[i])
        img = cv2.resize(img, (img_h, img_w))
        imgs.append(img)

    imgs = np.stack(imgs, axis=3)
    print(imgs.shape)

    imgs = imgs.astype(np.float32) / 255.

    for i in range(3):
        pixels = imgs[:, :, i, :].ravel()  # resize to one row
        means.append(np.mean(pixels))
        stdevs.append(np.std(pixels))

    means.reverse()  # BGR --> RGB
    stdevs.reverse()

    print("normMean = {}".format(means))
    print("normStd = {}".format(stdevs))
    return means,stdevs

# norm_mean,norm_std = compute_img_mean_std(all_image_path)

norm_mean = (0.763035, 0.54564625, 0.5700399)
norm_std = (0.1409281, 0.15261264, 0.16997051)

df_original = pd.read_csv(os.path.join(data_dir, 'HAM10000_metadata.csv'))
df_original['path'] = df_original['image_id'].map(imageid_path_dict.get)
df_original['cell_type'] = df_original['dx'].map(lesion_type_dict.get)
df_original['cell_type_idx'] = pd.Categorical(df_original['cell_type']).codes
df_original.head()

# this will tell us how many images are associated with each lesion_id
df_undup = df_original.groupby('lesion_id').count()
# now we filter out lesion_id's that have only one image associated with it
df_undup = df_undup[df_undup['image_id'] == 1]
df_undup.reset_index(inplace=True)
df_undup.head()

# here we identify lesion_id's that have duplicate images and those that have only one image.
def get_duplicates(x):
    unique_list = list(df_undup['lesion_id'])
    if x in unique_list:
        return 'unduplicated'
    else:
        return 'duplicated'

# create a new colum that is a copy of the lesion_id column
df_original['duplicates'] = df_original['lesion_id']
# apply the function to this new column
df_original['duplicates'] = df_original['duplicates'].apply(get_duplicates)
df_original.head()

df_original['duplicates'].value_counts()

# now we filter out images that don't have duplicates
df_undup = df_original[df_original['duplicates'] == 'unduplicated']
df_undup.shape

# now we create a val set using df because we are sure that none of these images have augmented duplicates in the train set
y = df_undup['cell_type_idx']
_, df_val = train_test_split(df_undup, test_size=0.2, random_state=101, stratify=y)
df_val.shape

df_val['cell_type_idx'].value_counts()

# This set will be df_original excluding all rows that are in the val set
# This function identifies if an image is part of the train or val set.
def get_val_rows(x):
    # create a list of all the lesion_id's in the val set
    val_list = list(df_val['image_id'])
    if str(x) in val_list:
        return 'val'
    else:
        return 'train'

# identify train and val rows
# create a new colum that is a copy of the image_id column
df_original['train_or_val'] = df_original['image_id']
# apply the function to this new column
df_original['train_or_val'] = df_original['train_or_val'].apply(get_val_rows)
# filter out train rows
df_train = df_original[df_original['train_or_val'] == 'train']
print(len(df_train))
print(len(df_val))

df_train['cell_type_idx'].value_counts()

df_val['cell_type'].value_counts()

# Copy fewer class to balance the number of 7 classes
data_aug_rate = [15,10,5,50,0,40,5]
for i in range(7):
    if data_aug_rate[i]:
        df_train=df_train.append([df_train.loc[df_train['cell_type_idx'] == i,:]]*(data_aug_rate[i]-1), ignore_index=True)
df_train['cell_type'].value_counts()

# # We can split the test set again in a validation set and a true test set:
# df_val, df_test = train_test_split(df_val, test_size=0.5)
df_train = df_train.reset_index()
df_val = df_val.reset_index()
# df_test = df_test.reset_index()

Here is where I build the model:
# feature_extract is a boolean that defines if we are finetuning or feature extracting.
# If feature_extract = False, the model is finetuned and all model parameters are updated.
# If feature_extract = True, only the last layer parameters are updated, the others remain fixed.
def set_parameter_requires_grad(model, feature_extracting):
    if feature_extracting:
        for param in model.parameters():
            param.requires_grad = False

def initialize_model(model_name, num_classes, feature_extract, use_pretrained=True):
    # Initialize these variables which will be set in this if statement. Each of these
    #   variables is model specific.
    model_ft = None
    input_size = 0

    if model_name == "resnet":
        """ Resnet18, resnet34, resnet50, resnet101
        """
        model_ft = models.resnet18(pretrained=use_pretrained)
        set_parameter_requires_grad(model_ft, feature_extract)
        num_ftrs = model_ft.fc.in_features
        model_ft.fc = nn.Linear(num_ftrs, num_classes)
        input_size = 224

    elif model_name == "vgg":
        """ VGG11_bn
        """
        model_ft = models.vgg11_bn(pretrained=use_pretrained)
        set_parameter_requires_grad(model_ft, feature_extract)
        num_ftrs = model_ft.classifier[6].in_features
        model_ft.classifier[6] = nn.Linear(num_ftrs,num_classes)
        input_size = 224

    elif model_name == "densenet":
        """ Densenet121
        """
        model_ft = models.densenet121(pretrained=use_pretrained)
        set_parameter_requires_grad(model_ft, feature_extract)
        num_ftrs = model_ft.classifier.in_features
        model_ft.classifier = nn.Linear(num_ftrs, num_classes)
        input_size = 224

    elif model_name == "inception":
        """ Inception v3
        """
        model_ft = models.inception_v3(pretrained=use_pretrained)
        set_parameter_requires_grad(model_ft, feature_extract)
        # Handle the auxilary net
        num_ftrs = model_ft.AuxLogits.fc.in_features
        model_ft.AuxLogits.fc = nn.Linear(num_ftrs, num_classes)
        # Handle the primary net
        num_ftrs = model_ft.fc.in_features
        model_ft.fc = nn.Linear(num_ftrs,num_classes)
        input_size = 299

    else:
        print("Invalid model name, exiting...")
        exit()
    return model_ft, input_size

# resnet,vgg,densenet,inception
model_name = 'resnet'
num_classes = 7
feature_extract = False
# Initialize the model for this run
model_ft, input_size = initialize_model(model_name, num_classes, feature_extract, use_pretrained=True)
# Define the device:
device = torch.device('cuda:0')
# Put the model on the device:
model = model_ft.to(device)

# norm_mean = (0.49139968, 0.48215827, 0.44653124)
# norm_std = (0.24703233, 0.24348505, 0.26158768)
# define the transformation of the train images.
train_transform = transforms.Compose([transforms.Resize((input_size,input_size)),transforms.RandomHorizontalFlip(),
                                      transforms.RandomVerticalFlip(),transforms.RandomRotation(20),
                                      transforms.ColorJitter(brightness=0.1, contrast=0.1, hue=0.1),
                                        transforms.ToTensor(), transforms.Normalize(norm_mean, norm_std)])
# define the transformation of the val images.
val_transform = transforms.Compose([transforms.Resize((input_size,input_size)), transforms.ToTensor(),
                                    transforms.Normalize(norm_mean, norm_std)])

# Define a pytorch dataloader for this dataset
class HAM10000(Dataset):
    def __init__(self, df, transform=None):
        self.df = df
        self.transform = transform

    def __len__(self):
        return len(self.df)

    def __getitem__(self, index):
        # Load data and get label
        X = Image.open(self.df['path'][index])
        y = torch.tensor(int(self.df['cell_type_idx'][index]))

        if self.transform:
            X = self.transform(X)

        return X, y

# Define the training set using the table train_df and using our defined transitions (train_transform)
training_set = HAM10000(df_train, transform=train_transform)
train_loader = DataLoader(training_set, batch_size=64, shuffle=True, num_workers=4)
# Same for the validation set:
validation_set = HAM10000(df_val, transform=train_transform)
val_loader = DataLoader(validation_set, batch_size=64, shuffle=False, num_workers=4)

# we use Adam optimizer, use cross entropy loss as our loss function
optimizer = optim.Adam(model.parameters(), lr=1e-5)
criterion = nn.CrossEntropyLoss().to(device)

Lastly, is the training process with a prediction function:
# this function is used during training process, to calculation the loss and accuracy
class AverageMeter(object):
    def __init__(self):
        self.reset()

    def reset(self):
        self.val = 0
        self.avg = 0
        self.sum = 0
        self.count = 0

    def update(self, val, n=1):
        self.val = val
        self.sum += val * n
        self.count += n
        self.avg = self.sum / self.count

total_loss_train, total_acc_train = [],[]
def train(train_loader, model, criterion, optimizer, epoch):
    model.train()
    train_loss = AverageMeter()
    train_acc = AverageMeter()
    curr_iter = (epoch - 1) * len(train_loader)
    for i, data in enumerate(train_loader):
        images, labels = data
        N = images.size(0)
        # print('image shape:',images.size(0), 'label shape',labels.size(0))
        images = Variable(images).to(device)
        labels = Variable(labels).to(device)

        optimizer.zero_grad()
        outputs = model(images)

        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()
        prediction = outputs.max(1, keepdim=True)[1]
        train_acc.update(prediction.eq(labels.view_as(prediction)).sum().item()/N)
        train_loss.update(loss.item())
        curr_iter += 1
        if (i + 1) % 100 == 0:
            print('[epoch %d], [iter %d / %d], [train loss %.5f], [train acc %.5f]' % (
                epoch, i + 1, len(train_loader), train_loss.avg, train_acc.avg))
            total_loss_train.append(train_loss.avg)
            total_acc_train.append(train_acc.avg)
    return train_loss.avg, train_acc.avg

def validate(val_loader, model, criterion, optimizer, epoch):
    model.eval()
    val_loss = AverageMeter()
    val_acc = AverageMeter()
    with torch.no_grad():
        for i, data in enumerate(val_loader):
            images, labels = data
            N = images.size(0)
            images = Variable(images).to(device)
            labels = Variable(labels).to(device)

            outputs = model(images)
            prediction = outputs.max(1, keepdim=True)[1]

            val_acc.update(prediction.eq(labels.view_as(prediction)).sum().item()/N)

            val_loss.update(criterion(outputs, labels).item())

    print('------------------------------------------------------------')
    print('[epoch %d], [val loss %.5f], [val acc %.5f]' % (epoch, val_loss.avg, val_acc.avg))
    print('------------------------------------------------------------')
    return val_loss.avg, val_acc.avg

import cv2
from PIL import Image, ImageOps
import numpy as np

model = model_ft
model.load_state_dict(torch.load("/content/drive/MyDrive/input/trainbest.pth"))
model.eval()

def import_and_predict(image_data, model):
  size = (224, 224)
  image = ImageOps.fit(image_data, size, Image.ANTIALIAS)
  img = np.asarray(image)
  image_reshape = img[np.newaxis,...]
  prediction = model.predict(img_reshape)
  return prediction

image = Image.open('/content/0365-0596-abd-88-05-0712-gf03.jpg')
# st.image(image, use_column_width = True)
predictions = import_and_predict(image, model)
class_names = ["Melanocytic nevi", "dermatofibroma", "Benign keratosis-like lesions", "Basal cell carcinoma", "Actinic keratoses", "Vascular lesions", "Dermatofibroma"]
string = "It is: " + class_names[np.argmax(predictions)]
print(string)

Here is the error that comes immediately after this is executed.
---------------------------------------------------------------------------
ModuleAttributeError                      Traceback (most recent call last)
<ipython-input-219-d563271b78c6> in <module>()
     32 image = Image.open('/content/0365-0596-abd-88-05-0712-gf03.jpg')
     33 # st.image(image, use_column_width = True)
---> 34 predictions = import_and_predict(image, model)
     35 class_names = ["Melanocytic nevi", "dermatofibroma", "Benign keratosis-like lesions", "Basal cell carcinoma", "Actinic keratoses", "Vascular lesions", "Dermatofibroma"]
     36 string = "It is: " + class_names[np.argmax(predictions)]

1 frames
<ipython-input-219-d563271b78c6> in import_and_predict(image_data, model)
     27   img = np.asarray(image)
     28   image_reshape = img[np.newaxis,...]
---> 29   prediction = model.predict(img_reshape)
     30   return prediction
     31

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __getattr__(self, name)
    777                 return modules[name]
    778         raise ModuleAttributeError("'{}' object has no attribute '{}'".format(
--> 779             type(self).__name__, name))
    780
    781     def __setattr__(self, name: str, value: Union[Tensor, 'Module']) -> None:

ModuleAttributeError: 'ResNet' object has no attribute 'predict'

If anyone can help me fix the issue and get this to work as a classifier for skin diseases, I would be ever so thankful.

Output:
nn.Module don't have a predict function, just call the object for inference:
prediction = model(img_reshape)

This will call the object's __call__ function which, in turns, callsthe model forward function.