Instruction: PyTorch: How to insert before a certain element

Input:
Currently I have a 2D tensor, for each row, I want to insert a new element e before the first index of a specified value v. Additional information: cannot guarantee each row could have a such value. If there isn't, just append the element
Example: Supporse e is 0, v is 10, Given a tensor
[[9, 6, 5, 4, 10],
 [8, 7, 3, 5, 5],
 [4, 9, 10, 10, 10]]

I want to get
[[9, 6, 5, 4, 0, 10],
 [8, 7, 3, 5, 5, 0],
 [4, 9, 0, 10, 10, 10]]

Are there some Torch-style ways to do this? The worst case I can treat this as a trivial Python problem but I think the corresponding solution is a little time-consuming.

Output:
I haven't yet found a full PyTorch solution. I'll keep looking, but here is somewhere to start:
>>> v, e = 10, 0
>>> v, e = torch.tensor([v]), torch.tensor([e])

>>> x = torch.tensor([[ 9,  6,  5,  4, 10],
                      [ 8,  7,  3,  5,  5],
                      [ 4,  9, 10, 10, 10],
                      [10,  9,  7, 10,  2]])

To deal with the edge case where v is not found in one of the rows you can add a temporary column to x. This will ensure every row has a value v in it. We will use x_ as a helper tensor:
>>> x_ = torch.cat([x, v.repeat(x.size(0))[:, None]], axis=1)
tensor([[ 9,  6,  5,  4, 10, 10],
        [ 8,  7,  3,  5,  5, 10],
        [ 4,  9, 10, 10, 10, 10],
        [10,  9,  7, 10,  2, 10]])

Find the indices of the first value v on each row:
>>> bp = (x_ == v).int().argmax(axis=1)
tensor([4, 5, 2, 0])

Finally, the easiest way to insert values at different positions in each row is with a list comprehension:
>>> torch.stack([torch.cat([xi[:bpi], e, xi[bpi:]]) for xi, bpi in zip(x, bp)])
tensor([[ 9,  6,  5,  4,  0, 10],
        [ 8,  7,  3,  5,  5,  0],
        [ 4,  9,  0, 10, 10, 10],
        [ 0, 10,  9,  7, 10,  2]])

Edit - If v cannot occur in the first position, then no need for x_:
>>> x
tensor([[ 9,  6,  5,  4, 10],
        [ 8,  7,  3,  5,  5],
        [ 4,  9, 10, 10, 10]])

>>> bp = (x == v).int().argmax(axis=1) - 1

>>> torch.stack([torch.cat([xi[:bpi], e, xi[bpi:]]) for xi, bpi in zip(x, bp)])
tensor([[ 9,  6,  5,  0,  4, 10],
        [ 8,  7,  3,  5,  0,  5],
        [ 4,  0,  9, 10, 10, 10]])


Instruction: How to stack matrices with different size

Input:
I have a list of matrices with size of (63,32,1,600,600), when I want to stack it with torch.stack(matrices).cpu().detach().numpy() it's raising with error:
"stack expects each tensor to be equal size, but got [32, 1, 600, 600] at entry 0 and [16, 1, 600, 600] at entry 62". Is tried for resizing but it did not work. I appreciate any recommendations.

Output:
If I understand correctly what you're trying to do is stack the outputted mini-batches together into a single batch. My bet is that your last batch is partially filled (only has 16 elements instead of 32).
Instead of using torch.stack (creating a new axis), I would simply concatenate with torch.cat on the batch axis (axis=0). Assuming matrices is a list of torch.Tensors.
torch.cat(matrices).cpu().detach().numpy()

As torch.cat concatenates on axis=0 by default.


Instruction: 1D CNN on Pytorch: mat1 and mat2 shapes cannot be multiplied (10x3 and 10x2)

Input:
I have a time series with sample of 500 size and 2 types of labels and want to construct a 1D CNN with pytorch on them:
class Simple1DCNN(torch.nn.Module):
    def __init__(self):
        super(Simple1DCNN, self).__init__()
        self.layer1 = torch.nn.Conv1d(in_channels=50,
                                      out_channels=20,
                                      kernel_size=5,
                                      stride=2)
        self.act1 = torch.nn.ReLU()
        self.layer2 = torch.nn.Conv1d(in_channels=20,
                                      out_channels=10,
                                      kernel_size=1)

        self.fc1 = nn.Linear(10* 1 * 1, 2)
    def forward(self, x):
        x = x.view(1, 50,-1)
        x = self.layer1(x)
        x = self.act1(x)
        x = self.layer2(x)
        x = self.fc1(x)

        return x

model = Simple1DCNN()
model(torch.tensor(np.random.uniform(-10, 10, 500)).float())

But got this error message:
Traceback (most recent call last):
  File "so_pytorch.py", line 28, in <module>
    model(torch.tensor(np.random.uniform(-10, 10, 500)).float())
  File "/Users/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "so_pytorch.py", line 23, in forward
    x = self.fc1(x)
  File "/Users/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/Users/lib/python3.8/site-packages/torch/nn/modules/linear.py", line 93, in forward
    return F.linear(input, self.weight, self.bias)
  File "/Users/lib/python3.8/site-packages/torch/nn/functional.py", line 1692, in linear
    output = input.matmul(weight.t())
RuntimeError: mat1 and mat2 shapes cannot be multiplied (10x3 and 10x2)

what am I doing wrong?

Output:
The shape of the output of the line x = self.layer2(x) (which is also the input of the next line x = self.fc1(x)) is torch.Size([1, 10, 3]).
Now from the definition of self.fc1, it expects the last dimension of it's input to be 10 * 1 * 1 which is 10 whereas your input has 3 hence the error.
I don't know what it is you're trying to do, but assuming what you want to do is;

label the entire 500 size sequence to one of two labels, the you do this.

# replace self.fc1 = nn.Linear(10* 1 * 1, 2) with
self.fc1 = nn.Linear(10 * 3, 2)

# replace x = self.fc1(x) with
x = x.view(1, -1)
x = self.fc1(x)

label 10 timesteps each to one of two labels, then you do this.

# replace self.fc1 = nn.Linear(10* 1 * 1, 2) with
self.fc1 = nn.Linear(2, 2)

The output shape for 1 will be (batch size, 2), and for 2 will be (batch size, 10, 2).


Instruction: Loss with custom backward function in PyTorch - exploding loss in simple MSE example

Input:
Before working on something more complex, where I knew I would have to implement my own backward pass, I wanted to try something nice and simple. So, I tried to do linear regression with mean squared error loss using PyTorch. This went wrong (see third implementation option below) when I defined my own backward method and I suspect it's because I'm not thinking very clearly about what I need to send PyTorch as gradients. So, I suspect what I need is some explanation/clarification/advice on what PyTorch expects me to provide in what form here.
I am using PyTorch 1.7.0, so a bunch of old examples no longer work (different way of working with user-defined autograd functions as described in the documentation).
First approach (standard PyTorch MSE loss function)
Let's first do it the standard way without a custom loss function:
import torch
import torch.nn as nn
import torch.nn.functional as F

# Let's generate some fake data
torch.manual_seed(42)
resid = torch.rand(100)
inputs = torch.tensor([ [ xx ] for xx in range(100)] , dtype=torch.float32)
labels = torch.tensor([ (2 + 0.5*yy + resid[yy]) for yy in range(100)], dtype=torch.float32)

# Now we define a linear regression model
class linearRegression(torch.nn.Module):
    def __init__(self, inputSize, outputSize):
        super(linearRegression, self).__init__()
        self.bn = torch.nn.BatchNorm1d(num_features=1)
        self.linear = torch.nn.Linear(inputSize, outputSize)

    def forward(self, inx):
        x = self.bn(inx) # Adding BN to standardize input helps us use a higher learning rate
        x = self.linear(x)
        return x

model = linearRegression(1, 1)

# Using the standard mse_loss of PyTorch
epochs = 25
mseloss = F.mse_loss
optimizer = torch.optim.SGD(model.parameters(), lr=1e-2, momentum=1e-3)
scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=20, gamma=0.1)

for epoch in range(epochs):
    model.train()
    optimizer.zero_grad()
    outputs = model(inputs)
    loss = mseloss(outputs.view(-1), labels)
    loss.backward()
    optimizer.step()
    scheduler.step()
    print(f'epoch {epoch}, loss {loss}')

This train just fine and I get to a loss of about 0.0824 and a plot of the fit looks fine.
Second approach (custom loss function, but relying on PyTorch's automatic gradient calculation)
So, now I replace the loss function with my own implementation of the MSE loss, but I still rely on PyTorch autograd. The only things I change here are defining the custom loss function, correspondingly defining the loss based on that, and a minor detail for how I hand over the predictions and true labels to the loss function.
#######################################################3
class MyMSELoss(nn.Module):

    def __init__(self):
        super(MyMSELoss, self).__init__()

    def forward(self, inputs, targets):
        tmp = (inputs-targets)**2
        loss =  torch.mean(tmp)
        return loss
#######################################################3

model = linearRegression(1, 1)

mseloss = MyMSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=1e-2, momentum=1e-3)
scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=20, gamma=0.1)

for epoch in range(epochs):
    model.train()
    outputs = model(inputs)
    loss = mseloss(outputs.view(-1), labels)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()
    scheduler.step()

    print(f'epoch {epoch}, loss {loss}')

This gives completely identical results as using the standard MSE loss function. Loss over epochs looks like this:
epoch 0, loss 884.2006225585938
epoch 1, loss 821.930908203125
epoch 2, loss 718.7732543945312
epoch 3, loss 538.1835327148438
epoch 4, loss 274.50909423828125
epoch 5, loss 55.115299224853516
epoch 6, loss 2.405021905899048
epoch 7, loss 0.47621214389801025
epoch 8, loss 0.1584305614233017
epoch 9, loss 0.09725229442119598
epoch 10, loss 0.0853077694773674
epoch 11, loss 0.08297089487314224
epoch 12, loss 0.08251354098320007
epoch 13, loss 0.08242412656545639
epoch 14, loss 0.08240655809640884
epoch 15, loss 0.08240310847759247
epoch 16, loss 0.08240246027708054
epoch 17, loss 0.08240233361721039
epoch 18, loss 0.08240240067243576
epoch 19, loss 0.08240223675966263
epoch 20, loss 0.08240225911140442
epoch 21, loss 0.08240220695734024
epoch 22, loss 0.08240220695734024
epoch 23, loss 0.08240220695734024
epoch 24, loss 0.08240220695734024

Third approach (custom loss function with my own backward method)
Now, the final version, where I implement my own gradients for the MSE. For that I define my own backward method in the loss function class and apparently need to do mseloss = MyMSELoss.apply.
from torch.autograd import Function

#######################################################
class MyMSELoss(Function):

    @staticmethod
    def forward(ctx, y_pred, y):
        ctx.save_for_backward(y_pred, y)
        return ( (y - y_pred)**2 ).mean()

    @staticmethod
    def backward(ctx, grad_output):
        y_pred, y = ctx.saved_tensors
        grad_input = torch.mean( -2.0 * (y - y_pred)).repeat(y_pred.shape[0])
        # This fails, as does grad_input = -2.0 * (y-y_pred)
        # I've also messed around with the sign and that's not the sole problem, either.
        return grad_input, None

#######################################################

model = linearRegression(1, 1)
mseloss = MyMSELoss.apply
optimizer = torch.optim.SGD(model.parameters(), lr=1e-2, momentum=1e-3)
scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=20, gamma=0.1)

for epoch in range(epochs):
    model.train()
    outputs = model(inputs)
    loss = mseloss(outputs.view(-1), labels)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()
    scheduler.step()
    print(f'epoch {epoch}, loss {loss}')

This is where things go wrong and instead of the training loss decreasing, I get increasing training loss. Now it looks like this:
epoch 0, loss 884.2006225585938
epoch 1, loss 3471.384033203125
epoch 2, loss 47768555520.0
epoch 3, loss 1.7422577779621402e+33
epoch 4, loss inf
epoch 5, loss nan
epoch 6, loss nan
epoch 7, loss nan
epoch 8, loss nan
epoch 9, loss nan
epoch 10, loss nan
epoch 11, loss nan
epoch 12, loss nan
epoch 13, loss nan
epoch 14, loss nan
epoch 15, loss nan
epoch 16, loss nan
epoch 17, loss nan
epoch 18, loss nan
epoch 19, loss nan
epoch 20, loss nan
epoch 21, loss nan
epoch 22, loss nan
epoch 23, loss nan
epoch 24, loss nan

Output:
(2 is constant can be neglected)
So change your backward function to this:
@staticmethod
def backward(ctx, grad_output):
    y_pred, y = ctx.saved_tensors
    grad_input = 2 * (y_pred - y) / y_pred.shape[0]
    return grad_input, None


Instruction: using ModuleList, still getting ValueError: optimizer got an empty parameter list

Input:
With Pytorch I am attempting to use ModuleList to ensure model parameters are detected, and can be optimized. When calling the SGD optimizer I get the following error:

ValueError: optimizer got an empty parameter list

Can you please review the code below and advise?
class LR(nn.Module):
    def ___init___(self):
        super(LR, self).___init___()
        self.linear = nn.ModuleList()
        self.linear.append(nn.Linear(in_features=28*28, out_features=128, bias=True))

    def forward(self, x):
        y_p = torch.sigmoid(self.linear(x))
        return y_p

LR_model = LR()
optimizer = torch.optim.SGD(params = LR_model.parameters(), lr=learn_rate)

Output:
This seems to be a copy-paste issue: your __init__ has 3 underscores instead of 2, both at __init__(self) and super(LR, self).__init__(). Thus the init itself failed. Delete the extra underscores and try again or try the below code:
class LR(nn.Module):
    def __init__(self):
        super(LR, self).__init__()
        self.linear = nn.ModuleList()
        self.linear.append(nn.Linear(in_features=28*28,
                                     out_features=128,
                                     bias=True))

    def forward(self, x):
        y_p = torch.sigmoid(self.linear(x))
        return y_p

    LR_model = LR()
    optimizer = torch.optim.SGD(params = list(LR_model.parameters()),
                                lr=learn_rate)


Instruction: Pytorch: 1D target tensor expected, multi-target not supported

Input:
I want to train a 1D CNN on time series. I get the following error message 1D target tensor expected, multi-target not supported
Here is the code with simulated data corresponding to the structures of my data as well as the error message
import torch
from torch.utils.data import DataLoader
import torch.utils.data as data
import torch.nn as nn

import numpy as np
import random
from tqdm.notebook import tqdm

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print(device)

train_dataset = []
n_item = 20
for i in range(0,n_item):
    train_data = np.random.uniform(-10, 10, 500)
    train_dataset.append(train_data)
train_dataset = np.asarray(train_dataset)
train_dataset.shape
ecg_train = torch.from_numpy(train_dataset).float()
labels_train = np.random.randint(2, size=n_item)
labels_train = torch.from_numpy(labels_train).long()

val_dataset = []
n_item = 10
for i in range(0,n_item):
    val_data = np.random.uniform(-10, 10, 500)
    val_dataset.append(val_data)
val_dataset = np.asarray(val_dataset)
val_dataset.shape
ecg_validation = torch.from_numpy(val_dataset).float()
labels_validation = np.random.randint(2, size=n_item)
labels_validation = torch.from_numpy(labels_validation).long()

class ECGNet(data.Dataset):
    """ImageNet Limited dataset."""

    def __init__(self, ecgs, labls, transform=None):
            self.ecg = ecgs
            self.target = labls
            self.transform = transform

    def __getitem__(self, idx):
        ecgVec = self.ecg[idx] #.reshape(10, -1)
        labelID = self.target[idx].reshape(1)

        return ecgVec,labelID

    def __len__(self):
        return len(self.ecg)

train_data = ECGNet(ecg_train,
                             labels_train,
                   )
print("size of Training dataset: {}".format(len(train_data)))

validation_data = ECGNet(ecg_validation,
                             labels_validation,
                   )
print("size of Training dataset: {}".format(len(validation_data)))

batch_size = 1
train_dataloader = DataLoader(dataset = train_data,
                              batch_size=batch_size,
                              shuffle = True,
                              num_workers = 0)

val_dataloader = DataLoader(dataset = validation_data,
                              batch_size=batch_size,
                              shuffle = True,
                              num_workers = 0)

def train_epoch(model, train_dataloader, optimizer, loss_fn):
    losses = []
    correct_predictions = 0
    # Iterate mini batches over training dataset
    for images, labels in tqdm(train_dataloader):
        images = images.to(device)
    #labels = labels.squeeze_()
        labels = labels.to(device)

        #labels = labels.to(device=device, dtype=torch.int64)
        # Run predictions
        output = model(images)
        # Set gradients to zero
        optimizer.zero_grad()
        # Compute loss
        loss = loss_fn(output, labels)
        # Backpropagate (compute gradients)
        loss.backward()
        # Make an optimization step (update parameters)
        optimizer.step()
        # Log metrics
        losses.append(loss.item())
        predicted_labels = output.argmax(dim=1)
        correct_predictions += (predicted_labels == labels).sum().item()
    accuracy = 100.0 * correct_predictions / len(train_dataloader.dataset)
    # Return loss values for each iteration and accuracy
    mean_loss = np.array(losses).mean()
    return mean_loss, accuracy

def evaluate(model, dataloader, loss_fn):
    losses = []
    correct_predictions = 0
    with torch.no_grad():
        for images, labels in dataloader:
            images = images.to(device)
            #labels = labels.squeeze_()
            labels = labels.to(device=device, dtype=torch.int64)
            # Run predictions
            output = model(images)
            # Compute loss
            loss = loss_fn(output, labels)
            # Save metrics
            predicted_labels = output.argmax(dim=1)
            correct_predictions += (predicted_labels == labels).sum().item()
            losses.append(loss.item())
    mean_loss = np.array(losses).mean()
    accuracy = 100.0 * correct_predictions / len(dataloader.dataset)
    # Return mean loss and accuracy
    return mean_loss, accuracy

def train(model, train_dataloader, val_dataloader, optimizer, n_epochs, loss_function):
    # We will monitor loss functions as the training progresses
    train_losses = []
    val_losses = []
    train_accuracies = []
    val_accuracies = []

    for epoch in range(n_epochs):
        model.train()
        train_loss, train_accuracy = train_epoch(model, train_dataloader, optimizer, loss_fn)
        model.eval()
        val_loss, val_accuracy = evaluate(model, val_dataloader, loss_fn)
        train_losses.append(train_loss)
        val_losses.append(val_loss)
        train_accuracies.append(train_accuracy)
        val_accuracies.append(val_accuracy)
        print('Epoch {}/{}: train_loss: {:.4f},        train_accuracy: {:.4f}, val_loss: {:.4f},        val_accuracy: {:.4f}'.format(epoch+1, n_epochs,
                                     train_losses[-1],
                                     train_accuracies[-1],
                                     val_losses[-1],
                                     val_accuracies[-1]))
    return train_losses, val_losses, train_accuracies, val_accuracies

class Simple1DCNN(torch.nn.Module):
    def __init__(self):
        super(Simple1DCNN, self).__init__()
        self.layer1 = torch.nn.Conv1d(in_channels=50,
                                      out_channels=20,
                                      kernel_size=5,
                                      stride=2)
        self.act1 = torch.nn.ReLU()
        self.layer2 = torch.nn.Conv1d(in_channels=20,
                                      out_channels=10,
                                      kernel_size=1)

        self.fc1 = nn.Linear(10* 3, 2)
    def forward(self, x):
        print(x.shape)
        x = x.view(1, 50,-1)
        print(x.shape)
        x = self.layer1(x)
        print(x.shape)
        x = self.act1(x)
        print(x.shape)
        x = self.layer2(x)
        print(x.shape)
        x = x.view(1,-1)
        print(x.shape)
        x = self.fc1(x)
        print(x.shape)
        print(x)

        return x

model_a = Simple1DCNN()
model_a = model_a.to(device)

criterion = nn.CrossEntropyLoss()

loss_fn = torch.nn.CrossEntropyLoss()
n_epochs_a = 50
learning_rate_a = 0.01
alpha_a = 1e-5
momentum_a = 0.9
optimizer = torch.optim.SGD(model_a.parameters(),
                            momentum = momentum_a,
                            nesterov = True,
                            weight_decay = alpha_a,
                            lr=learning_rate_a)
train_losses_a, val_losses_a, train_acc_a, val_acc_a = train(model_a,
                                                             train_dataloader,
                                                             val_dataloader,
                                                             optimizer,
                                                             n_epochs_a,
                                                             loss_fn
                                                            )

Error message:
cpu
size of Training dataset: 20
size of Training dataset: 10
  0%|          | 0/20 [00:00<?, ?it/s]
torch.Size([1, 500])
torch.Size([1, 50, 10])
torch.Size([1, 20, 3])
torch.Size([1, 20, 3])
torch.Size([1, 10, 3])
torch.Size([1, 30])
torch.Size([1, 2])
tensor([[ 0.5785, -1.0169]], grad_fn=<AddmmBackward>)
Traceback (most recent call last):
  File "SO_question.py", line 219, in <module>
    train_losses_a, val_losses_a, train_acc_a, val_acc_a = train(model_a,
  File "SO_question.py", line 137, in train
    train_loss, train_accuracy = train_epoch(model, train_dataloader, optimizer, loss_fn)
  File "SO_question.py", line 93, in train_epoch
    loss = loss_fn(output, labels)
  File "/Users/mymac/Documents/programming/python/mainenv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/Users/mymac/Documents/programming/python/mainenv/lib/python3.8/site-packages/torch/nn/modules/loss.py", line 961, in forward
    return F.cross_entropy(input, target, weight=self.weight,
  File "/Users/mymac/Documents/programming/python/mainenv/lib/python3.8/site-packages/torch/nn/functional.py", line 2468, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
  File "/Users/mymac/Documents/programming/python/mainenv/lib/python3.8/site-packages/torch/nn/functional.py", line 2264, in nll_loss
    ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
RuntimeError: 1D target tensor expected, multi-target not supported

What am I doing wrong?

Output:
You are using nn.CrossEntropyLoss as the criterion for your training. You correctly passed the labels as indices of the ground truth class: 0s and 1s. However, as the error message suggests, it needs to be a 1D tensor!
Simply remove the reshape in ECGNet's __getitem__:
def __getitem__(self, idx):
    ecgVec = self.ecg[idx]
    labelID = self.target[idx]
    return ecgVec,labelID

Edit

I want to increase the batch_size to 8. But now I get the error [...]

You are doing a lot of broadcasting (flattening) which surely will affect the batch size. As a general rule of thumb never fiddle with axis=0. For instance, if you have an input shape of (8, 500), straight off you have a problem when doing x.view(1, 50, -1). Since the resulting tensor will be (1, 50, 80) (the desired shape would have been (8, 50, 10)). Instead, you could broadcast with x.view(x.size(0), 50, -1).
Same with x.view(1, -1) later down forward. You are looking to flatten the tensor, but you should not flatten it along with the batches, they need to stay separated! It's safer to use torch.flatten, yet I prefer nn.Flatten which flattens from axis=1 to axis=-1 by default.

My personal advice is to start with a simple setup (without train loops etc...) to verify the architecture and intermediate output shapes. Then, add the necessary logic to handle the training.
class ECGNet(data.Dataset):
    """ImageNet Limited dataset."""

    def __init__(self, ecgs, labls, transform=None):
        self.ecg = ecgs
        self.target = labls
        self.transform = transform

    def __getitem__(self, idx):
        ecgVec = self.ecg[idx]
        labelID = self.target[idx]
        return ecgVec, labelID

    def __len__(self):
        return len(self.ecg)

class Simple1DCNN(nn.Module):
    def __init__(self):
        super(Simple1DCNN, self).__init__()
        self.layer1 = nn.Conv1d(in_channels=50,
                                out_channels=20,
                                kernel_size=5,
                                stride=2)
        self.act1 = nn.ReLU()
        self.layer2 = nn.Conv1d(in_channels=20,
                                out_channels=10,
                                kernel_size=1)

        self.fc1 = nn.Linear(10*3, 2)
        self.flatten = nn.Flatten()

    def forward(self, x):
        x = x.view(x.size(0), 50, -1)
        x = self.layer1(x)
        x = self.act1(x)
        x = self.layer2(x)
        x = self.flatten(x)
        x = self.fc1(x)

        return x

batch_size = 8
train_data = ECGNet(ecg_train, labels_train)
train_dl = DataLoader(dataset=train_data,
                      batch_size=batch_size,
                      shuffle=True,
                      num_workers=0)

model = Simple1DCNN()
criterion = nn.CrossEntropyLoss()

Then
>>> x, y = next(iter(train_dl))
>>> y_hat = model(x)

>>> y_hat.shape
torch.Size([8, 2])

Also, make sure your loss works:
>>> criterion(y_hat, y)
tensor(..., grad_fn=<NllLossBackward>)


Instruction: Pip does not recognize torchaudio libary

Input:
When i try the command:
pip install torchaudio

i get this error:
 ERROR: Could not find a version that satisfies the requirement torchaudio
 ERROR: No matching distribution found for torchaudio

I use windows 10

Output:
You could try doing this:
 pip install torchaudio -f https://download.pytorch.org/whl/torch_stable.html

Found on the github.
https://github.com/pytorch/audio


Instruction: RuntimeError: multi-target not supported at /pytorch/aten/src/THCUNN/generic/ClassNLLCriterion.cu:15____

Input:
I face that error
RuntimeError: multi-target not supported at /pytorch/aten/src/THCUNN/generic/ClassNLLCriterion.cu:15____
My input is binary vector of 340, target is binary vector of 8, For '"
loss = criterion(outputs, stat_batch), I got outputs.shape= [64,8] and stat_batch.shape=[64,8]
Here is the model
class MMP(nn.Module):

    def __init__(self, M=1):
        super(MMP, self).__init__()
        # input layer
        self.layer1 = nn.Sequential(
            nn.Conv1d(340, 256,  kernel_size=1, stride=1, padding=0),
            nn.ReLU())
        self.layer2 = nn.Sequential(
            nn.Conv1d(256, 128, kernel_size=1, stride=1, padding=0),
            nn.ReLU())
        self.layer3 = nn.Sequential(
            nn.Conv1d(128, 64, kernel_size=1, stride=1, padding=0),
            nn.ReLU())
        self.drop1 = nn.Sequential(nn.Dropout())
        self.batch1 = nn.BatchNorm1d(128)
        # LSTM
        self.lstm1=nn.Sequential(nn.LSTM(
        input_size=64,
        hidden_size=128,
        num_layers=2,
        bidirectional=True,
        batch_first= True))
        self.fc1 = nn.Linear(128*2,8)
        self.sof = nn.Softmax(dim=-1)

    def forward(self, x):
        out = self.layer1(x)
        out = self.layer2(out)
        out = self.layer3(out)
        out = self.drop1(out)
        out = out.squeeze()
        out = out.unsqueeze(0)
        #out = out.batch1(out)
        out,_ = self.lstm1(out)
        print("lstm",out.shape)
        out = self.fc1(out)
        out =out.squeeze()
        #out = out.squeeze()
        out = self.sof(out)
        return out

#traiin_model
criterion = nn.CrossEntropyLoss()
if CUDA:
    criterion = criterion.cuda()
optimizer = optim.SGD(model.parameters(), lr=LEARNING_RATE, momentum=0.9)

for epoch in range(N_EPOCHES):
    tot_loss=0
    # Training
    for i, (seq_batch, stat_batch) in enumerate(training_generator):
        # Transfer to GPU
        seq_batch, stat_batch = seq_batch.to(device), stat_batch.to(device)
        print(i)
        print(seq_batch)
        print(stat_batch)
        optimizer.zero_grad()
        # Model computation
        seq_batch = seq_batch.unsqueeze(-1)
        outputs = model(seq_batch)
        if CUDA:
            loss = criterion(outputs, stat_batch).float().cuda()
        else:
            loss = criterion(outputs.view(-1), stat_batch.view(-1))
        print(f"Epoch: {epoch},number: {i}, loss:{loss.item()}...\n\n")

        tot_loss += loss.item(print(f"Epoch: {epoch},file_number: {i}, loss:{loss.item()}...\n\n"))
        loss.backward()
        optimizer.step()

Output:
Your target stat_batch must have a shape of (64,) because nn.CrossEntropyLoss takes in class indices, not one-hot-encoding.
Either construct your label tensor appropriately or use stat_batch.argmax(axis=1) instead.


Instruction: Python re-write jupyter notebook to python script problem

Input:
I have a very simple RNN model which I can execute on Jupyter notebook without any problem, now I want to embed this model in my Django project, so I downloaded the .ipynb file then saved it as .py file, then I just put it in my Django project.
However, the first problem is VSCode says I have unresolved import 'MLutils, the MLutils is my utility file in the same folder.
The second problem is if I just run this file, I'll get this error RuntimeError: cannot perform reduction function argmax on a tensor with no elements because the operation does not have an identity
but if I use the Run Cell Above button of the VSCode, I'll get the correct result.
The code I want to run is this, called MLTrain.py
#!/usr/bin/env python
# coding: utf-8

# In[1]:

import torch
import torch.nn as nn
import matplotlib.pyplot as plt

from MLutils import ALL_LETTERS, N_LETTERS
from MLutils import load_data, letter_to_tensor, line_to_tensor, random_training_example

class RNN(nn.Module):
    # implement RNN from scratch rather than using nn.RNN
    # # number of possible letters, hidden size, categories number
    def __init__(self, input_size, hidden_size, output_size):
        super(RNN, self).__init__()

        self.hidden_size = hidden_size
        #Define 2 different liner layers
        self.i2h = nn.Linear(input_size + hidden_size, hidden_size)
        self.i2o = nn.Linear(input_size + hidden_size, output_size)
        self.softmax = nn.LogSoftmax(dim=1)

    def forward(self, input_tensor, hidden_tensor):
        # combine input and hidden tensor
        combined = torch.cat((input_tensor, hidden_tensor), 1)
        # apply the Linear layers and the softmax
        hidden = self.i2h(combined)
        output = self.i2o(combined)
        output = self.softmax(output)
        # return 2 different tensors
        return output, hidden

    # need some initial hidden states in the begining.
    def init_hidden(self):
        return torch.zeros(1, self.hidden_size)
# dictionary with the country as the key and names as values
category_lines, all_categories = load_data()
# number of categories
n_categories = len(all_categories)

# a Hyperparameter
n_hidden = 128
# number of possible letters, hidden size, output size
rnn = RNN(N_LETTERS, n_hidden, n_categories)

# one step
input_tensor = letter_to_tensor('A')
hidden_tensor = rnn.init_hidden()

output, next_hidden = rnn(input_tensor, hidden_tensor)
#print(output.size())
#>>> size: [1,18]
#print(next_hidden.size())
#>>> size: [1,128]
# whole sequence/name
input_tensor = line_to_tensor('if')
hidden_tensor = rnn.init_hidden()

output, next_hidden = rnn(input_tensor[0], hidden_tensor)
print(output.size())
print(next_hidden.size())

# apply softmax in the end.
# this is the likelyhood of each character of each category
def category_from_output(output):
    # return index of the greatest value
    category_idx = torch.argmax(output).item()
    return all_categories[category_idx]
print(category_from_output(output))

criterion = nn.NLLLoss()
# hyperparameter
learning_rate = 0.005
optimizer = torch.optim.SGD(rnn.parameters(), lr=learning_rate)
# whole name as tensor,
def train(line_tensor, category_tensor):
    hidden = rnn.init_hidden()
    #line_tensor.size()[0]: the length of the name
    for i in range(line_tensor.size()[0]):
        # apply the current character and the previous hidden state.
        output, hidden = rnn(line_tensor[i], hidden)

    loss = criterion(output, category_tensor)

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    return output, loss.item()

current_loss = 0
all_losses = []
plot_steps, print_steps = 100, 500
n_iters = 2000
for i in range(n_iters):
    category, line, category_tensor, line_tensor = random_training_example(category_lines, all_categories)

    output, loss = train(line_tensor, category_tensor)
    current_loss += loss

    if (i+1) % plot_steps == 0:
        all_losses.append(current_loss / plot_steps)
        current_loss = 0

    if (i+1) % print_steps == 0:
        guess = category_from_output(output)
        correct = "CORRECT" if guess == category else f"WRONG ({category})"
        print(f"{i+1} {(i+1)/n_iters*100} {loss:.4f} {line} / {guess} {correct}")

plt.figure()
plt.plot(all_losses)
plt.show()
# model can be saved

def predict(input_line):
    print(f"\n> {input_line}")
    with torch.no_grad():
        line_tensor = line_to_tensor(input_line)

        hidden = rnn.init_hidden()

        for i in range(line_tensor.size()[0]):
            output, hidden = rnn(line_tensor[i], hidden)

        guess = category_from_output(output)
        print(guess)

if __name__ == "__main__":
    predict("abcde 1 ifelse")

# In[ ]:

The MLutils.py is this
import io
import os
import unicodedata
import string
import glob

import torch
import random

# alphabet small + capital letters + " .,;'"
ALL_LETTERS = string.ascii_letters + " .,;'"
N_LETTERS = len(ALL_LETTERS)

# Turn a Unicode string to plain ASCII, thanks to https://stackoverflow.com/a/518232/2809427
def unicode_to_ascii(s):
    return ''.join(
        c for c in unicodedata.normalize('NFD', s)
        if unicodedata.category(c) != 'Mn'
        and c in ALL_LETTERS
    )

def load_data():
    # Build the category_lines dictionary, a list of names per language
    category_lines = {}
    all_categories = []

    def find_files(path):
        return glob.glob(path)

    # Read a file and split into lines
    def read_lines(filename):
        lines = io.open(filename, encoding='utf-8').read().strip().split('\n')
        return [unicode_to_ascii(line) for line in lines]

    for filename in find_files('data/categories/*.txt'):
        category = os.path.splitext(os.path.basename(filename))[0]
        all_categories.append(category)

        lines = read_lines(filename)
        category_lines[category] = lines

    return category_lines, all_categories

# Find letter index from all_letters, e.g. "a" = 0
def letter_to_index(letter):
    return ALL_LETTERS.find(letter)

# Just for demonstration, turn a letter into a <1 x n_letters> Tensor
def letter_to_tensor(letter):
    tensor = torch.zeros(1, N_LETTERS)
    tensor[0][letter_to_index(letter)] = 1
    return tensor

# Turn a line into a <line_length x 1 x n_letters>,
# or an array of one-hot letter vectors
def line_to_tensor(line):
    tensor = torch.zeros(len(line), 1, N_LETTERS)
    for i, letter in enumerate(line):
        tensor[i][0][letter_to_index(letter)] = 1
    return tensor

def random_training_example(category_lines, all_categories):

    def random_choice(a):
        random_idx = random.randint(0, len(a) - 1)
        return a[random_idx]

    category = random_choice(all_categories)
    line = random_choice(category_lines[category])
    category_tensor = torch.tensor([all_categories.index(category)], dtype=torch.long)
    line_tensor = line_to_tensor(line)
    return category, line, category_tensor, line_tensor

if __name__ == '__main__':
    print(ALL_LETTERS)
    print(unicode_to_ascii('Ślusàrski'))

    # category_lines, all_categories = load_data()
    # print(category_lines['Flow'][:5])

    # print(letter_to_tensor('J')) # [1, 57]
    # print(line_to_tensor('Jones').size()) # [5, 1, 57]

The Run Above Button is this

If I comment out the print here, I'll get this error

Error Full-Text
Traceback (most recent call last):
  File "c:\Users\Leslie\Desktop\todo_drf\MLModel\MLTrain.py", line 106, in <module>
    category, line, category_tensor, line_tensor = random_training_example(category_lines, all_categories)
  File "c:\Users\Leslie\Desktop\todo_drf\MLModel\MLutils.py", line 84, in random_training_example
    category = random_choice(all_categories)
  File "c:\Users\Leslie\Desktop\todo_drf\MLModel\MLutils.py", line 81, in random_choice
    random_idx = random.randint(0, len(a) - 1)
  File "C:\Users\Leslie\AppData\Local\Programs\Python\Python39\lib\random.py", line 338, in randint
    return self.randrange(a, b+1)
  File "C:\Users\Leslie\AppData\Local\Programs\Python\Python39\lib\random.py", line 316, in randrange
    raise ValueError("empty range for randrange() (%d, %d, %d)" % (istart, istop, width))
ValueError: empty range for randrange() (0, 0, 0)

That's really wired or I am really stupid on somewhere.
Someone please help me!
Many thanks.

Output:
I figured out.
If I use this file structure
MLModel

data

categories

Dataset.txt

MLTrain.py
MLutils.py
For File path, I should use MLModel\data\categories\*.txt in .py scripts.

But I should use data\categories\*.txt in jupyter notebook.
Anyway pay attention to file path when you want to re-write from .ipynb to .py
----From a young man just lost his whole night time.


Instruction: How to convert output to 1/0

Input:
I am Relatively new to building deep learrning models and I seem to be completely confused and stuck with errors related to shape and size.
Here’s the LSTM model and relevant code:
    class LSTMTagger(nn.Module):

        def __init__(self):
            super(LSTMTagger, self).__init__()
            self.embedding =
    nn.Embedding(wv.vectors.shape[0],512)#embedding_matrix.shape[1])
            self.lstm1 = nn.LSTM(input_size = 512, hidden_size = 64, dropout =
    0.1,batch_first=True,bidirectional = True)
            self.dropout = nn.Dropout(p = 0.25)
            self.linear1 = nn.Linear(in_features = 128, out_features = 64)
            self.dropout = nn.Dropout(p = 0.25)
            self.linear2 = nn.Linear(in_features = 64, out_features = 1)
            self.sigmoid = nn.Sigmoid()

        def forward(self, X):
            X_embed = self.embedding(X)
            outr1, _ = self.lstm1(X_embed)
            xr = self.dropout(outr1)
            xr= self.linear1(xr)
            xr = self.dropout(xr)
            xr= self.linear2(xr)
            outr4 = self.sigmoid(xr)
            outr4 = outr4.view(1,-1)

            return outr4

model = LSTMTagger()
torch.multiprocessing.set_sharing_strategy('file_system')
if torch.cuda.device_count() > 1:
  print("Using ", torch.cuda.device_count(), " GPUs")
  # dim = 0 [30, xxx] -> [10, ...], [10, ...], [10, ...] on 3 GPUs

# model =model.load_state_dict(torch.load('best_model_state.bin'))
model = nn.DataParallel(model, device_ids=[0]) #py r
torch.cuda.empty_cache()
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

model = model.to(device)

def train_epoch(
      model,
      data_loader,
      loss_fn,
      optimizer,
      device,
      scheduler,
      n_examples
    ):
      model = model.train()
      losses = []
      correct_predictions = 0
      for d in data_loader:
        print(f"Input ids: {np.shape(d['input_ids'])}\n len: {len(d['input_ids'][0])}")
        input_ids = d["input_ids"].to(device)
        targets = d["targets"].to(device)
        outputs = model(input_ids)
        _, preds = torch.max(outputs, dim=1)
        print(f"outputs is {np.shape(outputs)}")
        print(f"targets is {targets}")
        # continue
        loss = criterion(outputs.squeeze(), targets)
        # loss.backward()
        # nn.utils.clip_grad_norm_(model.parameters(), clip)
        # optimizer.step()
        # loss = loss_fn(outputs, targets)
        correct_predictions += torch.sum(preds == targets)
        losses.append(loss.item())
        loss.backward()
        nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
        optimizer.step()
        scheduler.step()
        optimizer.zero_grad()
      return correct_predictions.double() / n_examples, np.mean(losses)
EPOCHS = 6
optimizer = optim.Adam(model.parameters(), lr=2e-5)
total_steps = len(data_train) * EPOCHS
scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=1)
loss_fn = nn.CrossEntropyLoss().to(device)
history = defaultdict(list)
best_accuracy = 0
criterion = nn.BCELoss()

print('starting training')
# exit()

for epoch in range(EPOCHS):
      print(f'Epoch {epoch + 1}/{EPOCHS}')
      print('-' * 10)
      train_acc, train_loss = train_epoch(
        model,
        data_train,
        loss_fn,
        optimizer,
        device,
        scheduler,
        len(df_train)
      )

In this instance the sample input is a tensor of size: torch.Size([1, 512])
, that looks like this:
tensor([[44561, 972, 7891, 94, 2191, 131, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0]], device=‘cuda:0’)

and out put label (targets from train_epoch function) in case is just a simple 1 or 0 label in tensor form such as tensor([1], device=‘cuda:0’).
I have been facing issues consistently with this approach. Initially the output was 1x512x1. So, I added
outr4 = outr4.view(1,-1)

after the sigmoid layer. Then, the output shape was reduced to 1x512 and I used squeeze function but, still, I face errors such as this one:
ValueError: Using a target size (torch.Size([1])) that is different to the input size (torch.Size([512])) is deprecated. Please ensure they have the same size.

I have spent a lot of time trying to figure out what was going on but to no avail. Isn’t the output supposed to be either ?1 or 0, instead of being a 1x512 shaped tensor?
I am relatively new to building models, so please excuse my lack of knowledge.

Output:
I suggest you track the shapes of your tensors as they pass through your forward function, e.g. by printing X.shape after every operation.
Then you'll be less confused as you can understand the transformations.
In your case I think it goes as follows:

Input: [1, 512, 1]
Embedding: [1, 512, 512]
LSTM: [1, 512, 64x2]
Lin1: [1, 512, 64]
Lin2: [1, 512, 1]

Then your activation function doesn't reshape the tensor, it just squashes the values in the last dimension to fit between 0 and 1.
Logically, you run into an issue as you have 512 outputs (one for every word/token) instead of 1 (for the sentence). You have never reduced the word dimension to 1.
To fix this, you have to flatten/pool the 512 dimension at some point in your model.
For example, you could average over the word dimension after you run it through your LSTM:

LSTM: [1, 512, 64x2]
Avg: [1, 64x2]
Lin1: [1, 64]
Lin2: [1, 1]

Or you can take only the last hidden state of your LSTM, which would also be of shape [1, 1, 128].
EDIT: Also, be careful of using so much padding.
This might have undesired influence on your outcomes. You should try to work with a mask that remembers which inputs were actual words and which were padding spots. For example, averaging over so much padding will greatly lower the results; you should only average over the actual tokens.
PyTorch has some functionality for the LSTM as well for this, in the form of  pack_padded_sequence ( https://pytorch.org/docs/stable/generated/torch.nn.utils.rnn.pack_padded_sequence.html ) and pad_packed_sequence. An example of their usage : https://suzyahyah.github.io/pytorch/2019/07/01/DataLoader-Pad-Pack-Sequence.html


Instruction: Scale down image represented in a tensor

Input:
I use the MNIST dataset to learn Pytorch.
This is from the documentation to get a picture.
import torch.nn.functional as F
import torch
from torchvision import datasets, transforms

Tensor comes from the torchvision dataset.
# Create prediction
images, labels = next(iter(trainloader))
images[0].shape

This is the Tensor:
tensor([[-1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,
         -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,
         -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,
         -1.0000, -1.0000, -1.0000, -1.0000],
        [-1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,
         -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,
         -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,
         -1.0000, -1.0000, -1.0000, -1.0000],
        [-1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,
         -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,
         -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,
         -1.0000, -1.0000, -1.0000, -1.0000],
        [-1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,
         -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,
         -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,
         -1.0000, -1.0000, -1.0000, -1.0000],
        [-1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,
         -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -0.2549, -0.5059, -1.0000,
         -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,
         -1.0000, -1.0000, -1.0000, -1.0000],
        [-1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,
         -1.0000, -0.8667, -0.3961,  0.2471,  0.6784,  0.9922,  0.8824,  0.3882,
         -0.8353, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,
         -1.0000, -1.0000, -1.0000, -1.0000],
        [-1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -0.5922,
          0.3490,  0.6235,  0.9922,  0.9922,  0.9922,  0.9922,  0.9922,  0.9922,
         -0.2235, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,
         -1.0000, -1.0000, -1.0000, -1.0000],
        [-1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -0.3647,  0.9451,
          0.9922,  0.9922,  0.9922,  0.9922,  0.9922,  0.9922,  0.8510,  0.9922,
          0.8745, -0.3725, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,
         -1.0000, -1.0000, -1.0000, -1.0000],
        [-1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -0.0118,  0.9922,
          0.9922,  0.9922,  0.9843,  0.8902,  0.3255, -0.1922, -0.6314,  0.7725,
          0.9922,  0.7804, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,
         -1.0000, -1.0000, -1.0000, -1.0000],
        [-1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,  0.4431,  0.9922,
          0.9922,  0.7725, -0.1608, -1.0000, -1.0000, -1.0000, -1.0000,  0.6235,
          0.9922,  0.7804, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,
         -1.0000, -1.0000, -1.0000, -1.0000],
        [-1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -0.0745,  0.9922,
          0.7333, -0.7020, -1.0000, -1.0000, -1.0000, -1.0000, -0.3020,  0.9843,
          0.9922,  0.7804, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,
         -1.0000, -1.0000, -1.0000, -1.0000],
        [-1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -0.8980,  0.2078,
         -0.0902, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -0.2863,  0.9922,
          0.9922,  0.7804, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,
         -1.0000, -1.0000, -1.0000, -1.0000],
        [-1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,
         -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,  0.0039,  0.9922,
          0.9922,  0.7804, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,
         -1.0000, -1.0000, -1.0000, -1.0000],
        [-1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,
         -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -0.8275,  0.8275,  0.9922,
          0.9922, -0.0118, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,
         -1.0000, -1.0000, -1.0000, -1.0000],
        [-1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,
         -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,  0.5765,  0.9922,  1.0000,
          0.9529, -0.3725, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,
         -1.0000, -1.0000, -1.0000, -1.0000],
        [-1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,
         -1.0000, -1.0000, -1.0000, -0.9922, -0.3804,  0.9765,  0.9922,  0.9922,
          0.0039, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,
         -1.0000, -1.0000, -1.0000, -1.0000],
        [-1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,
         -1.0000, -1.0000, -0.8667,  0.0275,  0.9922,  0.9922,  0.9922,  0.9922,
         -0.4667, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,
         -1.0000, -1.0000, -1.0000, -1.0000],
        [-1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,
          0.0039,  0.8980,  0.9137,  0.9922,  0.9922,  0.9922,  0.9922,  0.9922,
          0.9216,  0.8980,  0.8980,  0.8980,  0.8980,  0.8980,  0.6314, -0.1843,
         -0.6941, -1.0000, -1.0000, -1.0000],
        [-1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,
          0.6784,  0.9922,  0.9922,  0.9922,  0.9922,  0.9922,  0.9922,  0.9922,
          0.9922,  1.0000,  0.9922,  0.9922,  0.9922,  0.9922,  0.9922,  0.9922,
          0.8353,  0.1451, -1.0000, -1.0000],
        [-1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,
          0.6784,  0.9922,  0.9922,  0.9922,  0.9922,  0.9922,  0.9922,  0.9922,
          0.9922,  0.9922,  0.9922,  0.9922,  0.9922,  0.9922,  0.8745,  0.5059,
          0.4196, -0.5686, -1.0000, -1.0000],
        [-1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,
         -0.1765,  0.9216,  0.9922,  0.9765,  0.6706,  0.6706,  0.2863, -0.4118,
         -0.4118, -0.4118, -0.4118, -0.4118, -0.4118, -0.4118, -0.5529, -1.0000,
         -1.0000, -1.0000, -1.0000, -1.0000],
        [-1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,
         -1.0000, -0.4275, -0.2471, -0.2863, -1.0000, -1.0000, -1.0000, -1.0000,
         -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,
         -1.0000, -1.0000, -1.0000, -1.0000],
        [-1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,
         -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,
         -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,
         -1.0000, -1.0000, -1.0000, -1.0000],
        [-1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,
         -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,
         -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,
         -1.0000, -1.0000, -1.0000, -1.0000],
        [-1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,
         -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,
         -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,
         -1.0000, -1.0000, -1.0000, -1.0000],
        [-1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,
         -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,
         -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,
         -1.0000, -1.0000, -1.0000, -1.0000],
        [-1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,
         -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,
         -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,
         -1.0000, -1.0000, -1.0000, -1.0000],
        [-1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,
         -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,
         -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000, -1.0000,
         -1.0000, -1.0000, -1.0000, -1.0000]])

Out[181]: torch.Size([1, 28, 28])

I want to scale the image down to a 14x14 picture, so I guess I need a torch.Size([1, 14, 14])
I tried this, but it results in a different format:
F.interpolate(images[0], 14).shape

Out[184]: torch.Size([1, 28, 14])

I expected this to work, but it results in an error:
F.interpolate(images[0], (14, 14))

ValueError: size shape must match input shape. Input is 1D, size is 2

Does anyone how to to get my desired result?

Output:
From the docs:

The input dimensions are interpreted in the form: mini-batch x
channels x [optional depth] x [optional height] x width.

Currently the first 28 in your shape, and the 14 in the output size, are interpreted as the number of channels/colors, not the image height. Therefore the scaling does not happen in that dimension. Unsqueeze your input to be of shape (1, 1, 28, 28) to get the correct behavior.


Instruction: Create a custom dataset from a folder with separate files

Input:
I am using Pytorch's custom dataset feature to create a custom dataset from separate files in one folder. Each file contains 123 rows and 123 columns, and all data points are integers.
My issue is that the resources I've come across cater to files in one .csv, mine are not. More so, opening the images after being transformed as an image doesn't run as well. I'm not sure how to proceed from here on as my code gives:
AttributeError: 'Image' object has no attribute 'read'

import os
from torch.utils.data import DataLoader, Dataset
from numpy import genfromtxt

# Custom dataset
class CONCEPTDataset(Dataset):
    """ Concept Dataset """

    def __init__(self, file_dir, transforms=None):
        """
        Args:
            file_dir (string): Directory with all the images.
            transforms (optional): Changes on the data.
        """
        self.file_dir = file_dir
        self.transforms = transforms

        self.concepts = os.listdir(file_dir)
        self.concepts.sort()
        self.concepts = [os.path.join(file_dir, concept) for concept in self.concepts]

    def __len__(self):
        return len(self.concepts)

    def __getitem__(self, idx):
        image = self.concepts[idx]

        # csv file to a numpy array using genfromtxt
        data = genfromtxt(image, delimiter=',')

        data = self.transforms(data.unsqueeze(0))
        return data

Output: PIL.Image.fromarray is used to convert an array to a PIL Image while Image.open is used to load an image file from the file system. You don't need either of those two since you already have a NumPy array representing your image and are looking to return it. PyTorch will convert it to torch.Tensor automatically if you plug your dataset to a torch.data.utils.DataLoader.


Instruction: High loss in neural network sequence classification

Input:
I am using neural network to classify sequence of length 340 to 8 classes, I am using cross entropy as loss. I am getting very high number for the loss . I am wondering if I did mistake in calculating the loss for each epoch. Or should i use other loss function .
criterion = nn.CrossEntropyLoss()
if CUDA:
    criterion = criterion.cuda()
optimizer = optim.SGD(model.parameters(), lr=LEARNING_RATE, momentum=0.9)
loss_list = []
for epoch in range(N_EPOCHES):
    tot_loss=0
    running_loss =0
    model.train()
    loss_values = []
    acc_list = []
    acc_list = torch.FloatTensor(acc_list)
    sum_acc = 0
    # Training
    for i, (seq_batch, stat_batch) in enumerate(training_generator):
        # Transfer to GPU
        seq_batch, stat_batch = seq_batch.to(device), stat_batch.to(device)
        optimizer.zero_grad()
        # Model computation
        seq_batch = seq_batch.unsqueeze(-1)
        outputs = model(seq_batch)
        loss = criterion(outputs.argmax(1), stat_batch.argmax(1))
        loss.backward()
        optimizer.step()
        # print statistics
        running_loss += loss.item()*seq_batch.size(0)
        loss_values.append(running_loss/len(training_set))
        if i % 2000 == 1999:  # print every 2000 mini-batches
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 50000),"acc",(outputs.argmax(1) == stat_batch.argmax(1)).float().mean())
            running_loss = 0.0
        sum_acc += (outputs.argmax(1) == stat_batch.argmax(1)).float().sum()
    print("epoch" , epoch, "acc", sum_acc/len(training_generator))
print('Finished Training')

[1,  2000] loss: 14.205 acc tensor(0.5312, device='cuda:0')
[1,  4000] loss: 13.377 acc tensor(0.4922, device='cuda:0')
[1,  6000] loss: 13.159 acc tensor(0.5508, device='cuda:0')
[1,  8000] loss: 13.050 acc tensor(0.5547, device='cuda:0')
[1, 10000] loss: 12.974 acc tensor(0.4883, device='cuda:0')
epoch 1 acc tensor(133.6352, device='cuda:0')
[2,  2000] loss: 12.833 acc tensor(0.5781, device='cuda:0')
[2,  4000] loss: 12.834 acc tensor(0.5391, device='cuda:0')
[2,  6000] loss: 12.782 acc tensor(0.5195, device='cuda:0')
[2,  8000] loss: 12.774 acc tensor(0.5508, device='cuda:0')
[2, 10000] loss: 12.762 acc tensor(0.5156, device='cuda:0')
epoch 2 acc tensor(139.2496, device='cuda:0')
[3,  2000] loss: 12.636 acc tensor(0.5469, device='cuda:0')
[3,  4000] loss: 12.640 acc tensor(0.5469, device='cuda:0')
[3,  6000] loss: 12.648 acc tensor(0.5508, device='cuda:0')
[3,  8000] loss: 12.637 acc tensor(0.5586, device='cuda:0')
[3, 10000] loss: 12.620 acc tensor(0.6016, device='cuda:0')
epoch 3 acc tensor(140.6962, device='cuda:0')
[4,  2000] loss: 12.520 acc tensor(0.5547, device='cuda:0')
[4,  4000] loss: 12.541 acc tensor(0.5664, device='cuda:0')
[4,  6000] loss: 12.538 acc tensor(0.5430, device='cuda:0')
[4,  8000] loss: 12.535 acc tensor(0.5547, device='cuda:0')
[4, 10000] loss: 12.548 acc tensor(0.5820, device='cuda:0')
epoch 4 acc tensor(141.6522, device='cuda:0')

Output:
I am getting very high number for the loss

What makes you think this is high? What do you compare this to?
Yes, you should use nn.CrossEntropyLoss for multi-class classification tasks. And your training loss seems perfectly fine to me. At initialization, you should have loss = -log(1/8) = ~2.


Instruction: Applying a simple transformation to get a binary image using pytorch

Input:
I'd like to binarize image before passing it to the dataloader, I have created a dataset class which works well. but in the __getitem__() method  I'd like to threshold the image:
    def __getitem__(self, idx):
        # Open image, apply transforms and return with label
        img_path = os.path.join(self.dir, self.filelist[filename"])
        image = Image.open(img_path)
        label = self.x_data.iloc[idx]["label"]

        # Applying transformation to the image
        if self.transforms is not None:
           image = self.transforms(image)

        # applying threshold here:
        my_threshold = 240
        image = image.point(lambda p: p < my_threshold and 255)
        image = torch.tensor(image)

        return image, label

And then I tried to invoke the dataset:
    data_transformer = transforms.Compose([
        transforms.Resize((10, 10)),
        transforms.Grayscale()
        //transforms.ToTensor()
    ])

train_set = MyNewDataset(data_path, data_transformer, rows_train)

Since I have applied the threshold on a PIL object I need to apply afterwards a conversion to a tensor object , but for some reason it crashes. can somebody please assist me?

Output:
Why not apply the binarization after the conversion from PIL.Image to torch.Tensor?
class ThresholdTransform(object):
  def __init__(self, thr_255):
    self.thr = thr_255 / 255.  # input threshold for [0..255] gray level, convert to [0..1]

  def __call__(self, x):
    return (x > self.thr).to(x.dtype)  # do not change the data type

Once you have this transformation, you simply add it:
data_transformer = transforms.Compose([
        transforms.Resize((10, 10)),
        transforms.Grayscale(),
        transforms.ToTensor(),
        ThresholdTransform(thr_255=240)
    ])


Instruction: Customized loss function in PyTorch which uses DNN outputs and additional variables

Input:
(I am sorry if my English is not good)
I can create my own loss function in PyTorch if the function requires only DNN output vector(predicted) and DNN output vector(ground truth).
I want to use additional variables to calculate the loss.
I make my training and test data like below;
DNN input:

Data_A -> processing 1 -> Data_X

DNN output:

Data_A -> processing 1 -> Data_X
Data_B -> processing 1 -> Data_P
Data_X , Data_P -> processing 2 -> Data_Y

and I divide Data_X and Data_Y into train data and test data.
x_train, x_test, y_train, y_test = train_test_split(Data_X,Data_Y,test_size=0.2, random_state=0)
I want to use Data_A, Data_B, Data_Y(predicted), and Data_Y(ground truth) to calculate the loss.
I saw many examples for customized loss function which only use Data_Y(predicted) and Data_Y(ground truth). I could use such a customized loss function before.
However, I don't know what to do when I want to use another additional variables.
Is there a good way? Thank you for your help!

Output:
You have no restrictions over the structure of your loss function (as long as the gradients make sense).
For instance, you can have:
class MyLossLayer(nn.Module):
  def __init__(self):
    super(MyLossLayer, self).__init__()

  def forward(self, pred_a, pred_b, gt_target):
    # I'm just guessing here - do whatever you want as long as you do not screw the gradients.
    loss = pred_a * (pred_b - target)
    return loss.mean()


Instruction: Cuda 10.2 not recognised on Pip installed Pytorch 1.7.1

Input:
Pip installing Pytorch 1.7.1 here on Gtx1660(latest drivers installed) doesn't recognise the installed Cuda toolkit 10.2 on my machine(Windows10).
As this is a personal project, I don't wish to use Anaconda.
How to resolve this?
#INSTALLATION (10.2 CUDA SUPPORT)
pip install torch===1.7.1 torchvision===0.8.2 torchaudio===0.7.2 -f https://download.pytorch.org/whl/torch_stable.html

#CODE
import torch
device = torch.device("cuda")
# device = torch.device("cuda:0") #ALSO NOT WORKING
cuda_available = torch.cuda.is_available()
cuda_init = torch.cuda.is_initialized()
print(f'device_current = {device},cuda_available = {cuda_available} and cuda_init = {cuda_init}\n')

torch.cuda.init()
print(f'device_current = {device} and cuda_init = {cuda_init}')

#TERMINAL
device_current = cuda:0,cuda_available = False and cuda_init = False

Traceback (most recent call last):
  File "f:\Script\Ai\Pytorch\mod_test\test1.py", line 8, in <module>
    torch.cuda.init()
  File "C:\Users\User0\AppData\Local\Programs\Python\Python39\lib\site-packages\torch\cuda\__init__.py", line 137, in init
    _lazy_init()
  File "C:\Users\User0\AppData\Local\Programs\Python\Python39\lib\site-packages\torch\cuda\__init__.py", line 166, in _lazy_init
    raise AssertionError("Torch not compiled with CUDA enabled")
AssertionError: Torch not compiled with CUDA enabled

Output:
I could be wrong, but here is what I found while searching on their package registry.
For a PyTorch 1.7.1 pip install. The instructions on pytorch.org are:
## cu101
pip install torch==1.7.1+cu101 torchvision==0.8.2+cu101 torchaudio===0.7.2 -f https://download.pytorch.org/whl/torch_stable.html

## cu110
pip install torch===1.7.1+cu110 torchvision===0.8.2+cu110 torchaudio===0.7.2 -f https://download.pytorch.org/whl/torch_stable.html

Yet the command for CUDA 10.2 support is:
## cu102
pip install torch===1.7.1 torchvision===0.8.2 torchaudio===0.7.2 -f https://download.pytorch.org/whl/torch_stable.html

This seems off because the CUDA version is not stated anywhere in the command. Yet download.pytorch.org provides support for this particular PyTorch version with cu92, cu101, cu102, and cu110.

You can either use:
pip install torch==1.7.0 torchvision==0.8.1 -f https://download.pytorch.org/whl/cu102/torch_stable.html

or try this instead from the main Torch stable directory:
pip install torch==1.7.1+cu102 torchvision==0.8.2+cu102 torchaudio===0.7.2 -f https://download.pytorch.org/whl/torch_stable.html


Instruction: PyTorch CNN linear layer shape after conv2d

Input:
I was trying to learn PyTorch and came across a tutorial where a CNN is defined like below,
class Net(Module):
    def __init__(self):
        super(Net, self).__init__()

        self.cnn_layers = Sequential(
            # Defining a 2D convolution layer
            Conv2d(1, 4, kernel_size=3, stride=1, padding=1),
            BatchNorm2d(4),
            ReLU(inplace=True),
            MaxPool2d(kernel_size=2, stride=2),
            # Defining another 2D convolution layer
            Conv2d(4, 4, kernel_size=3, stride=1, padding=1),
            BatchNorm2d(4),
            ReLU(inplace=True),
            MaxPool2d(kernel_size=2, stride=2),
        )

        self.linear_layers = Sequential(
            Linear(4 * 7 * 7, 10)
        )

    # Defining the forward pass
    def forward(self, x):
        x = self.cnn_layers(x)
        x = x.view(x.size(0), -1)
        x = self.linear_layers(x)
        return x

I understood how the cnn_layers are made. After the cnn_layers, the data should be flattened and given to linear_layers.
I don't understand how the number of features to Linear is 4*7*7. I understand that 4 is the output dimension from the last Conv2d layer.
How is 7*7 coming in to picture? Does stride or padding got any role in that?
Input image shape is [1, 28, 28]

Output: Conv2d layers have a kernel size of 3, stride and padding of 1, which means it doesn't change the spatial size of an image. There are two MaxPool2d layers which reduce the spatial dimensions from (H, W) to (H/2, W/2). So, for each batch, output of the last convolution with 4 output channels has a shape of (batch_size, 4, H/4, W/4). In the forward pass feature tensor is flattened by x = x.view(x.size(0), -1) which makes it in the shape (batch_size, H*W/4). I assume H and W are 28, for which the linear layer would take inputs of shape (batch_size, 196).


Instruction: Pretrained model or training from scratch for object detection?

Input:
I have a dataset composed of 10k-15k pictures for supervised object detection which is very different from Imagenet or Coco (pictures are much darker and represent completely different things, industrial related).
The model currently used is a FasterRCNN which extracts features with a Resnet used as a backbone.
Could train the backbone of the model from scratch in one stage and then train the whole network in another stage be beneficial for the task, instead of loading the network pretrained on Coco and then retraining all the layers of the whole network in a single stage?

Output:
From my experience, here are some important points:

your train set is not big enough to train the detector from scratch (though depends on network configuration, fasterrcnn+resnet18 can work). Better to use a pre-trained network on the imagenet;
the domain the network was pre-trained on is not really that important. The network, especially the big one, need to learn all those arches, circles, and other primitive figures in order to use the knowledge for detecting more complex objects;
the brightness of your train images can be important but is not something to stop you from using a pre-trained network;
training from scratch requires much more epochs and much more data. The longer the training is the more complex should be your LR control algorithm. At a minimum, it should not be constant and change the LR based on the cumulative loss. and the initial settings depend on multiple factors, such as network size, augmentations, and the number of epochs;
I played a lot with fasterrcnn+resnet (various number of layers) and the other networks. I recommend you to use maskcnn instead of fasterrcnn. Just command it not to use the masks and not to do the segmentation. I don't know why but it gives much better results.
don't spend your time on mobilenet, with your train set size you will not be able to train it with some reasonable AP and AR. Start with maskrcnn+resnet18 backbone.


Instruction: how to save the generated images from this code separated

Input:
I have run StarGAN Code from github, this code generate all the generated images in one picture.
How I can save all the generated images separated to single folder? I do not want to save all the images in one picture.
this is how it generate the output (sample image
I want to save the generated images from the trained model, not as samples with all the images in one picture but just a file that contains all the generated images.
This is the part of the code i want to change it
# Translate fixed images for debugging.
if (i+1) % self.sample_step == 0:
    with torch.no_grad():
        x_fake_list = [x_fixed]
        for c_fixed in c_fixed_list:
            x_fake_list.append(self.G(x_fixed, c_fixed))
        x_concat = torch.cat(x_fake_list, dim=3)
        sample_path = os.path.join(self.sample_dir, '{}-images.jpg'.format(i+1))
        save_image(self.denorm(x_concat.data.cpu()), sample_path, nrow=1, padding=0)
        print('Saved real and fake images into {}...'.format(sample_path))

Output:
The generator self.G is called on each element of c_fixed_list to generate images. All results are concatenated, then saved using torchvision.utils.save_image.
I don't see what's holding you from saving the images inside the loop. Something that would resemble:
for j, c_fixed in enumerate(c_fixed_list):
    x_fake = self.G(x_fixed, c_fixed)
    for k in range(len(x_fake)):
        sample_path = os.path.join(self.sample_dir, f'{i+1}-{k}-feat{j}-image.jpg')
        save_image(self.denorm(x_fake.data[k].cpu()), sample_path, nrow=1, padding=0)


Instruction: AllenNLP 2.0: Can't get FBetaMultiLabelMeasure to run

Input:
I would like to compute the f1-score for a classifier trained with allen-nlp. I used the working code from a allen-nlp guide, which computed accuracy, not F1, so I tried to adjust the metric in the code.
According to the documentation, CategoricalAccuracy and FBetaMultiLabelMeasure take the same inputs. (predictions: torch.Tensor of shape [batch_size, ..., num_classes], gold_labels: torch.Tensor of shape [batch_size, ...])
But for some reason the input that worked perfectly well for the accuracy results in a RuntimeError when given to the f1-multi-label metric.
I condensed the problem to the following code snippet:
>>> from allennlp.training.metrics import CategoricalAccuracy, FBetaMultiLabelMeasure
>>> import torch
>>> labels = torch.LongTensor([0, 0, 2, 1, 0])
>>> logits = torch.FloatTensor([[ 0.0063, -0.0118,  0.1857], [ 0.0013, -0.0217,  0.0356], [-0.0028, -0.0512,  0.0253], [-0.0460, -0.0347,  0.0400], [-0.0418,  0.0254,  0.1001]])
>>> labels.shape
torch.Size([5])
>>> logits.shape
torch.Size([5, 3])
>>> ca = CategoricalAccuracy()
>>> f1 = FBetaMultiLabelMeasure()
>>> ca(logits, labels)
>>> f1(logits, labels)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".../lib/python3.8/site-packages/allennlp/training/metrics/fbeta_multi_label_measure.py", line 130, in __call__
true_positives = (gold_labels * threshold_predictions).bool() & mask & pred_mask
RuntimeError: The size of tensor a (5) must match the size of tensor b (3) at non-singleton dimension 1

Why is this error happening? What am I missing here?

Output:
You want to use FBetaMeasure, not FBetaMultiLabelMeasure. "Multilabel" means you can specify more than one correct answer, but "Categorical Accuracy" only allows one correct answer. That means you have to specify another dimension in your labels.
I suspect the documentation of FBetaMultiLabelMeasure is misleading. I'll look into fixing it.


Instruction: Pyorch: Applying a batch of filters (kernels) on one single picture using conv2d

Input:
I have a batch of filters, i.e., w, whose size is torch.Size([64, 3, 7, 7]) as follows:

Also, I have a picture p from Imagenet as follows:

How can I apply the filters to the picture and get a grid of 64x64 where each cell contains the same picture on which a different filter has been applied? I would like to make the grid using torchvision.utils.make_grid but do not know how?
My try
y = F.conv2d(p, w)

The size of y is torch.Size([1, 64, 250, 250]) which does not make sense to me.

Output:
Each of your filters has size [3, 7, 7], so they would take an RGB image and produce a single channel output which is stacked in the channel dimension so your output [1, 64, H, W] makes perfect sense.
To visualize these filters:
import torch
import torch.nn as nn
import torch.nn.functional as F
import torchvision
from torchvision import transforms
from PIL import Image

import matplotlib.pyplot as plt

torch.random.manual_seed(42)

transform = transforms.Compose([transforms.ToTensor()])
img = transform(Image.open('dog.jpg')).unsqueeze(0)
print('Image size: ', img.shape)

filters = torch.randn(64, 3, 7, 7)

out = F.conv2d(img, filters)
print('Output size: ', out.shape)

list_of_images = [out[:,i] for i in range(64)]

grid = torchvision.utils.make_grid(list_of_images, normalize=True)
plt.imshow(grid.numpy().transpose(1,2,0))
plt.show()

This is a more accurate representation of the output. It is however not very attractive -- we can obtain the colored version by processing each color channel independently. (The grayscale version can be obtained by summing over the color channels)
color_out = []
for i in range(3):
    color_out.append(F.conv2d(img[:,i:i+1], filters[:,i:i+1]))
out = torch.stack(color_out, 2)
print('Output size: ', out.shape)

list_of_images = [out[0,i] for i in range(64)]
print(list_of_images[0].shape)

grid = torchvision.utils.make_grid(list_of_images, normalize=True)
plt.imshow(grid.numpy().transpose(1,2,0))
plt.show()


Instruction: Modifying the Learning Rate in the middle of the Model Training in Deep Learning

Input:
Below is the code to configure TrainingArguments consumed from the HuggingFace transformers library to finetune the GPT2 language model.
training_args = TrainingArguments(
        output_dir="./gpt2-language-model", #The output directory
        num_train_epochs=100, # number of training epochs
        per_device_train_batch_size=8, # batch size for training #32, 10
        per_device_eval_batch_size=8,  # batch size for evaluation #64, 10
        save_steps=100, # after # steps model is saved
        warmup_steps=500,# number of warmup steps for learning rate scheduler
        prediction_loss_only=True,
        metric_for_best_model = "eval_loss",
        load_best_model_at_end = True,
        evaluation_strategy="epoch",
        learning_rate=0.00004, # learning rate
    )

early_stop_callback = EarlyStoppingCallback(early_stopping_patience  = 3)

trainer = Trainer(
        model=gpt2_model,
        args=training_args,
        data_collator=data_collator,
        train_dataset=train_dataset,
        eval_dataset=test_dataset,
        callbacks = [early_stop_callback],
 )

The number of epochs as 100 and learning_rate as 0.00004 and also the early_stopping is configured with the patience value as 3.
The model ran for 5/100 epochs and noticed that the difference in loss_value is negligible. The latest checkpoint is saved as checkpoint-latest.
Now Can I modify the learning_rate may be to 0.01 from 0.00004 and resume the training from the latest saved checkpoint - checkpoint-latest? Doing that will be efficient?
Or to train with the new learning_rate value should I start the training from the beginning?

Output:
No, you don't have to restart your training.
Changing the learning rate is like changing how big a step your model take in the direction determined by your loss function.
You can also think of it as transfer learning where the model has some experience (no matter how little or irrelevant) and the weights are in a state most likely better than a randomly initialised one.
As a matter of fact, changing the learning rate mid-training is considered an art in deep learning and you should change it if you have a very very good reason to do it.
You would probably want to write down when (why, what, etc) you did it if you or someone else wants to "reproduce" the result of your model.


Instruction: Pytorch: why does torch.where method does not work like numpy.where?

Input:
In order to replace positive values with a certain number and negative ones with another number in a random vector using Numpy one can do the following:
npy_p = np.random.randn(4,6)
quant = np.where(npy_p>0, c_plus , np.where(npy_p<0, c_minus , npy_p))

However where method in Pytorch throws out the following error:

expected scalar type double but found float

Can you help me with that?

Output:
I can't reproduce this error, maybe it will be better if you could share a specific example where it failed (it might be the values you try to fill the tensor with):
import torch
x = torch.rand(4,6)
res = torch.where(x > 0.3,torch.tensor(0.), torch.where(x < 0.1, torch.tensor(-1.), x))

Where x is and it's of dtype float32:
tensor([[0.1391, 0.4491, 0.2363, 0.3215, 0.7740, 0.4879],
        [0.3051, 0.0870, 0.2869, 0.2575, 0.8825, 0.8201],
        [0.4419, 0.1138, 0.0825, 0.9489, 0.1553, 0.6505],
        [0.8376, 0.7639, 0.9291, 0.0865, 0.5984, 0.3953]])

And the res is:
tensor([[ 0.1391,  0.0000,  0.2363,  0.0000,  0.0000,  0.0000],
        [ 0.0000, -1.0000,  0.2869,  0.2575,  0.0000,  0.0000],
        [ 0.0000,  0.1138, -1.0000,  0.0000,  0.1553,  0.0000],
        [ 0.0000,  0.0000,  0.0000, -1.0000,  0.0000,  0.0000]])

The problem is caused because you mix data types in the torch.where, if you explicitly use the same datatype as the tensor in your constants it works fine.


Instruction: Difference between versions 9.2,10.1,10.2,11.0 of cuda for PyTorch 1.7

Input:
Recently,I need to install pytorch ,when I check out the website :
install pytorch website
It shows four different version 9.2,10.1,10.2,11.0 to choose ,And I have cuda version 10.0 and driver version 450 installed on my computer,I thought it would fail to enable gpu when using pytorch
,After I choose 10.1 and  try torch.cuda.is_available() and it returns True
I have two questions:

Why does everything turn out to be working even my cuda version is not the same as any of one I mentioned ?

What's the difference between choosing cuda verison 9.2,10.1,10.2,11.0 ?

Output:
PyTorch doesn't use the system's CUDA installation when installed from a package manager (either conda or pip). Instead, it comes with a copy of the CUDA runtime and will work as long as your system is compatible with that version of PyTorch. By compatible I mean that the GPU supports the particular version of CUDA and the GPU's compute capability is one that the PyTorch binaries (for the selected version) are compiled with support for.
Therefore the version reported by nvcc (the version installed on the system) is basically irrelevant. The version you should be looking at is
import torch
# print the version of CUDA being used by pytorch
print(torch.version.cuda)

The only time the system's version of CUDA should matter is if you compiled PyTorch from source.
As for which version of CUDA to select. You will probably want the newest version of CUDA that your system is compatible with. This is because newer versions generally include performance improvements compared to older versions.


Instruction: Pytorch object detection model optimization

Input:
I want to reduce the object detection model size. For the same, I tried optimising Faster R-CNN model for object detection using pytorch-mobile optimiser, but the .pt zip file generated is of the same size as that of the original model size.
I used the code mention below
import torch
import torchvision
from torch.utils.mobile_optimizer import optimize_for_mobile

model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)

model.eval()
script_model = torch.jit.script(model)
from torch.utils.mobile_optimizer import optimize_for_mobile
script_model_vulkan = optimize_for_mobile(script_model, backend='Vulkan')
torch.jit.save(script_model_vulkan, "frcnn.pth")

Output:
You have to quantize your model first
follow these steps here
& then use these methods
from torch.utils.mobile_optimizer import optimize_for_mobile
script_model_vulkan = optimize_for_mobile(script_model, backend='Vulkan')
torch.jit.save(script_model_vulkan, "frcnn.pth")

EDIT:
Quantization process for resnet50 model
import torchvision
model = torchvision.models.resnet50(pretrained=True)
import os
import torch

def print_model_size(mdl):
    torch.save(mdl.state_dict(), "tmp.pt")
    print("%.2f MB" %(os.path.getsize("tmp.pt")/1e6))
    os.remove('tmp.pt')
print_model_size(model) # will print original model size
backend = "qnnpack"
model.qconfig = torch.quantization.get_default_qconfig(backend)
torch.backends.quantized.engine = backend
model_static_quantized = torch.quantization.prepare(model, inplace=False)
model_static_quantized = torch.quantization.convert(model_static_quantized, inplace=False)

print_model_size(model_static_quantized) ## will print quantized model size


Instruction: Difference between torch.flatten() and nn.Flatten()

Input: What are the differences between torch.flatten() and torch.nn.Flatten()?

Output:
Flattening is available in three forms in PyTorch

As a tensor method (oop style) torch.Tensor.flatten applied directly on a tensor: x.flatten().

As a function (functional form) torch.flatten applied as: torch.flatten(x).

As a module (layer nn.Module) nn.Flatten(). Generally used in a model definition.

All three are identical and share the same implementation, the only difference being nn.Flatten has start_dim set to 1 by default to avoid flattening the first axis (usually the batch axis). While the other two flatten from axis=0 to axis=-1 - i.e. the entire tensor - if no arguments are given.


Instruction: IndexError: Dimension out of range - PyTorch dimension expected to be in range of [-1, 0], but got 1

Input:
Despite already numerous answers on this very topic, failing to see in the example below (extract from https://gist.github.com/lirnli/c16ef186c75588e705d9864fb816a13c on Variational Recurrent Networks) which input and output dimensions trigger the error.
Having tried to change dimensions in torch.cat and also suppress the call to squeeze(),  the error persists,
<ipython-input-51-cdc928891ad7> in generate(self, hidden, temperature)
     56         x_sample = x = x_out.div(temperature).exp().multinomial(1).squeeze()
     57         x = self.phi_x(x)
---> 58         tc = torch.cat([x,z], dim=1)
     59
     60         hidden_next = self.rnn(tc,hidden)

IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

Thus how to shape the dimensions in x and z in tc = torch.cat([x,z], dim=1)?
Note the code as follows,
import torch
from torch import nn, optim
from torch.autograd import Variable

class VRNNCell(nn.Module):
    def __init__(self):
        super(VRNNCell,self).__init__()
        self.phi_x = nn.Sequential(nn.Embedding(128,64), nn.Linear(64,64), nn.ELU())
        self.encoder = nn.Linear(128,64*2) # output hyperparameters
        self.phi_z = nn.Sequential(nn.Linear(64,64), nn.ELU())
        self.decoder = nn.Linear(128,128) # logits
        self.prior = nn.Linear(64,64*2) # output hyperparameters
        self.rnn = nn.GRUCell(128,64)

    def forward(self, x, hidden):
        x = self.phi_x(x)
        # 1. h => z
        z_prior = self.prior(hidden)
        # 2. x + h => z
        z_infer = self.encoder(torch.cat([x,hidden], dim=1))
        # sampling
        z = Variable(torch.randn(x.size(0),64))*z_infer[:,64:].exp()+z_infer[:,:64]
        z = self.phi_z(z)
        # 3. h + z => x
        x_out = self.decoder(torch.cat([hidden, z], dim=1))
        # 4. x + z => h
        hidden_next = self.rnn(torch.cat([x,z], dim=1),hidden)
        return x_out, hidden_next, z_prior, z_infer

    def calculate_loss(self, x, hidden):
        x_out, hidden_next, z_prior, z_infer = self.forward(x, hidden)
        # 1. logistic regression loss
        loss1 = nn.functional.cross_entropy(x_out, x)
        # 2. KL Divergence between Multivariate Gaussian
        mu_infer, log_sigma_infer = z_infer[:,:64], z_infer[:,64:]
        mu_prior, log_sigma_prior = z_prior[:,:64], z_prior[:,64:]
        loss2 = (2*(log_sigma_infer-log_sigma_prior)).exp() \
                + ((mu_infer-mu_prior)/log_sigma_prior.exp())**2 \
                - 2*(log_sigma_infer-log_sigma_prior) - 1
        loss2 = 0.5*loss2.sum(dim=1).mean()
        return loss1, loss2, hidden_next

    def generate(self, hidden=None, temperature=None):
        if hidden is None:
            hidden=Variable(torch.zeros(1,64))
        if temperature is None:
            temperature = 0.8
        # 1. h => z
        z_prior = self.prior(hidden)
        # sampling
        z = Variable(torch.randn(z_prior.size(0),64))*z_prior[:,64:].exp()+z_prior[:,:64]
        z = self.phi_z(z)
        # 2. h + z => x
        x_out = self.decoder(torch.cat([hidden, z], dim=1))
        # sampling
        x_sample = x = x_out.div(temperature).exp().multinomial(1).squeeze()
        x = self.phi_x(x)
        # 3. x + z => h
        # hidden_next = self.rnn(torch.cat([x,z], dim=1),hidden)
        tc = torch.cat([x,z], dim=1)
        hidden_next = self.rnn(tc,hidden)
        return x_sample, hidden_next

    def generate_text(self, hidden=None,temperature=None, n=100):
        res = []
        hidden = None
        for _ in range(n):
            x_sample, hidden = self.generate(hidden,temperature)
            res.append(chr(x_sample.data[0]))
        return "".join(res)

# Test
net = VRNNCell()
x = Variable(torch.LongTensor([12,13,14]))
hidden = Variable(torch.rand(3,64))
output, hidden_next, z_infer, z_prior = net(x, hidden)
loss1, loss2, _ = net.calculate_loss(x, hidden)
loss1, loss2

hidden = Variable(torch.zeros(1,64))
net.generate_text()

Output:
The error

IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

means that you're trying to access an index that doesn't exist in the tensor. For instance, the following code would cause the same IndexError you're experiencing.
# sample input tensors
In [210]: x = torch.arange(4)
In [211]: z = torch.arange(6)

# trying to concatenate along the second dimension
# but the tensors have only one dimension (i.e., `0`).

In [212]: torch.cat([x, z], dim=1)

So, one way to overcome this is to promote the tensors to higher dimensions before concatenation, if that is what you need.
# promoting tensors to 2D before concatenation
In [216]: torch.cat([x[None, :], z[None, :]], dim=1)
Out[216]: tensor([[0, 1, 2, 3, 0, 1, 2, 3, 4, 5]])

Thus, in your case, you've to analyze and understand what shape you need for x so that it can be concatenated with z along dimension 1 and then the tc passed as input to self.rnn() along with hidden.
As far as I can see, x[None, :] , z[None, :] should work.

Debugging for successful training
The code you posted has been written for PyTorch v0.4.1. A lot has changed in the PyTorch Python API since then, but the code was not updated.
Below are the changes you need to make the code run and train successfully. Copy the below functions and paste it at appropriate places in your code.
def generate(self, hidden=None, temperature=None):
        if hidden is None:
            hidden=Variable(torch.zeros(1,64))
        if temperature is None:
            temperature = 0.8
        # 1. h => z
        z_prior = self.prior(hidden)
        # sampling
        z = Variable(torch.randn(z_prior.size(0),64))*z_prior[:,64:].exp()+z_prior[:,:64]
        z = self.phi_z(z)
        # 2. h + z => x
        x_out = self.decoder(torch.cat([hidden, z], dim=1))
        # sampling
        x_sample = x = x_out.div(temperature).exp().multinomial(1).squeeze()
        x = self.phi_x(x)
        # 3. x + z => h
        x = x[None, ...]   # changed here
        xz = torch.cat([x,z], dim=1)  # changed here
        hidden_next = self.rnn(xz,hidden) # changed here
        return x_sample, hidden_next

def generate_text(self, hidden=None,temperature=None, n=100):
        res = []
        hidden = None
        for _ in range(n):
            x_sample, hidden = self.generate(hidden,temperature)
            res.append(chr(x_sample.data))      # changed here
        return "".join(res)

for epoch in range(max_epoch):
    batch = next(g)
    loss_seq = 0
    loss1_seq, loss2_seq = 0, 0
    optimizer.zero_grad()
    for x in batch:
        loss1, loss2, hidden = net.calculate_loss(Variable(x),hidden)
        loss1_seq += loss1.data  # changed here
        loss2_seq += loss2.data  # changed here
        loss_seq = loss_seq + loss1+loss2
    loss_seq.backward()
    optimizer.step()
    hidden.detach_()
    if epoch%100==0:
        print('>> epoch {}, loss {:12.4f}, decoder loss {:12.4f}, latent loss {:12.4f}'.format(epoch, loss_seq.data, loss1_seq, loss2_seq))  # changed here
        print(net.generate_text())
        print()

Note: After these changes, the training loop at my end proceeds without any errors on PyTorch v1.7.1. Have a look at the comments with # changed here to understand the changes.


Instruction: How to add a L1 or L2 regularization to weights in pytorch

Input:
In tensorflow, we can add a L1 or L2 regularizations in the sequential model. I couldn't find equivalent approach in pytorch. How can we add regularizations to weights in pytorch in the definition of the net:
class Net(torch.nn.Module):
    def __init__(self, n_feature, n_hidden, n_output):
        super(Net, self).__init__()
        self.hidden = torch.nn.Linear(n_feature, n_hidden)   # hidden layer
        """ How to add a L1 regularization after a certain hidden layer?? """
        """ OR How to add a L1 regularization after a certain hidden layer?? """
        self.predict = torch.nn.Linear(n_hidden, n_output)   # output layer

    def forward(self, x):
        x = F.relu(self.hidden(x))      # activation function for hidden layer
        x = self.predict(x)             # linear output
        return x

net = Net(n_feature=1, n_hidden=10, n_output=1)     # define the network
# print(net)  # net architecture
optimizer = torch.optim.SGD(net.parameters(), lr=0.2)
loss_func = torch.nn.MSELoss()  # this is for regression mean squared loss

Output:
Generally L2 regularization is handled through the weight_decay argument for the optimizer in PyTorch (you can assign different arguments for different layers too). This mechanism, however, doesn't allow for L1 regularization without extending the existing optimizers or writing a custom optimizer.
According to the tensorflow docs they use a reduce_sum(abs(x)) penalty for L1 regularization and a reduce_sum(square(x)) penalty for L2 regularization. Probably the easiest way to achieve this is to just directly add these penalty terms to the loss function used for gradient computation during training.
# set l1_weight and l2_weight to non-zero values to enable penalties

# inside the training loop (given input x and target y)
...
pred = net(x)
loss = loss_func(pred, y)

# compute penalty only for net.hidden parameters
l1_penalty = l1_weight * sum([p.abs().sum() for p in net.hidden.parameters()])
l2_penalty = l2_weight * sum([(p**2).sum() for p in net.hidden.parameters()])
loss_with_penalty = loss + l1_penalty + l2_penalty

optimizer.zero_grad()
loss_with_penalty.backward()
optimizer.step()

# The pre-penalty loss is the one we ultimately care about
print('loss:', loss.item())


Instruction: Why PyTorch fails to instantiate this neural network from Torch Hub?

Input:
I am on Ubuntu, running PyTorch 1.7.1 and torchvision 0.8.2:
print(torch.__version__)
print(torchvision.__version__)

>> 1.7.1
>> 0.8.2

When I attempt to instantiate resnet18 with:
model_resnet18 = torch.hub.load('pytorch/vision', 'resnet18', pretrained=True)

it fails with:
Using cache found in /home/stark/.cache/torch/hub/pytorch_vision_master
Traceback (most recent call last):
  File "/home/stark/Work/test/binary_classifier.py", line 253, in <module>
    build_and_train()
  File "/home/stark/Work/test/binary_classifier.py", line 174, in build_and_train
    model_resnet18 = torch.hub.load('pytorch/vision', 'resnet18', pretrained=True)
  File "/home/stark/anaconda3/envs/torch-env/lib/python3.8/site-packages/torch/hub.py", line 370, in load
    model = _load_local(repo_or_dir, model, *args, **kwargs)
  File "/home/stark/anaconda3/envs/torch-env/lib/python3.8/site-packages/torch/hub.py", line 396, in _load_local
    hub_module = import_module(MODULE_HUBCONF, hubconf_path)
  File "/home/stark/anaconda3/envs/torch-env/lib/python3.8/site-packages/torch/hub.py", line 71, in import_module
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/stark/.cache/torch/hub/pytorch_vision_master/hubconf.py", line 14, in <module>
    from torchvision.models.mobilenetv2 import mobilenet_v2
ModuleNotFoundError: No module named 'torchvision.models.mobilenetv2'

I am on Python 3.8 and already on the latest PyTorch version, so what else can I do to fix this?
Any pointers are greatly appreciated!

Output:
You could use torchvision's API instead:
model_resnet18 = torchvision.models.resnet18(pretrained=True)


Instruction: Get a 10x10 patch from a 100x100 pytorch tensor with torus style wrap around the boundries

Input:
How can I get a 10x10 patch from a 100x100 pytorch tensor, with the added constraint that if a patch would go outside the boundaries of the array, then it wraps around the edges (as if the array was a torus, with the top joined to the bottom, and the left joined to the right)?
I wrote this code that will do the job, I'm looking for something more elegant, efficient and clear:
def shift_matrix(a, distances) -> Tensor:
  x, y = distances
  a = torch.cat((a[x:], a[0:x]), dim=0)
  a = torch.cat((a[:, y:], a[:, :y]), dim=1)
  return a

def randomly_shift_matrix(a) -> Tensor:
  return shift_matrix(a, np.random.randint(low = 0, high = a.size()))

def random_patch(a, size) -> Tensor:
  full_shifted_matrix = randomly_shift_matrix(a)
  return full_shifted_matrix[0:size[0], 0:size[1]]

I feel like something with negative index slices should work.  I haven't found it though.
You can see the code in google colab here.

Output:
You are looking for torch.roll
def random_patch(a, size) -> Tensor:
  shifts = np.random.randint(low = 0, high = a.size())
  return torch.roll(a, shifts=shifts, dims=(0, 1))[:size[0], :size[1]]


Instruction: Add a layer to model state dictionary in PyTorch

Input:
I have a neural network in PyTorch like,
class Net1(nn.Module):

    def __init__(self):
        super(Net, self).__init__()

        self.conv_g = ResBlock(2,81)
        self.conv_low1 = ResBlock(2,90)
        self.conv_low2 = ResBlock(2,90)
        self.conv_low3 = ResBlock(2,90)
        self.conv_low4 = ResBlock(2,90)
        self.conv_low5 = ResBlock(2,90)
        self.conv_low6 = ResBlock(2,90)
        self.conv_low7 = ResBlock(2,90)
        self.conv_low8 = ResBlock(2,90)
        self.conv_low9 = ResBlock(2,90)

        self.stagIII = stageIII()

    def forward(self,x):
        ...

        return out

To this network I have added a new layer m whose weights I know.
class Net2(nn.Module):

    def __init__(self):
        super(Net, self).__init__()

        self.m = nn.Linear(5,1)

        self.conv_g = ResBlock(2,81)
        self.conv_low1 = ResBlock(2,90)
        self.conv_low2 = ResBlock(2,90)
        self.conv_low3 = ResBlock(2,90)
        self.conv_low4 = ResBlock(2,90)
        self.conv_low5 = ResBlock(2,90)
        self.conv_low6 = ResBlock(2,90)
        self.conv_low7 = ResBlock(2,90)
        self.conv_low8 = ResBlock(2,90)
        self.conv_low9 = ResBlock(2,90)

        self.stagIII = stageIII()

    def forward(self,x):
        ...

        return out

The state dictionary of m.state_dict() is
OrderedDict([('weight', tensor([[ 1.0000,  2.0000,  3.0000,  4.0000, 5.0000]])), ('bias', tensor([0.]))])

Now given that I have Net1.state_dict() and m.state_dict(), how can I append m dictionary to Net1 dictionary to obtain state dictionary of Net2?

Output:
You can manually add them:
net2 = Net2()
net2.load_state_dict(net1_state_dict,strict=False)  # load what you can from the state_dict of Net1
net2.m.load_state_dict(m_state_dict)   # load sub module
# save the entire one for future use
torch.save(net2.state_dict(), 'merged_net2.pth.tar')