Instruction: After finetuning Faster RCNN object detection model, how to visualize bbox prediction?

Input:
I finetuned pytorch torchvision model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True) on my own custom dataset.

I followed this guide https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html#torchvision-object-detection-finetuning-tutorial but only trained Faster RCNN, not Mask RCNN.

I successfully finished training with no error, and the model returned a dict containing predicted boxes, labels, and scores.

In the guide I followed, they show how to visualize masks predicted by the model trained.
Is there a similar method to visualize bounding box? I'm having a lot of trouble figuring this out.

Thank you

Output:
The prediction from FasterRCNN is of the form:

>>> predictions = model([input_img_tensor])
[{'boxes': tensor([[419.6865, 170.0683, 536.0842, 493.7452],
          [159.0727, 180.3606, 298.8194, 434.4604],
          [439.7836, 222.6208, 452.0138, 271.8359],
          [444.3562, 224.4628, 456.1511, 265.5336],
          [437.7808, 226.5965, 446.2904, 271.2691]], grad_fn=<StackBackward>),
  'labels': tensor([ 1,  1, 32, 32, 32]),
  'scores': tensor([0.9997, 0.9996, 0.5827, 0.2102, 0.0943], grad_fn=<IndexBackward>)}]

where the predicted boxes are of [x1, y1, x2, y2] format, with values between 0 and H and 0 and W.

You can use OpenCV's rectangle function to overlay bounding boxes on image.

import cv2
img = cv2.imread('input_iamge.png', cv2.COLOR_BGR2RGB)

for i in range(len(predictions[0]['boxes'])):
    x1, x2, x3, x4 = map(int, predictions[0]['boxes'][i].tolist())
    print(x1, x2, x3, x4)
    image = cv2.rectangle(img, (x1, x2), (x3, x4), (255, 0, 0), 1)

cv2_imshow('img', image)


Instruction: How to calculate a Forward Pass with matrix operations in PyTorch?

Input:
I have got an input x, layer 1 weight matrix, and layer 2 weight matrix.

Now I want to calculate the output of this pre-trained neural network via hand:

x * weights1 * weights2

While doing this I receive a RuntimeError: The size of tensor a (6) must match the size of tensor b (4) at non-singleton dimension 1

class Net(nn.Module):

def __init__(self):
    super().__init__()
    self.fc1 = nn.Linear(4,6)
    self.fc2 = nn.Linear(6,2)
    self.fc3 = nn.Linear(2,1)

def forward(self, x):
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))
    x = F.relu(self.fc3(x))
    return x

net = Net()
X = torch.randn(1000,4)
net.fc2.weight*(net.fc1.weight * X[0])

Output:
You are confusing element-wise multiplication (* operator) with matrix multiplication (@ operator).
Try:

net.fc2.weight @ (net.fc1.weight @ X[0])


Instruction: Cuda and pytorch memory usage

Input:
I am using Cuda and Pytorch:1.4.0.

When I try to increase batch_size, I've got the following error:

CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 4.00 GiB total capacity;  2.74 GiB already allocated; 7.80 MiB free; 2.96 GiB reserved in total by PyTorch)

I haven't found anything about Pytorch memory usage.

Also, I don't understand why I have only 7.80 mib available?

Should I just use a videocard with better perfomance, or can I free some memory? FYI, I have a GTX 1050 TI, python 3,7 and torch==1.4.0 and my os is Windows 10.

Output:
I had the same problem, the following worked for me:

torch.cuda.empty_cache()
# start training from here

Even after this if you get the error, then you should decrease the batch_size


Instruction: Is there any tool to profile each layer of a DNN in Pytorch and Tensorflow?

Input:
I want to know the execution time of each layer of a DNN.
Is there any tool can profile each layer?
I know I can insert some print in a model, but I am not sure it will be accurate.

Output:
To check the size and parameters of each layer you can use the torch summary.

To compute time the execution time of each layer, you can use the torchprof.

I don't know any project that merges both libs. Maybe it's your opportunity (lol)


Instruction: Extracting the encoded representations of an image in PyTorch?

Input:
I have a template autoencoder neural net model made in PyTorch that I'm using on the Omniglot dataset. I'd like to extract the encoded representations of the image, but I'm unsure how.

# Load data
mean = 0.5
std = 0.5
batch_size = 128
img_transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((mean,), (std,))
])
dataset = Omniglot('.', download=True, transform=img_transform)
dataloader = DataLoader(dataset, batch_size=batch_size, shuffle=True)

# Define Autoencoder
class Autoencoder(nn.Module):

    def __init__(self, n=64):
        super(Autoencoder, self).__init__()
        self.encoder = nn.Sequential(
            nn.Linear(105*105, 256, bias=True),
            nn.ReLU(True),
            nn.Linear(256, 64, bias=True),
            nn.ReLU(True),
            nn.Linear(64, n, bias=True),
            nn.ReLU(True)
        )
        self.decoder = nn.Sequential(
            nn.Linear(n, 64, bias=True),
            nn.ReLU(True),
            nn.Linear(64, 256, bias=True),
            nn.ReLU(True),
            nn.Linear(256, 105*105, bias=True),
            nn.Tanh()
        )

    def forward(self, x):
        x = self.encoder(x)
        x = self.decoder(x)
        return x

def train(num_epochs, dataloader, model, criterion, optimizer):
    for epoch in range(num_epochs):
        for data in dataloader:
            img, label = data
            img = img.view(img.size(0), -1)
            img = Variable(img).cuda()
            output = model(img)
            loss = criterion(output, img)
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
    return model

# Train model
num_epochs = 25
learning_rate = 1e-3
model = Autoencoder().cuda()
criterion = nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
trained_model = train(num_epochs, dataloader, model, criterion, optimizer)

Output:
You can simply return the encoded output in the forward function as follows:

class Autoencoder(nn.Module):
...
    def forward(self, x):
        x = self.encoder(x)
        encoded_x = x
        x = self.decoder(x)
        return x, encoded_x

Modify the training function a little bit:

output, encoded_output = model(img)

OR you can simply call encoder:

encoded_output = model.encoder(img)


Instruction: Pytorch custom Dataset class giving wrong output

Input:
I am trying to use this class I built for a dataset but it saying that it should be a PIL or ndarray. Im not quite sure whats wrong with it.  Here is the class that I am using

class RotateDataset(Dataset):
    def __init__(self, image_list, size,transform = None):
        self.image_list = image_list
        self.size = size
        self.transform = transform
    def __len__(self):
        return len(self.image_list)
    def __getitem__(self, idx):

        img = cv2.imread(self.image_list[idx])
        image_height, image_width = img.shape[:2]
        print("ID: ", idx)
        if idx % 2 == 0:
            label = 0 # Set label
            # chose negative or positive rotation
            rotation_degree = random.randrange(35, 50, 1)
            posnegrot = np.random.randint(2)
            if posnegrot == 0:
                #positive rotation
                #rotation_matrix = cv2.getRotationMatrix2D((num_cols/2, num_rows/2), rotation_degree, 1)
                #img = cv2.warpAffine(img, rotation_matrix, (num_cols, num_rows))

                img = rotate_image(img, rotation_degree)
                img = crop_around_center(img, *largest_rotated_rect(image_width,
                                                                image_height,
                                                                math.radians(rotation_degree)))
            else:
                # Negative rotation
                rotation_degree = -rotation_degree
                img = crop_around_center(img, *largest_rotated_rect(image_width,
                                                                image_height,
                                                                math.radians(rotation_degree)))

        else:
           label = 1
        img = cv2.resize(img, self.size, cv2.INTER_AREA)
        return self.transform(img), self.transform(label)

The error that it is giving me is

TypeError: pic should be PIL Image or ndarray. Got class 'int'

It should give me a img (tensor) and a label (tensor)
but I dont think it is doing it correctly.

TypeError                                 Traceback (most recent call last)
<ipython-input-34-f47943b2600c> in <module>
      2     train_loss = 0.0
      3     net.train()
----> 4     for image, label in enumerate(train_loader):
      5         if train_on_gpu:
      6             image, label = image.cuda(), label.cuda()

~\Anaconda3\envs\TF2\lib\site-packages\torch\utils\data\dataloader.py in __next__(self)
    343
    344     def __next__(self):
--> 345         data = self._next_data()
    346         self._num_yielded += 1
    347         if self._dataset_kind == _DatasetKind.Iterable and \

~\Anaconda3\envs\TF2\lib\site-packages\torch\utils\data\dataloader.py in _next_data(self)
    383     def _next_data(self):
    384         index = self._next_index()  # may raise StopIteration
--> 385         data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
    386         if self._pin_memory:
    387             data = _utils.pin_memory.pin_memory(data)

~\Anaconda3\envs\TF2\lib\site-packages\torch\utils\data\_utils\fetch.py in fetch(self, possibly_batched_index)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

~\Anaconda3\envs\TF2\lib\site-packages\torch\utils\data\_utils\fetch.py in <listcomp>(.0)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

<ipython-input-28-6c77357ff619> in __getitem__(self, idx)
     35             label = 1
     36         img = cv2.resize(img, self.size, cv2.INTER_AREA)
---> 37         return self.transform(img), self.transform(label)

~\Anaconda3\envs\TF2\lib\site-packages\torchvision\transforms\transforms.py in __call__(self, pic)
     99             Tensor: Converted image.
    100         """
--> 101         return F.to_tensor(pic)
    102
    103     def __repr__(self):

~\Anaconda3\envs\TF2\lib\site-packages\torchvision\transforms\functional.py in to_tensor(pic)
     53     """
     54     if not(_is_pil_image(pic) or _is_numpy(pic)):
---> 55         raise TypeError('pic should be PIL Image or ndarray. Got {}'.format(type(pic)))
     56
     57     if _is_numpy(pic) and not _is_numpy_image(pic):

TypeError: pic should be PIL Image or ndarray. Got <class 'int'>

Output:
As discussed in the comments, the problem was applying transform on label as well. The label should instead simply be written as tensor:

return self.transform(img), torch.tensor(label)


Instruction: How are neural networks, loss and optimizer connected in PyTorch?

Input:
I've seen answers to this question, but I still don't understand it at all. As far as I know, this is the most basic setup:

net = CustomClassInheritingFromModuleWithDefinedInitAndForward()
criterion = nn.SomeLossClass()
optimizer = optim.SomeOptimizer(net.parameters(), ...)
for _, data in enumerate(trainloader, 0):
    inputs, labels = data
    optimizer.zero_grad()
    outputs = net(inputs)
    loss = criterion(outputs, labels)
    loss.backward()
    optimizer.step()

What I don't understand is:

Optimizer is initialized with net.parameters(), which I thought are internal weights of the net.

Loss does not access these parameters nor the net itself. It only has access to net's outputs and input labels.

Optimizer does not access loss either.

So if loss only works on outputs and optimizer only on net.parameters, how can they be connected?

Output:
  Optimizer is initialized with net.parameters(), which I thought are internal weights of the net.

This is because the optimizer will modify the parameters of your net during the training.

  Loss does not access these parameters nor the net itself. It only has access to net's outputs and input labels.

The loss only computes an error between a prediction and the truth.

  Optimizer does not access loss either.

It accesses the tensors that were computed during loss.backward


Instruction: How to feed the output of a finetuned bert model as inpunt to another finetuned bert model?

Input:
I finetuned two separate bert model (bert-base-uncased) on sentiment analysis and pos tagging tasks. Now, I want to feed the output of the pos tagger (batch, seqlength, hiddensize) as input to the sentiment model.The original bert-base-uncased model is in 'bertModel/' folder which contains 'model.bin' and 'config.json'. Here is my code:

class DeepSequentialModel(nn.Module):
def __init__(self, sentiment_model_file, postag_model_file, device):
    super(DeepSequentialModel, self).__init__()

    self.sentiment_model = SentimentModel().to(device)
    self.sentiment_model.load_state_dict(torch.load(sentiment_model_file, map_location=device))
    self.postag_model = PosTagModel().to(device)
    self.postag_model.load_state_dict(torch.load(postag_model_file, map_location=device))

    self.classificationLayer = nn.Linear(768, 1)

def forward(self, seq, attn_masks):
    postag_context = self.postag_model(seq, attn_masks)
    sent_context = self.sentiment_model(postag_context, attn_masks)
    logits = self.classificationLayer(sent_context)
    return logits

class PosTagModel(nn.Module):
def __init__(self,):
    super(PosTagModel, self).__init__()
    self.bert_layer = BertModel.from_pretrained('bertModel/')
    self.classificationLayer = nn.Linear(768, 43)

def forward(self, seq, attn_masks):
    cont_reps, _ = self.bert_layer(seq, attention_mask=attn_masks)
    return cont_reps

class SentimentModel(nn.Module):
def __init__(self,):
    super(SentimentModel, self).__init__()
    self.bert_layer = BertModel.from_pretrained('bertModel/')
    self.cls_layer = nn.Linear(768, 1)

def forward(self, input, attn_masks):
    cont_reps, _ = self.bert_layer(encoder_hidden_states=input, encoder_attention_mask=attn_masks)
    cls_rep = cont_reps[:, 0]
    return cls_rep

But I get the below error. I appreciate it if someone could help me. Thanks!

    cont_reps, _ = self.bert_layer(encoder_hidden_states=input, encoder_attention_mask=attn_masks)
    result = self.forward(*input, **kwargs)
    TypeError: forward() got an unexpected keyword argument 'encoder_hidden_states'

Output:
To formulate this as an answer, too, and keep it properly visible for future visitors, the forward() call of transformers does not support these arguments in version 2.1.1, or any earlier version, for that matter. note that the link in my comment is in fact pointing to a different forward function, but otherwise the point still holds.

Passing encoder_hidden_states to forward() was first possible in version 2.2.0.


Instruction: Use pytorch to bulid multi-linear,but the result i get is not what i want?

Input:
import sklearn
    from sklearn.datasets import load_boston
    import torch
    import torch.nn as nn

    boston = load_boston()

    num_epochs = 10

    linear_model = nn.Linear(13,1,bias = True)
    criterion = nn.MSELoss()
    optimizer = torch.optim.SGD(linear_model.parameters(), lr=0.01)

    for epoch in range(num_epochs):

        inputs = torch.from_numpy(boston.data)
        targets = torch.from_numpy(boston.target)
        targets = targets.float()
        inputs = inputs.float()

        outputs = linear_model(inputs)
        loss = criterion(outputs,targets)

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        print('epoch:{}/{} ...... loss:{:.4f}'.format(epoch,num_epochs,loss.item()))

Output:
The problem with your code seems to be data normalization, or, the lack of it.

I edited your code to add data normalization ((x - mean) / std), set epochs to 50 and switched the optimiser to Adam so it converges faster.

import sklearn
from sklearn.datasets import load_boston
import torch
import numpy as np
import torch.nn as nn

def normalize(X):
    mean = np.mean(X)
    std = np.std(X)
    return ((X - mean) / std), mean, std

boston = load_boston()

m_in = np.zeros(13)
s_in = np.zeros(13)

b_in = boston.data
b_out = boston.target

for i in range(13):
    b_in[:, i], m_in[i], s_in[i] = normalize(b_in[:, i])

b_out, m_out, s_out = normalize(b_out)

num_epochs = 50

linear_model = nn.Linear(13, 1, bias=True)
criterion = nn.MSELoss()
optimizer = torch.optim.Adam(linear_model.parameters(), lr=0.01)

linear_model.train()
for epoch in range(num_epochs):
    inputs = torch.from_numpy(b_in)
    targets = torch.from_numpy(b_out)
    targets = targets.float().unsqueeze(1)
    inputs = inputs.float()

    outputs = linear_model(inputs)
    loss = criterion(outputs, targets)

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    print('epoch: {}/{} ...... loss: {:.4f}'.format(epoch, num_epochs, loss.item()))

The new output looks like this:

epoch: 0/50 ...... loss: 1.2610
epoch: 1/50 ...... loss: 1.1395
epoch: 2/50 ...... loss: 1.0332
epoch: 3/50 ...... loss: 0.9421
epoch: 4/50 ...... loss: 0.8660
epoch: 5/50 ...... loss: 0.8040
epoch: 6/50 ...... loss: 0.7548
epoch: 7/50 ...... loss: 0.7164
epoch: 8/50 ...... loss: 0.6863
epoch: 9/50 ...... loss: 0.6623
epoch: 10/50 ...... loss: 0.6421
epoch: 11/50 ...... loss: 0.6240
epoch: 12/50 ...... loss: 0.6068
epoch: 13/50 ...... loss: 0.5895
epoch: 14/50 ...... loss: 0.5717
epoch: 15/50 ...... loss: 0.5532
epoch: 16/50 ...... loss: 0.5339
epoch: 17/50 ...... loss: 0.5143
epoch: 18/50 ...... loss: 0.4946
epoch: 19/50 ...... loss: 0.4752
epoch: 20/50 ...... loss: 0.4565
epoch: 21/50 ...... loss: 0.4389
epoch: 22/50 ...... loss: 0.4226
epoch: 23/50 ...... loss: 0.4079
epoch: 24/50 ...... loss: 0.3949
epoch: 25/50 ...... loss: 0.3835
epoch: 26/50 ...... loss: 0.3736
epoch: 27/50 ...... loss: 0.3652
epoch: 28/50 ...... loss: 0.3580
epoch: 29/50 ...... loss: 0.3517
epoch: 30/50 ...... loss: 0.3461
epoch: 31/50 ...... loss: 0.3410
epoch: 32/50 ...... loss: 0.3361
epoch: 33/50 ...... loss: 0.3315
epoch: 34/50 ...... loss: 0.3270
epoch: 35/50 ...... loss: 0.3226
epoch: 36/50 ...... loss: 0.3183
epoch: 37/50 ...... loss: 0.3142
epoch: 38/50 ...... loss: 0.3103
epoch: 39/50 ...... loss: 0.3068
epoch: 40/50 ...... loss: 0.3036
epoch: 41/50 ...... loss: 0.3008
epoch: 42/50 ...... loss: 0.2984
epoch: 43/50 ...... loss: 0.2963
epoch: 44/50 ...... loss: 0.2945
epoch: 45/50 ...... loss: 0.2930
epoch: 46/50 ...... loss: 0.2917
epoch: 47/50 ...... loss: 0.2904
epoch: 48/50 ...... loss: 0.2892
epoch: 49/50 ...... loss: 0.2881

The features in the dataset seem to have different ranges and this poses a problem for the learning algorithm. By normalizing the data, we make it much easier for the model to learn.

[1] This is a great article that explains why normalization is needed: https://towardsdatascience.com/understand-data-normalization-in-machine-learning-8ff3062101f0


Instruction: How can I solve "TypeError: max() received an invalid combination of arguments - got (Linear, int), but expected"?

Input:
How I solve this error, I am trying to train a ticket classification model

I'm trying to make a ticket sorter with the Pytorch library, but I have this error. I can't understand what I've done wrong
Can you help me?

data_transforms = {
    'train' : transforms.Compose([
        transforms.RandomResizedCrop(224),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
    'val' : transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ])

}

data_dir = 'dataset_billete_argentino'
image_datasets = {x: datasets.ImageFolder(os.path.join(data_dir, x),
                                          data_transforms[x])
                  for x in ['train', 'val']}
dataloaders = {x: torch.utils.data.DataLoader(image_datasets[x], batch_size=4,
                                              shuffle=True, num_workers=4)
               for x in ['train', 'val']}
dataset_sizes = {x: len(image_datasets[x]) for x in ['train', 'val']}
class_name = image_datasets['train'].classes

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3,6,5)
        self.pool = nn.MaxPool2d(2,2)
        self.conv2 = nn.Conv2d(6,16,5)
        self.fc1 = nn.Linear(16 * 53 * 53, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 2)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(x.size(0), 16* 53 * 53)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3
        return x

net = Net()

def train_model(model, criterion, optimizer, scheduler, num_epochs=25):
    since = time.time()

    best_model_wts = copy.deepcopy(model.state_dict())
    best_acc = 0.0

    for epoch in range(num_epochs):
        print('Epoch {}/{}'.format(epoch, num_epochs - 1))
        print('-' * 10)

        # Each epoch has a training and validation phase
        for phase in ['train', 'val']:
            if phase == 'train':
                model.train()  # Set model to training mode
            else:
                model.eval()   # Set model to evaluate mode

            running_loss = 0.0
            running_corrects = 0

            # Iterate over data.
            for inputs, labels in dataloaders[phase]:
                inputs = inputs.to(device)
                labels = labels.to(device)

                # zero the parameter gradients
                optimizer.zero_grad()

                # forward
                # track history if only in train
                with torch.set_grad_enabled(phase == 'train'):
                    outputs = model(inputs)
                    _, preds = torch.max(outputs, 1)
                    loss = criterion(outputs, labels)

                    # backward + optimize only if in training phase
                    if phase == 'train':
                        loss.backward()
                        optimizer.step()

                # statistics
                running_loss += loss.item() * inputs.size(0)
                running_corrects += torch.sum(preds == labels.data)
            if phase == 'train':
                scheduler.step()

            epoch_loss = running_loss / dataset_sizes[phase]
            epoch_acc = running_corrects.double() / dataset_sizes[phase]

            print('{} Loss: {:.4f} Acc: {:.4f}'.format(
                phase, epoch_loss, epoch_acc))

            # deep copy the model
            if phase == 'val' and epoch_acc > best_acc:
                best_acc = epoch_acc
                best_model_wts = copy.deepcopy(model.state_dict())

        print()

    time_elapsed = time.time() - since
    print('Training complete in {:.0f}m {:.0f}s'.format(
        time_elapsed // 60, time_elapsed % 60))
    print('Best val Acc: {:4f}'.format(best_acc))

    # load best model weights
    model.load_state_dict(best_model_wts)
    return model

from torch.optim import lr_scheduler

exp_lr_scheduler = lr_scheduler.StepLR(optimizer, step_size=7, gamma=0.1)

net = train_model(net, criterion, optimizer, exp_lr_scheduler,
                  num_epochs=25)

And give this error
TypeError: max() received an invalid combination of arguments - got (Linear, int), but expected one of:
 * (Tensor input)
 * (Tensor input, name dim, bool keepdim, tuple of Tensors out)
 * (Tensor input, Tensor other, Tensor out)
 * (Tensor input, int dim, bool keepdim, tuple of Tensors out)

Epoch 0/24
----------
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-27-29dfe3459d8a> in <module>
      4
      5 net = train_model(net, criterion, optimizer, exp_lr_scheduler,
----> 6                   num_epochs=25)

<ipython-input-19-1a5d4f162548> in train_model(model, criterion, optimizer, scheduler, num_epochs)
     31                 with torch.set_grad_enabled(phase == 'train'):
     32                     outputs = model(inputs)
---> 33                     _, preds = torch.max(outputs, 1)
     34                     loss = criterion(outputs, labels)
     35

TypeError: max() received an invalid combination of arguments - got (Linear, int), but expected one of:
 * (Tensor input)
 * (Tensor input, name dim, bool keepdim, tuple of Tensors out)
 * (Tensor input, Tensor other, Tensor out)
 * (Tensor input, int dim, bool keepdim, tuple of Tensors out)

Output:
As the error message says the problem is in this line:
_, preds = torch.max(outputs, 1)

There are two problems here:

As @Idodo said, you're giving 2 arguments and neither of them is a tensor. According to the message they are a Linear and a int, respectively.

If you remove the int you still have an error, because you're trying to compute a max value of a nn.Linear, which is not possible. Assessing your code I got the second error. In your model's forward method you have:

x = self.fc3

That's the problem. You must do:
x = self.fc3(x)


Instruction: pytorch: sum of cross entropy over all classes

Input:
I want to compute sum of cross entropy over all classes for each prediction, where the input is batch (size n), and the output is batch (size n).

The simplest way is for loop (for 1000 classes):

def sum_of_CE_lost(input):
    L = 0
    for c in range(1000):
        L = L + torch.nn.CrossEntropyLoss(input, c)
    return L

However, it is very slow. What is a better way? How can we parallelized it for GPU (CUDA)?

Output:
I found the answer:

torch.nn.functional.log_softmax (input).sum() / input.shape[0]

We divide by input.shape[0] because cross_entropy() takes, by default the mean across the batch dimension.


Instruction: How to cache Pytorch models for use when not connected to the internet?

Input:
I'm using vgg19 in a classification problem. I have access to the campus research computer to train on, but the nodes where the computation is done don't have access to the internet. So running a line of code like self.net = models.vgg19(pretrained=True) fails with the error urllib.error.URLError: <urlopen error [Errno 101] Network is unreachable>

Is there a way I could cache the model on the head node (where I have internet access), and load the model from the cache instead of the internet on the compute node?

Output:
If you just save the weights of pretrained networks somewhere, you can load them just like you can load any other network weights.

Saving:

import torchvision

#  I am assuming we have internet access here
model = torchvision.models.vgg16(pretrained=True)
torch.save(model.state_dict(), "Somewhere")

Loading:

import torchvision

def create_vgg16(dict_path=None):
    model = torchvision.models.vgg16(pretrained=False)
    if (dict_path != None):
        model.load_state_dict(torch.load(dict_path))
    return model

model = create_vgg16("Somewhere")


Instruction: Where is '_DataLoaderIter' in pytorch 1.3.1?

Input:
When I use pytorch 1.3.1 with python3.7.4, like this

import torch

from torch.utils.data.dataloader import _DataLoaderIter

Here is an errer : cannot import name '_DataLoaderIter' from 'torch.utils.data.dataloader'

How should I solve that?

Should I uninstall 1.3.1?

I found _DataLoaderIter is in dataloader.pyi :

class _DataLoaderIter:
    def __init__(self, loader: DataLoader) -> None:...
    def __len__(self) -> int: ...
    def __iter__(self) -> _DataLoaderIter: ...
    def __next__(self) -> Any: ...

But I can't find it in dataloader.py.

Output: _DataLoaderIter does not exist any more. This code is the latest one that contains _DataLoaderIter. You can use _SingleProcessDataLoaderIter or _MultiProcessingDataLoaderIter. I don't think the .pyi file you mentioned is in version 1.3.1.


Instruction: 2-step 3D rotation of x,y,z tensors

Input:
In Python, using PyTorch, I'm trying to rotate 3 cubes x, y, and z, first about the x-axis and then about the z-axis. However, the rotation about the z-axis seems to be behaving in a way I would not expect.
I am creating 3 cubes to rotate below and then rotating them 90 degrees about the x-axis and then 90 degrees about the z-axis.
import torch
from numpy import pi
import matplotlib.pyplot as plt

def rotation(x,y,z,theta,phi):
    ### A function for rotating about the x and then z axes ###
    xx = (x*torch.cos(theta)) - (((y*torch.cos(phi)) - (z*torch.sin(phi)))*torch.sin(theta))
    yy = (x*torch.sin(theta)) + (((y*torch.cos(phi)) - (z*torch.sin(phi)))*torch.cos(theta))
    zz = (y*torch.sin(phi)) + (z*torch.cos(phi))
    return xx,yy,zz

### Creating the 3 cubes: x, y, z ###
l = torch.arange(-2,3,1)
x,y,z=torch.meshgrid(l,l,l)

###  Scaling the cubes so they can be differentiated from one another ###
x = x.clone().T
y = y.clone().T*2
z = z.clone().T*3

### Defining the amount of rotation about the x and z axes
phi = torch.tensor([pi/2]).to(torch.float) # about the x axis
theta = torch.tensor([pi/2]).to(torch.float) # about the z axis

### Performing the rotation
x_r,y_r,z_r = rotation(x, y, z, theta, phi)

By visualising the first slice of each cube I can see that the rotation has not been successful as at first glance it looks like the cubes have actually been rotated about the x-axis followed by the y-axis instead.

Is there a specific way that Python handles rotations like this that I'm missing, such as the axes changing along with rotations, meaning the initial rotation matrix operation no longer applies?

Extra information
If one were to replace theta as 0 instead of pi/2, it can be seen that the first rotation behaves as expected by looking at the first slice of each rotated cube:
Code for visualising:
plt.figure()
plt.subplot(231)
x_before = plt.imshow(x[0,:,:])
plt.xlabel('x-before'); plt.colorbar(x_before,fraction=0.046, pad=0.04)
plt.subplot(232)
y_before = plt.imshow(y[0,:,:])
plt.xlabel('y-before'); plt.colorbar(y_before,fraction=0.046, pad=0.04)
plt.subplot(233)
z_before = plt.imshow(z[0,:,:])
plt.xlabel('z-before'); plt.colorbar(z_before,fraction=0.046, pad=0.04)
plt.subplot(234)
x_after = plt.imshow(x_r[0,:,:])
plt.xlabel('x-after'); plt.colorbar(x_after,fraction=0.046, pad=0.04)
plt.subplot(235)
y_after = plt.imshow(y_r[0,:,:])
plt.xlabel('y-after'); plt.colorbar(y_after,fraction=0.046, pad=0.04)
plt.subplot(236)
z_after = plt.imshow(z_r[0,:,:])
plt.xlabel('z-after'); plt.colorbar(z_after,fraction=0.046, pad=0.04)
plt.tight_layout()

Output:
This sounds like a global vs local axis problem - you are rotating 90 degrees around the X axis to start with, which moves your cube so that its local Y axis ends up pointing along the global Z axis. Applying the next rotation of 90 degrees around global Z looks like a rotation around the local Y axis.

To solve this you either need to apply different rotations to achieve the orientation you want (in this case rotate -90 degrees around global Y, since the local Z axis is now facing down the negative global Y axis), or write a different rotation function that can rotate around any vector and track the local axes of the cube (by passing them through the same rotations as the cube itself).

You may also be able to work in local coordinates, by applying the rotations in reverse order, ie a global rotation around Y by 90, followed by a global rotation around X by 90 will be equivalent to local rotations in X then Y.


Instruction: How can I output some data during a model.fit() run in tensorflow?

Input:
I would like to print the value and/or the shape of a tensor during a model.fit() run and not before.
In PyTorch I can just put a print(input.shape) statement into the model.forward() function.
Is there something similar in TensorFlow?

Output:
You can pass a callback object to the model.fit() method and then perform actions at different stages during fitting.

https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/Callback

class MyCustomCallback(tf.keras.callbacks.Callback):

  def on_train_batch_begin(self, batch, logs=None):
    print('Training: batch {} begins at {}'.format(batch, datetime.datetime.now().time()))

  def on_train_batch_end(self, batch, logs=None):
    print('Training: batch {} ends at {}'.format(batch, datetime.datetime.now().time()))

  def on_test_batch_begin(self, batch, logs=None):
    print('Evaluating: batch {} begins at {}'.format(batch, datetime.datetime.now().time()))

  def on_test_batch_end(self, batch, logs=None):
    print('Evaluating: batch {} ends at {}'.format(batch, datetime.datetime.now().time()))

model = get_model()
model.fit(x_train, y_train, callbacks=[MyCustomCallback()])

https://www.tensorflow.org/guide/keras/custom_callback


Instruction: PyTorch MNIST example not converge

Input:
I'm writing a toy example performing the MNIST classification. Here is the full code of my example:
import matplotlib
matplotlib.use("Agg")
import torch
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F
from torch.utils.data import DataLoader

import torchvision.transforms as transforms
import torchvision.datasets as datasets

import matplotlib.pyplot as plt
import os
from os import system, listdir
from os.path import join, isfile, isdir, dirname

def img_transform(image):
    transform=transforms.Compose([
        # transforms.ToTensor(),
        transforms.Normalize((0.1307,), (0.3081,))])
    return transform(image)

def normalize_output(img):
    img = img - img.min()
    img = img / img.max()
    return img

def save_checkpoint(state, filename='checkpoint.pth.tar'):
    torch.save(state, filename)

class Net(nn.Module):
    """docstring for Net"""
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 32, 3, 1)
        self.conv2 = nn.Conv2d(32, 64, 3, 1)
        self.fc1 = nn.Linear(9216, 128)
        self.fc2 = nn.Linear(128, 10)

    def forward(self, x):
        x = self.conv1(x)
        x = F.relu(x)
        x = self.conv2(x)
        x = F.max_pool2d(x, 2)
        x = torch.flatten(x, 1)
        x = self.fc1(x)
        x = F.relu(x)
        x = self.fc2(x)
        output = F.log_softmax(x, dim=1)
        return output

os.environ['CUDA_VISIBLE_DEVICES'] = '0'
data_images, data_labels = torch.load("./PATH/MNIST/processed/training.pt")
model = Net()
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=1e-2)
epochs = 5
batch_size = 30
num_batch = int(data_images.shape[0] / batch_size)
for epoch in range(epochs):
    for batch_idx in range(num_batch):
        data = data_images[ batch_idx*batch_size : (batch_idx+1)*batch_size ].float()
        label = data_labels[ batch_idx*batch_size : (batch_idx+1)*batch_size ]
        data = img_transform(data)
        data = data.unsqueeze_(1)
        pred_score = model(data)
        loss = criterion(pred_score, label)
        loss.backward()
        optimizer.step()
        if batch_idx % 200 == 0:
            print('epoch', epoch, batch_idx, '/', num_batch, 'loss', loss.item())
            _, pred = pred_score.topk(1)
            pred = pred.t().squeeze()
            correct = pred.eq(label)
            num_correct = correct.sum(0).item()
            print('acc=', num_correct/batch_size)

dict_to_save = {
    'epoch': epochs,
    'state_dict': model.state_dict(),
    'optimizer' : optimizer.state_dict(),
    }
ckpt_file = 'a.pth.tar'
save_checkpoint(dict_to_save, ckpt_file)
print('save to ckpt_file', ckpt_file)
exit()

The code is executable with MNIST dataset saved in the path ./PATH/MNIST/processed/training.pt
However, the training process does not converge, with the training accuracy always lower than 0.2. What's wrong with my implementation? I have tried different learning rates and batch size. It doesn't work.
Is there any other problem in my code?
Here are some of the training logs
epoch 0 0 / 2000 loss 27.2023868560791
acc= 0.1
epoch 0 200 / 2000 loss 2.3346288204193115
acc= 0.13333333333333333
epoch 0 400 / 2000 loss 2.691042900085449
acc= 0.13333333333333333
epoch 0 600 / 2000 loss 2.6452369689941406
acc= 0.06666666666666667
epoch 0 800 / 2000 loss 2.7910964488983154
acc= 0.13333333333333333
epoch 0 1000 / 2000 loss 2.966330051422119
acc= 0.1
epoch 0 1200 / 2000 loss 3.111387014389038
acc= 0.06666666666666667
epoch 0 1400 / 2000 loss 3.1988155841827393
acc= 0.03333333333333333

Output:
I see at least four issues that impact on the results you're getting:

You need to zero the gradient, ex:

optimizer.zero_grad()
loss.backward()
optimizer.step()

You're feeding nn.CrossEntropyLoss() with F.softmax. It expects logits. Remove this:

output = F.log_softmax(x, dim=1)

You're computing the loss and acc only for the current batch when you print it. So, it's not the correct result. To solve it you need to store all losses/accs and compute the average before print, for ex:

# During the loop
loss_value += loss.item()

# When printing:
print(loss_value/number_of_batch_losses_stored)

It's not a huge problem, but I'd say this learning rate should be lesser, ex: 1e-3.

As a tip to improve your pipeline, it's better to use a DataLoader to load your data. Have a look at torch.utils.data to learn how to do that. It's not efficient loading the batches the way you're doing because you're not using generators. Also, MNIST is already available on torchvision.datasets.MNIST. It'll save you some time if you load data from there.


Instruction: pytorch difference in 1.1.0 and 1.3.0 in Parameter

Input:
I am trying to learn PyTorch and have tried running some code that I got from the Kaggle website.

# Get all hidden layers' weights
for i in range(len(hidden_units)):
  fc_layers.extend([
    TrainNet.model.hidden_layers[i].weight.T.tolist(), # weights
    TrainNet.model.hidden_layers[i].bias.tolist() # bias
  ])

This gives the following error:

AttributeError                            Traceback (most recent call last)
<ipython-input-12-65f871b6f0b7> in <module>
  4 for i in range(len(hidden_units)):
  5     fc_layers.extend([
----> 6         TrainNet.model.hidden_layers[i].weight.T.tolist(), # weights
  7         TrainNet.model.hidden_layers[i].bias.tolist() # bias
  8     ])

AttributeError: 'Parameter' object has no attribute 'T'

If I print out the type of 'TrainNet.model.hidden_layers[i].weight' it is indeed type Parameter.
This block of code works without error on the Kaggle website run in there notebook (Google colab I believe) where the version of Torch is 1.3.0.

On my home machine where the error occurs my Anaconda distribution which I have just updated is running Torch 1.1.0.

Is this the source of the error and how do I sort it?

Thanks

Output:
class Parameter is a subclass of Tensor hence has all the attributes of a torch.Tensor class

Tensor.T was introduced in 1.2.0 and hence is not available in your 1.1.0

You can either update the pytorch version or use the permute method in it's place as below example

>>> t = torch.Tensor(np.random.randint(0,100,size=(2,3,4))) # -> random Tensor with t.shape = (2,3,4)
>>> t.shape
torch.Size([2, 3, 4])
>>> list(range(len(t.shape)))[::-1]
[2,1,0] # This is the sequence of dimensions t.T will return in 1.2.0 onwards
>>> t = t.permute(list(range(len(t.shape)))[::-1])
>>> t.shape
torch.Size([4, 3, 2])

It is equivalent to doing transpose of a matrix, i.e., reversing the sequence of dimensions but in N dimensional tensors


Instruction: I want to use the GPU instead of CPU while performing computations using PyTorch

Input:
I'm trying to switch the stress from CPU to GPU as my trusty RTX2070 can do it better than the CPU but I keep running into this problem and I'm quite new to AI so if you are kind enough to share some insights with me regarding any potential solution, it would be highly appreciated, thank you.

**I'm using PyTorch

Here's the code that I'm using :

# to measure run-time

# for csv dataset
import os
# to shuffle data
import random
# to get the alphabet
import string

# import statements for iterating over csv file
import cv2
# for plotting
import matplotlib.pyplot as plt
import numpy as np
# pytorch stuff
import torch
import torch.nn as nn
from PIL import Image

# generate the targets
# the targets are one hot encoding vectors
# print(torch.cuda.is_available())
nvcc_args = [
    '-gencode', 'arch=compute_30,code=sm_30',
    '-gencode', 'arch=compute_35,code=sm_35',
    '-gencode', 'arch=compute_37,code=sm_37',
    '-gencode', 'arch=compute_50,code=sm_50',
    '-gencode', 'arch=compute_52,code=sm_52',
    '-gencode', 'arch=compute_60,code=sm_60',
    '-gencode', 'arch=compute_61,code=sm_61',
    '-gencode', 'arch=compute_70,code=sm_70',
    '-gencode', 'arch=compute_75,code=sm_75'
]

alphabet = list(string.ascii_lowercase)
target = {}

# Initalize a target dict that has letters as its keys and empty one-hot encoding vectors of size 37 as its values
for letter in alphabet:
    target[letter] = [0] * 37

# Do the one-hot encoding for each letter now
curr_pos = 0
for curr_letter in target.keys():
    target[curr_letter][curr_pos] = 1
    curr_pos += 1

# extra symbols
symbols = ["space", "number", "period", "comma", "colon", "apostrophe", "hyphen", "semicolon", "question",
           "exclamation", "capitalize"]

# create vectors
for curr_symbol in symbols:
    target[curr_symbol] = [0] * 37

# create one-hot encoding vectors
for curr_symbol in symbols:
    target[curr_symbol][curr_pos] = 1
    curr_pos += 1

# collect all data from the csv file
data = []

for tgt in os.listdir("dataset"):
    if not tgt == ".DS_Store":
        for folder in os.listdir("dataset/" + tgt + "/Uploaded"):
            if not folder == ".DS_Store":
                for filename in os.listdir("dataset/" + tgt + "/Uploaded/" + folder):
                    if not filename == ".DS_Store":
                        # store the image and label
                        picture = []
                        curr_target = target[tgt]
                        image = Image.open("dataset/" + tgt + "/Uploaded/" + folder + "/" + filename)
                        image = image.convert('RGB')
                        # f.show()
                        image = np.array(image)
                        # resize image to 28x28x3
                        image = cv2.resize(image, (28, 28))
                        # normalize to 0-1
                        image = image.astype(np.float32) / 255.0
                        image = torch.from_numpy(image)
                        picture.append(image)
                        # convert the target to a long tensor
                        curr_target = torch.Tensor([curr_target])
                        picture.append(curr_target)
                        # append the current image & target
                        data.append(picture)

# create a dictionary of all the characters
characters = alphabet + symbols

index2char = {}
number = 0
for char in characters:
    index2char[number] = char
    number += 1

# find the number of each character in a dataset
def num_chars(dataset, index2char):
    chars = {}
    for _, label in dataset:
        char = index2char[int(torch.argmax(label))]
        # update
        if char in chars:
            chars[char] += 1
        # initialize
        else:
            chars[char] = 1
    return chars

# Create dataloader objects

# shuffle all the data
random.shuffle(data)

# batch sizes for train, test, and validation
batch_size_train = 30
batch_size_test = 30
batch_size_validation = 30

# splitting data to get training, test, and validation sets
# change once get more data
# 1600 for train
train_dataset = data[:22000]
# test has 212
test_dataset = data[22000:24400]
# validation has 212
validation_dataset = data[24400:]

# create the dataloader objects
train_loader = torch.utils.data.DataLoader(dataset=train_dataset, batch_size=batch_size_train, shuffle=True)
test_loader = torch.utils.data.DataLoader(dataset=test_dataset, batch_size=batch_size_test, shuffle=False)
validation_loader = torch.utils.data.DataLoader(dataset=validation_dataset, batch_size=batch_size_validation,
                                                shuffle=True)

# to check if a dataset is missing a char
test_chars = num_chars(test_dataset, index2char)

num = 0
for char in characters:
    if char in test_chars:
        num += 1
    else:
        break
print(num)

class CNN(nn.Module):
    def __init__(self):
        self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
        super(CNN, self).__init__()
        self.block1 = nn.Sequential(
            # 3x28x28
            nn.Conv2d(in_channels=3,
                      out_channels=16,
                      kernel_size=5,
                      stride=1,
                      padding=2),
            # batch normalization
            # nn.BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True),
            # 16x28x28
            nn.MaxPool2d(kernel_size=2),
            # 16x14x14
            nn.LeakyReLU()
        )
        # 16x14x14
        self.block2 = nn.Sequential(
            nn.Conv2d(in_channels=16,
                      out_channels=32,
                      kernel_size=5,
                      stride=1,
                      padding=2),
            # batch normalization
            # nn.BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True),
            # 32x14x14
            nn.MaxPool2d(kernel_size=2),
            # 32x7x7
            nn.LeakyReLU()
        )
        # linearly
        self.block3 = nn.Sequential(
            nn.Linear(32 * 7 * 7, 100),
            # batch normalization
            # nn.BatchNorm1d(100),
            nn.LeakyReLU(),
            nn.Linear(100, 37)
        )
        # 1x37

    def forward(self, x):
        out = self.block1(x)
        out = self.block2(out)
        # flatten the dataset
        out = out.view(-1, 32 * 7 * 7)
        out = self.block3(out)
        return out

# convolutional neural network model
model = CNN()
model.cuda()

# print summary of the neural network model to check if everything is fine.
print(model)
print("# parameter: ", sum([param.nelement() for param in model.parameters()]))

# setting the learning rate
learning_rate = 1e-4

# Using a variable to store the cross entropy method
criterion = nn.CrossEntropyLoss()

# Using a variable to store the optimizer
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

# list of all train_losses
train_losses = []

# list of all validation losses
validation_losses = []

# for loop that iterates over all the epochs
num_epochs = 20
for epoch in range(num_epochs):

    # variables to store/keep track of the loss and number of iterations
    train_loss = 0
    num_iter_train = 0

    # train the model
    model.train()

    # Iterate over train_loader
    for i, (images, labels) in enumerate(train_loader):
        # need to permute so that the images are of size 3x28x28
        # essential to be able to feed images into the model
        images = images.permute(0, 3, 1, 2)

        # Zero the gradient buffer
        # resets the gradient after each epoch so that the gradients don't add up
        optimizer.zero_grad()

        # Forward, get output
        outputs = model(images)

        # convert the labels from one hot encoding vectors into integer values
        labels = labels.view(-1, 37)
        y_true = torch.argmax(labels, 1)

        # calculate training loss
        loss = criterion(outputs, y_true)

        # Backward (computes all the gradients)
        loss.backward()

        # Optimize
        # loops through all parameters and updates weights by using the gradients
        # takes steps backwards to optimize (to reach the minimum weight)
        optimizer.step()
        # update the training loss and number of iterations
        train_loss += loss.data
        num_iter_train += 1

    print('Epoch: {}'.format(epoch + 1))
    print('Training Loss: {:.4f}'.format(train_loss / num_iter_train))
    # append training loss over all the epochs
    train_losses.append(train_loss / num_iter_train)

    # evaluate the model
    model.eval()

    # variables to store/keep track of the loss and number of iterations
    validation_loss = 0
    num_iter_validation = 0

    # Iterate over validation_loader
    for i, (images, labels) in enumerate(validation_loader):
        # need to permute so that the images are of size 3x28x28
        # essential to be able to feed images into the model
        images = images.permute(0, 3, 1, 2)

        # Forward, get output
        outputs = model(images)

        # convert the labels from one hot encoding vectors to integer values
        labels = labels.view(-1, 37)
        y_true = torch.argmax(labels, 1)

        # calculate the validation loss
        loss = criterion(outputs, y_true)

        # update the training loss and number of iterations
        validation_loss += loss.data
        num_iter_validation += 1

    print('Validation Loss: {:.4f}'.format(validation_loss / num_iter_validation))
    # append all validation_losses over all the epochs
    validation_losses.append(validation_loss / num_iter_validation)

    num_iter_test = 0
    correct = 0

    # Iterate over test_loader
    for images, labels in test_loader:
        # need to permute so that the images are of size 3x28x28
        # essential to be able to feed images into the model
        images = images.permute(0, 3, 1, 2)

        # Forward
        outputs = model(images)

        # convert the labels from one hot encoding vectors into integer values
        labels = labels.view(-1, 37)
        y_true = torch.argmax(labels, 1)

        # find the index of the prediction
        y_pred = torch.argmax(outputs, 1).type('torch.FloatTensor')

        # convert to FloatTensor
        y_true = y_true.type('torch.FloatTensor')

        # find the mean difference of the comparisons
        correct += torch.sum(torch.eq(y_true, y_pred).type('torch.FloatTensor'))

    print('Accuracy on the test set: {:.4f}%'.format(correct / len(test_dataset) * 100))
    print()

# learning curve function
def plot_learning_curve(train_losses, validation_losses):
    # plot the training and validation losses
    plt.ylabel('Loss')
    plt.xlabel('Number of Epochs')
    plt.plot(train_losses, label="training")
    plt.plot(validation_losses, label="validation")
    plt.legend(loc=1)

# plot the learning curve
plt.title("Learning Curve (Loss vs Number of Epochs)")
plot_learning_curve(train_losses, validation_losses)

torch.save(model.state_dict(), "model1.pth")

Output:
I'm also using a trusty RTX 2070 and this is how I do GPU acceleration (for 1 GPU):

cuda_ = "cuda:0"
device = torch.device(cuda_ if torch.cuda.is_available() else "cpu")
model = CNN()
model.to(device)

This is the most up-to-date and recommended way to do GPU acceleration, as it gives more flexibility (don't need to amend code even when GPU isn't available). You would do the same to pass your images into the GPU vram, via images = images.to(device).


Instruction: Multiple PyTorch networks running in parallel on different CPUs

Input:
I’m trying to have different PyTorch neural networks run in parallel on different CPUs but am finding that it isn’t leading to any sort of speed up compared to running them sequentially.

Below is my code that replicates the issue exactly. If you run this code it shows that with 2 processes it takes roughly twice as long as running it with 1 process but really it should take the same amount of time.

import time
import torch.multiprocessing as mp
import gym
import numpy as np
import copy
import torch.nn as nn
import torch

class NN(nn.Module):
    def __init__(self, output_dim):
        nn.Module.__init__(self)
        self.fc1 = nn.Linear(4, 50)
        self.fc2 = nn.Linear(50, 500)
        self.fc3 = nn.Linear(500, 5000)
        self.fc4 = nn.Linear(5000, output_dim)
        self.relu = nn.ReLU()

    def forward(self, x):
        x = self.relu(self.fc1(x))
        x = self.relu(self.fc2(x))
        x = self.relu(self.fc3(x))
        x = self.fc4(x)
        return x

def Worker(ix):
  print("Starting training for worker ", ix)
  env = gym.make('CartPole-v0')
  model = NN(2)
  for _ in range(2000):
    model(torch.Tensor(env.reset()))
  print("Finishing training for worker ", ix)

def overall_process(num_workers):
  workers = []
  for ix in range(num_workers):
    worker = mp.Process(target=Worker, args=(ix, ))
    workers.append(worker)
  [w.start() for w in workers]
  for worker in workers:
    worker.join()

  print("Finished Training")
  print(" ")

start = time.time()
overall_process(1)
print("Time taken: ", time.time() - start)
print(" ")

start = time.time()
overall_process(2)
print("Time taken: ", time.time() - start)

Does anyone know why this might be happening and how to fix it?

I thought that it is maybe because PyTorch networks automatically implement CPU parallelism in the background and so I tried adding the below 2 lines but it doesn’t always resolve the issue:

torch.set_num_threads(1)
torch.set_num_interop_threads(1)

Output: The answer is to set torch.set_num_threads(1) at the beginning of each worker process (rather than in the main process) as explained here: https://discuss.pytorch.org/t/multiple-networks-running-in-parallel-on-different-cpus/70482


Instruction: Why does loss skyrocket during convolutional neural net training?

Input: I am training a simple CNN in Pytorch for segmentation on a very small dataset (just a few images as this is just for proof of concept purposes). For some reason, loss skyrockets to as high as 6 and IoU drops to 0 (intersection over union accuracy metric) randomly during training before going back up. I was wondering why this could be happening?

Output:
Instability. It is actually common. Take a look at the published papers and you will see the same thing too. During gradient descent, there may be "rough patches" in the gradient landscape and gives a locally bad solution, hence high loss.

Having said that, some of these spikes can actually signify that you have made poor hyperparameter and network architecture choices. From my experience, one of the possible causes for the spikes is the usage of weight decay. Weight decay provides regularisation, but in my own work, i found it to cause a lot of instability. So nowadays, i don't use it anymore.

The spikes in your graph doesn't look too bad, i wouldn't worry about it.


Instruction: Re-using a classification CNN model for autoencoding - pytorch

Input:
I am very new to pytorch so I need a bit of handholding. I am trying to re-use an old CNN classification model -- reusing the already trained convolutional layers as the encoder in an autoencoder and then training the decoder layers. The below code is what I have.

class Autoencoder(nn.Module):
  def __init__(self, model, specs):

    super(Autoencoder, self).__init__()

    self.encoder = nn.Sequential(
        *list(model.conv_layer.children())
        )

    self.decoder = nn.Sequential(
        nn.ConvTranspose2d(in_channels=C7, out_channels=C6, kernel_size=pooling, padding=0),
        nn.ReLU(inplace=True),
        nn.ConvTranspose2d(in_channels=C6, out_channels=C5, kernel_size=pooling, padding=0),
        nn.ReLU(inplace=True),
        nn.ConvTranspose2d(in_channels=C5, out_channels=C4, kernel_size=pooling, padding=0),
        nn.ReLU(inplace=True),
        nn.ConvTranspose2d(in_channels=C4, out_channels=C3, kernel_size=pooling, padding=0),
        nn.ReLU(inplace=True),
        nn.ConvTranspose2d(in_channels=C3, out_channels=C2, kernel_size=pooling, padding=0),
        nn.ReLU(inplace=True),
        nn.ConvTranspose2d(in_channels=C2, out_channels=C1, kernel_size=pooling, padding=0),
        nn.ReLU(inplace=True),
        nn.ConvTranspose2d(in_channels=C1, out_channels=C0, kernel_size=pooling, padding=0),
        nn.ReLU(inplace=True),
        nn.ConvTranspose2d(in_channels=C0, out_channels=3, kernel_size=pooling, padding=0),
        nn.ReLU(inplace=True),
        )
    for param in self.encoder.parameters():
      param.requires_grad = False

    for p in self.decoder.parameters():
      if p.dim() > 1:
        nn.init.kaiming_normal_(p)
        pass

    def forward(self, x):
      x = self.encoder(x)
      x = self.decoder(x)
      return x

However, I am getting a "NotImplementedError". What am I doing wrong? When I initiate an instance of that class, I would be passing the pretrained CNN classification model and self.encoder should take care of taking the layers I am interested from the model (those in conv_layer). When I:

model = pretrainedCNNmodel
autoencoder = Autoencoder(model, specs)
print(autoencoder)

the print looks okay, it has all layers and everything I am hoping for, but when I try to train on it I get the "NotImplementedError:".

Edit

Here is the entire error:

---------------------------------------------------------------------------
NotImplementedError                       Traceback (most recent call last)
<ipython-input-20-9adc467b2472> in <module>()
      2 optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate, weight_decay=L2_lambda)
      3
----> 4 train(x, train_loader, test_loader, optimizer, criterion)

2 frames
<ipython-input-5-b25edb14cf5f> in train(model, train_loader, test_loader, optimizer, criterion)
     15       data, target = data.cuda(), target.cuda()
     16       optimizer.zero_grad()
---> 17       output = model(data)
     18       loss = criterion(output, target)
     19       loss.backward()

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--> 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in forward(self, *input)
     94             registered hooks while the latter silently ignores them.
     95         """
---> 96         raise NotImplementedError
     97
     98     def register_buffer(self, name, tensor):

NotImplementedError:

Output:
Since you have a bounty on this question, it cannot be closed. However, the exact same question was already asked and answered in this thread.

Basically, you have an indentation problem in your code: Your forward method is indented such that it is inside your __init__ method, instead of being part of the Autoencoder class.

Please see my other answer for more details.


Instruction: Assign the new value to a tensor at specific indices

Input:
This is a simple example.

Assume that I have an input tensor M. Now I have a tensor of indices of M with size 2 x 3 such as [[0, 1], [2,2], [0,1]] and a new array of values which is corresponding with the index tensor is [1, 2, 3]. I want to assign these values to the input M satisfying that the value is assigned to the element of M at index [0,1] will be the min value (1 in this example).

It means M[0,1] = 1 and M[2,2] = 2.

Can I do that by using some available functions in Pytorch without a loop?

Output:
It can be done without loops, but I am generally not sure whether it is such a great idea, due to significantly increased runtime.

The basic idea is relatively simple: Since tensor assignments always assign the last element, it is sufficient to sort your tuples in M in descending order, according to the respective values stored in the value list (let's call it v).

To do this in pytorch, let us consider the following example:

import torch as t
X = t.randn([3, 3])  # Random matrix of size 3x3
v = t.tensor([1, 2, 3])
M = t.tensor([[0, 2, 0],
              [1, 2, 1]])  # accessing the elements described above

# Showcase pytorch's result with "naive" tensor assignment:
X[tuple(M)] = v  # This would assign the value 3 to position (0, 1)

# To correct behavior, sort v in decreasing order.
v_desc = v.sort(decreasing=True)
# v now contains both the values and the indices of original position
print(v_desc)
# torch.return_types.sort(
#       values=tensor([3, 2, 1]),
#       indices=tensor([2, 1, 0]))

# Access M in the correct order:
M_desc = M[:, v_desc.indices]

# Finally assign correct order:
X[tuple(M_desc)] = v_desc

Again, this is relatively complicated, because it involves sorting the values, and "re-shuffling" of the tensors. You can surely save at least some memory if you perform operations in-place, which is something I disregarded for the sake of legibility.

As an answer whether this can also be achieved without sorting, I am fairly certain that the answer will be "no"; tensor assignments could only be done on fairly simple conditionals, but not something more complicated like your inter-dependent conditionals would require.


Instruction: How to use a customized dataset for training with PyTorch/few-shot-vid2vid

Input:
I’d like to use my own dataset created from the FaceForensics footage with few-show-vid2vid. So I generated image sequences with ffmpeg and keypoints with dlib. When I try to start the training script, I get the following error. What exactly is the problem? The provided small dataset was working for me.

CustomDatasetDataLoader
485 sequences
dataset [FaceDataset] was created
Resuming from epoch 1 at iteration 0
create web directory ./checkpoints/face/web...
---------- Networks initialized -------------
---------- Optimizers initialized -------------
./checkpoints/face/latest_net_G.pth not exists yet!
./checkpoints/face/latest_net_D.pth not exists yet!
model [Vid2VidModel] was created
Traceback (most recent call last):
  File "train.py", line 73, in <module>
    train()
  File "train.py", line 40, in train
    for idx, data in enumerate(dataset, start=trainer.epoch_iter):
  File "/home/keno/.local/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 819, in __next__
    return self._process_data(data)
  File "/home/keno/.local/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 846, in _process_data
    data.reraise()
  File "/home/keno/.local/lib/python3.7/site-packages/torch/_utils.py", line 369, in reraise
    raise self.exc_type(msg)
IndexError: Caught IndexError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/home/keno/.local/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/keno/.local/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/keno/.local/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/keno/repos/few-shot-vid2vid/data/fewshot_face_dataset.py", line 103, in __getitem__
    Li = self.get_face_image(keypoints, transform_L, ref_img.size)
  File "/home/keno/repos/few-shot-vid2vid/data/fewshot_face_dataset.py", line 168, in get_face_image
    x = keypoints[sub_edge, 0]
IndexError: index 82 is out of bounds for axis 0 with size 82

EDIT: How I set up my dataset. I created image sequences from the video footage with ffmpeg -i _video_ -o %05d.jpg, following the directory structure of the provided sample dataset. Then I generated the keypoints by using landmark detection with dlib, based on the code example provided on the dlib website. I expanded the sample code to 68 points and saved them to .txt files:

import re
import sys
import os
import dlib
import glob

# if len(sys.argv) != 4:
#     print(
#         "Give the path to the trained shape predictor model as the first "
#         "argument and then the directory containing the facial images.\n"
#         "For example, if you are in the python_examples folder then "
#         "execute this program by running:\n"
#         "    ./face_landmark_detection.py shape_predictor_68_face_landmarks.dat ../examples/faces\n"
#         "You can download a trained facial shape predictor from:\n"
#         "    http://dlib.net/files/shape_predictor_68_face_landmarks.dat.bz2")
#     exit()

predictor_path = sys.argv[1]
faces_folder_path = sys.argv[2]
text_file_path = sys.argv[3]

detector = dlib.get_frontal_face_detector()
predictor = dlib.shape_predictor(predictor_path)
win = dlib.image_window()

for f in glob.glob(os.path.join(faces_folder_path, "*.jpg")):
    file_number = os.path.split(f)
    print(file_number[1])
    file_number = os.path.splitext(file_number[1])
    file_number = file_number[0]
    export_path = os.path.join(text_file_path, '%s.txt' % file_number)
    text = open(export_path,"w+")

    print("Processing file: {}".format(f))
    img = dlib.load_rgb_image(f)

    win.clear_overlay()
    win.set_image(img)

    # Ask the detector to find the bounding boxes of each face. The 1 in the
    # second argument indicates that we should upsample the image 1 time. This
    # will make everything bigger and allow us to detect more faces.
    dets = detector(img, 1)
    print("Number of faces detected: {}".format(len(dets)))
    for k, d in enumerate(dets):
        print("Detection {}: Left: {} Top: {} Right: {} Bottom: {}".format(
            k, d.left(), d.top(), d.right(), d.bottom()))
        # Get the landmarks/parts for the face in box d.
        shape = predictor(img, d)
        for i in range(67):
            result = str(shape.part(i))
            result = result.strip("()")
            print(result)
            text.write(result + '\n')
        # Draw the face landmarks on the screen.
        win.add_overlay(shape)

    text.close()
    win.add_overlay(dets)

Output:
  for i in range(67):

This is incorrect, you should be using range(68) for 68 face landmarks.  You can verify this with python -c "for i in range(67): print(i)" which will only count from 0 to 66 (67 total numbers). python -c "for i in range(68): print(i)" will count from 0 to 67 (68 items) and get the whole face landmark set.


Instruction: Expected object of device type cuda but got device type cpu

Input:
I am trying to switch the training of my network from cpu to gpu but keep getting the following error.

I am getting the following error

Expected object of device type cuda but got device type cpu for argument #1 'self' in call to _thnn_conv2d_forward
Error occurs, No graph saved
Traceback (most recent call last):

  File "<ipython-input-6-2720a5ea768d>", line 12, in <module>
    tb.add_graph(network, images)

  File "E:\Anaconda\lib\site-packages\torch\utils\tensorboard\writer.py", line 707, in add_graph
    self._get_file_writer().add_graph(graph(model, input_to_model, verbose))

  File "E:\Anaconda\lib\site-packages\torch\utils\tensorboard\_pytorch_graph.py", line 291, in graph
    raise e

  File "E:\Anaconda\lib\site-packages\torch\utils\tensorboard\_pytorch_graph.py", line 285, in graph
    trace = torch.jit.trace(model, args)

  File "E:\Anaconda\lib\site-packages\torch\jit\__init__.py", line 882, in trace
    check_tolerance, _force_outplace, _module_class)

  File "E:\Anaconda\lib\site-packages\torch\jit\__init__.py", line 1034, in trace_module
    module._c._create_method_from_trace(method_name, func, example_inputs, var_lookup_fn, _force_outplace)

  File "E:\Anaconda\lib\site-packages\torch\nn\modules\module.py", line 530, in __call__
    result = self._slow_forward(*input, **kwargs)

  File "E:\Anaconda\lib\site-packages\torch\nn\modules\module.py", line 516, in _slow_forward
    result = self.forward(*input, **kwargs)

  File "<ipython-input-5-cd44a4e4fb73>", line 52, in forward
    t = F.relu(self.conv1(t))

  File "E:\Anaconda\lib\site-packages\torch\nn\modules\module.py", line 530, in __call__
    result = self._slow_forward(*input, **kwargs)

  File "E:\Anaconda\lib\site-packages\torch\nn\modules\module.py", line 516, in _slow_forward
    result = self.forward(*input, **kwargs)

  File "E:\Anaconda\lib\site-packages\torch\nn\modules\conv.py", line 345, in forward
    return self.conv2d_forward(input, self.weight)

  File "E:\Anaconda\lib\site-packages\torch\nn\modules\conv.py", line 342, in conv2d_forward
    self.padding, self.dilation, self.groups)

RuntimeError: Expected object of device type cuda but got device type cpu for argument #1 'self' in call to _thnn_conv2d_forward```**

I think it says that argument is in type cpu but I changed it in the training part.

I have the following code

Conv-neural network

class Network(nn.Module):
    def __init__(self):
    super(Network, self).__init__()
    self.conv1 = nn.Conv2d( in_channels= 1, out_channels= 6, kernel_size=5 )
    self.conv2 = nn.Conv2d( in_channels= 6, out_channels= 12, kernel_size=5 )

    self.fc1 = nn.Linear( in_features = 12*4*4, out_features = 120 )
    self.fc2 = nn.Linear( in_features = 120, out_features = 60 )
    self.out = nn.Linear( in_features = 60, out_features = 10 )

def forward(self, t):
    t = F.relu(self.conv1(t))
    t = F.max_pool2d(t, kernel_size=2, stride=2)

    t = F.relu(self.conv2(t))
    t = F.max_pool2d(t, kernel_size=2, stride=2)

    t = F.relu(self.fc1(t.reshape(-1, 12*4*4)))
    t = F.relu(self.fc2(t))
    t = self.out(t)

    return t

The training part

parameters = dict(
    lr = [.01, .001]
    , batch_size = [10, 100, 1000]
    , shuffle = [True, False]
    )

param_values = [v for v in parameters.values()]
param_values

for lr, batch_size, shuffle in product(*param_values):
network = Network()
network.to(device)
train_loader = torch.utils.data.DataLoader(train_set, batch_size=batch_size, shuffle = shuffle)
optimizer = optim.Adam(network.parameters(), lr=lr)
images, labels = next(iter(train_loader))
grid = torchvision.utils.make_grid(images)

comment = f' batch_size={batch_size} lr={lr} shuffle={shuffle}'
tb = SummaryWriter(comment = comment)
tb.add_image('images', grid)
tb.add_graph(network, images)

for epoch in range(10):

    total_loss = 0
    total_correct = 0

    for batch in train_loader: # Get batch
        images, labels = batch
        images = images.to(device)  # Changing data to gpu
        preds = network(images)
        loss = F.cross_entropy(preds, labels)

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        total_loss += loss.item() * batch_size
        total_correct += get_num_correct(preds, labels)

    tb.add_scalar('Loss:', total_loss, epoch)
    tb.add_scalar('Number Correct:', total_correct, epoch)
    tb.add_scalar('Accuracy:', total_correct/len(train_set), epoch)

    #tb.add_histogram('conv1.bias', network.conv1.bias, epoch)
    #tb.add_histogram('conv1.weight', network.conv1.weight, epoch)
    #tb.add_histogram('conv1.weight.grap', network.conv1.weight.grad, epoch)

    for name, weight in network.named_parameters():
        tb.add_histogram(name, weight, epoch)
        tb.add_histogram(f'{name}.grad', weight.grad, epoch)

    print("epoch:", epoch, "total_correct:", total_correct, "loss:",total_loss)

tb.close()

I am new to deep learning so any help will be highly appreciated. Thanks

Output:
You missed moving your labels to gpu i.e.

labels = labels.to(device)

You also need to move these to gpu:

images, labels = next(iter(train_loader))
images = images.to(device)
labels = labels.to(device)


Instruction: Different results from PyTorch's conv1d and SciPy's convolve

Input:
I'm building a PyTorch model to estimate Impuse Responses.
Currently I am calculating the loss from the real and estimated impulse response.
I would like to convolve both the estimated and real impulse response with a signal and then calculate the loss from those.

The pyroomaccoustics package uses SciPy's fftconvolve to convolve the impulse response with a given signal. I cannot use this since it would break PyTorch's computation graph. PyTorch's conv1d uses cross-correlation. From this answer it seems that by flipping the filter conv1d can be used for convolution.

I am confused as to why the following code gives a different result for conv1d and convolve and what must be changed to get the outputs to be equal.

import torch
from scipy.signal import convolve

a = torch.tensor([.1, .2, .3, .4, .5])
b = torch.tensor([.0, .1, .0])

a1 = a.view(1, 1, -1)
b1 = torch.flip(b, (0,)).view(1, 1, -1)

print(torch.nn.functional.conv1d(a1, b1).view(-1))
# >>> tensor([0.0200, 0.0300, 0.0400])

print(convolve(a, b))
# >>> [0.   0.01 0.02 0.03 0.04 0.05 0.  ]

Output:
Take a look at the mode parameter of scipy.signal.convolve.  Use mode='valid' to match PyTorch's conv1d:

In [20]: from scipy.signal import convolve

In [21]: a = np.array([.1, .2, .3, .4, .5])

In [22]: b = np.array([.0, .1, .0])

In [23]: convolve(a, b, mode='valid')
Out[23]: array([0.02, 0.03, 0.04])

To modify the call of PyTorch's conv1d to give the same output as the default behavior of scipy.signal.convolve (i.e. to match mode='full') for this example, set padding=2 in the call to conv1d.  More generally, for a given convolution kernel b, set the padding to len(b) - 1.


Instruction: Return predictions wav2vec fairseq

Input:
I'm trying to use wav2vec to train my own Automatic Speech Recognition System:

https://github.com/pytorch/fairseq/tree/master/examples/wav2vec

import torch
from fairseq.models.wav2vec import Wav2VecModel

cp = torch.load('/path/to/wav2vec.pt')
model = Wav2VecModel.build_model(cp['args'], task=None)
model.load_state_dict(cp['model'])
model.eval()

First of all how can I use a loaded model to return predictions from a wav file?

Second, how can I pre-train using annotated data?
I don't see any text mention in the manifest scripts.

Output:
After trying various things I was able to figure this out and trained a wav2vec model from scratch.

Some background: wav2vec uses semi-supervised learning to learn vector representations for preprocessed sound frames. This is similar to what word2vec does to learn word embeddings a text corpus. In the case of wav2vec it samples random parts of the sound file and learns to predict if a given part is in the near future from a current offset position. This is somewhat similar to the masked word task used to train transformers such as BERT. The nice thing about such prediction tasks is that they are self-supervised: the algorithm can be trained on unlabeled data since it uses the temporal structure of the data to produce labels and it uses random sampling to produce contrasting negative examples. It is a binary classification task (is the proposed processed sound frame in the near future of the current offset or not).
In training for this binary classification task, it learns vector representations of sound frames (one 512 dim vector for each 10ms of sound). These vector representations are useful features because they concentrate information relevant to predicting speech. These vectors can then be used instead of spectrogram vectors as inputs for speech to text algorithms such as wav2letter or deepSpeech. This is an important point: wav2vec is not a full automatic speech recognition (ASR) system. It is a useful component because by leveraging self-supervised learning on unlabeled data (audio files containing speech but without text transcriptions), it greatly reduces the need for labeled data (speech transcribed to text). Based on their article it appears that by using wav2vec in an ASR pipeline, the amount of labeled data needed can be reduced by a factor of at least 10 (10 to 100 times less transcribed speech is needed apparently). Since un-transcribed speech files are much easier to get than transcribed speech, this is a huge advantage of using wav2vec as an initial module in an ASR system.

So wav2vec is trained with data which is not annotated (no text is used to train it).

The thing which confused me was the following command for training (here) :

python train.py /manifest/path --save-dir /model/path ...(etc.).........

It turns out that since wav2vec is part of fairseq, the following fairseq command line tool should be used to train it:

fairseq-train

As the arguments to this command are pretty long, this can be done using a bash scipt such as

#!/bin/bash
fairseq-train /home/user/4fairseq --save-dir /home/user/4fairseq --fp16 --max-update 400000 --save-interval 1 --no-epoch-checkpoints \
--arch wav2vec --task audio_pretraining --lr 1e-06 --min-lr 1e-09 --optimizer adam --max-lr 0.005 --lr-scheduler cosine \
--conv-feature-layers "[(512, 10, 5), (512, 8, 4), (512, 4, 2), (512, 4, 2), (512, 4, 2), (512, 1, 1), (512, 1, 1)]" \
--conv-aggregator-layers "[(512, 2, 1), (512, 3, 1), (512, 4, 1), (512, 5, 1), (512, 6, 1), (512, 7, 1), (512, 8, 1), (512, 9, 1), (512, 10, 1), (512, 11, 1), (512, 12, 1), (512, 13, 1)]" \
--skip-connections-agg --residual-scale 0.5 --log-compression --warmup-updates 500 --warmup-init-lr 1e-07 --criterion binary_cross_entropy --num-negatives 10 \
--max-sample-size 150000 --max-tokens 1500000

most of the arguments are those suggested here, only the first two (which are filesystem paths) must be modified for your system.

Since I had audio voice files which were in mp3 format, I converted them to wav files the following bash script:

#!/bin/bash
for file in /home/user/data/soundFiles/*
do
  echo "$file"
  echo "${file%.*}.wav"
  ffmpeg -i "$file" "${file%.*}.wav"
done

They suggest that the audio files be of short duration, longer files should be split into smaller files. The files which I had were already pretty short so I did not do any splitting.

the script wav2vec_manifest.py must be used to create a training data manifest before training. It will create two files (train.tsv and valid.tsv) basically creating lists of which audio files should be used for training and which should be used for validation. The path at which these two files are located is the first argument to the fairseq-train method.

The second argument to the method fairseq-train is the path at which to save the model.
After training there will be these two model files:
checkpoint_best.pt
checkpoint_last.pt
These are updated at the end of each epoch so I was able to terminate the train process early and still have those saved model files


Instruction: Get matrix dimensions from pytorch layers

Input:
Here is an autoencoder I created from Pytorch tutorials :

epochs = 1000
from pylab import plt
plt.style.use('seaborn')
import torch.utils.data as data_utils
import torch
import torchvision
import torch.nn as nn
from torch.autograd import Variable

cuda = torch.cuda.is_available()
FloatTensor = torch.cuda.FloatTensor if cuda else torch.FloatTensor
import numpy as np
import pandas as pd
import datetime as dt

features = torch.tensor(np.array([ [1,2,3],[1,2,3],[100,200,500] ]))

print(features)

batch = 10
data_loader = torch.utils.data.DataLoader(features, batch_size=2, shuffle=False)

encoder = nn.Sequential(nn.Linear(3,batch), nn.Sigmoid())
decoder = nn.Sequential(nn.Linear(batch,3), nn.Sigmoid())
autoencoder = nn.Sequential(encoder, decoder)

optimizer = torch.optim.Adam(params=autoencoder.parameters(), lr=0.001)

encoded_images = []
for i in range(epochs):
    for j, images in enumerate(data_loader):
    #     images = images.view(images.size(0), -1)
        images = Variable(images).type(FloatTensor)
        optimizer.zero_grad()
        reconstructions = autoencoder(images)
        loss = torch.dist(images, reconstructions)
        loss.backward()
        optimizer.step()

#     encoded_images.append(encoder(images))

# print(decoder(torch.tensor(np.array([1,2,3])).type(FloatTensor)))

encoded_images = []
for j, images in enumerate(data_loader):
    images = images.view(images.size(0), -1)
    images = Variable(images).type(FloatTensor)

    encoded_images.append(encoder(images))

I can see the encoded images do have newly created dimension of 10. In order to understand the matrix operations going on under the hood I'm attempting to print the matrix dimensions of encoder and decoder but shape is not available on nn.Sequential

How to print the matrix dimensions of nn.Sequential ?

Output:
A nn.Sequential is not a "layer", but rather a "container". It can store several layers and manage their execution (and some other functionalities).
In your case, each nn.Sequential holds both the linear layer and the non-linear nn.Sigmoid activation. To get the shape of the weights of the first layer in a nn.Sequential you can simply do:

encoder[0].weight.shape


Instruction: Why is CNN convolution output size in PyTorch DQN tutorial computed with `kernel_size -1`?

Input:
Based on my understanding, CNN output size for 1D is

output_size = (input_size - kernel_size + 2*padding)//stride + 1

Refer to PyTorch DQN Tutorial. In the tutorial, it uses 0 padding, which is fine. However, it computes the output size as follows:

        def conv2d_size_out(size, kernel_size = 5, stride = 2):
            return (size - (kernel_size - 1) - 1) // stride  + 1

It the above a mistake or is there something I missed?

Output:
No, it's not a mistake because

size - (kernel_size - 1) - 1 = size - kernel_size + 2 * 0

with 0 as padding
(it's not code, its an equation sorry for the formatting)
I think the tutorial is using the formula for the output size from the official document which is

output_size = ((input_size + 2 * padding - dialation * (kernel_size - 1) - 1) // stride + 1

official doc for conv1d


Instruction: How to index/slice the last dimension of a PyTorch tensor/numpy array of unknown dimensions

Input:
For example, if I have a 2D tensor X, I can do slicing X[:,1:]; if I have a 3D tensor Y, then I can do similar slicing for the last dimension like Y[:,:,1:].

What is the right way to do the slicing when given a tensor Z of unknown dimension?
How about a numpy array?

Thanks!

Output:
PyTorch support NumPy-like indexing so you can use Ellipsis(...)
>>> z[..., -1:]

Example:
>>> x                     # (2,2) tensor
tensor([[0.5385, 0.9280],
        [0.8937, 0.0423]])
>>> x[..., -1:]
tensor([[0.9280],
        [0.0423]])

>>> y                     # (2,2,2) tensor
tensor([[[0.5610, 0.8542],
         [0.2902, 0.2388]],

        [[0.2440, 0.1063],
         [0.7201, 0.1010]]])
>>> y[..., -1:]
tensor([[[0.8542],
         [0.2388]],

        [[0.1063],
         [0.1010]]])

Ellipsis (...) expands to the number of : objects needed for the selection tuple to index all dimensions. In most cases, this means that length of the expanded selection tuple is x.ndim. There may only be a single ellipsis present.


Instruction: pytorch: model.forward() effects the training process even if the result is not part of the loss function

Input:
I have the following code to train a simple model:

train_loader, another_loader = get_loaders()
model = torch_utils.get_model()

ce = nn.CrossEntropyLoss()
optimizer = optim.SGD(model.parameters(), lr=0.001, momentum=0.9)
another_loader_iter = iter(another_loader)

for epoch in range(10):
    for i, (X, y) in enumerate(train_loader):
        model.train()
        X, y = X.to(device), y.to(device)
        pred = model.forward(X)
        loss1 = ce(pred, y)
        for j in range(2):
            X2, y2 = next(another_loader)
            pred2 = model.forward(X2) #**This line effects the training process - why???**

        new_loss = loss1
        optimizer.zero_grad()
        new_loss.backward()
        optimizer.step()

When I comment out pred2 = model.forward(X2) the model trains as it should but I want to use pred2 to add a new expression to the loss function.

Output:
I'd suggest the following changes:

for epoch in range(10):
    for i, (X, y) in enumerate(train_loader):
        model.train()
        X, y = X.to(device), y.to(device)
        pred = model(X)
        loss1 = ce(pred, y)
        for j in range(2):
            model.eval()
            X2, y2 = next(another_loader)
            pred2 = model(X2)

        new_loss = loss1
        optimizer.zero_grad()
        new_loss.backward()
        optimizer.step()

I'd also say you don't need new_loss = loss1, but think you're including more stuff here in your original code, right?

In any case, check if these changes help you.


Instruction: pytorch DataLoader: `Tensors must have same number of dimensions`

Input:
I am trying to fit an LSTM model in Pytorch. My data is too big to be read into memory and so I want to create mini-batches of data using the DataLoader function from Pytorch.

I have two features as input (X1, X2). I have one output feature (y). I am using 365 timesteps of X1 & X2 as features used to predict y.

The dimensions of my training array is:

(n_observations, n_timesteps, n_features) == (9498, 365, 2)

I don't understand why the code below isn't working because I have seen other examples where the X, y pairs have different numbers of dimensions (LSTM for runoff modelling, Pytorch's own docs )

Minimum Reproducible Example

import numpy as np
import torch
from torch.utils.data import DataLoader

train_x = torch.Tensor(np.random.random((9498, 365, 2)))
train_y = torch.Tensor(np.random.random((9498, 1)))
val_x = torch.Tensor(np.random.random((1097, 365, 2)))
val_y = torch.Tensor(np.random.random((1097, 1)))
test_x = torch.Tensor(np.random.random((639, 365, 2)))
test_y = torch.Tensor(np.random.random((639, 1)))

train_dataset = (train_x, train_y)
test_dataset = (test_x, test_y)
val_dataset = (val_x, val_y)

train_dataloader = DataLoader(train_dataset, batch_size=256)

iterator = train_dataloader.__iter__()
iterator.next()

Output:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-47-2a0b28b53c8f> in <module>
     13
     14 iterator = train_dataloader.__iter__()
---> 15 iterator.next()

/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py in __next__(self)
    344     def __next__(self):
    345         index = self._next_index()  # may raise StopIteration
--> 346         data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
    347         if self._pin_memory:
    348             data = _utils.pin_memory.pin_memory(data)

/opt/conda/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py in fetch(self, possibly_batched_index)
     45         else:
     46             data = self.dataset[possibly_batched_index]
---> 47         return self.collate_fn(data)

/opt/conda/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py in default_collate(batch)
     53             storage = elem.storage()._new_shared(numel)
     54             out = elem.new(storage)
---> 55         return torch.stack(batch, 0, out=out)
     56     elif elem_type.__module__ == 'numpy' and elem_type.__name__ != 'str_' \
     57             and elem_type.__name__ != 'string_':

RuntimeError: invalid argument 0: Tensors must have same number of dimensions: got 4 and 3 at /tmp/pip-req-build-4baxydiv/aten/src/TH/generic/THTensor.cpp:680

Output:
The torch.utils.data.DataLoader must get a torch.utils.data.Dataset as parameters. You're giving a tuple of tensors. I suggest you use the torch.utils.data.TensorDataset as follows:

from torch.utils.data import DataLoader, TensorDataset

train_x = torch.rand(9498, 365, 2)
train_y = torch.rand(9498, 1)

train_dataset = TensorDataset(train_x, train_y)
train_dataloader = DataLoader(train_dataset, batch_size=256)

for x, y in train_dataloader:
    print (x.shape)

Check if it solves your problem.


Instruction: PyTorch: Convolving a single channel image using torch.nn.Conv2d

Input:
I am trying to use a convolution layer to convolve a grayscale (single layer) image (stored as a numpy array). Here is the code:

conv1 = torch.nn.Conv2d(in_channels = 1, out_channels = 1, kernel_size = 33)
tensor1 = torch.from_numpy(img_gray)
out_2d_np = conv1(tensor1)

out_2d_np = np.asarray(out_2d_np)

I want my kernel to be 33x33 and the number of output layers should be equal to the number of input layers, which is 1 as the image's RGB channels are summed. Whenout_2d_np = conv1(tensor1) is run it yields the following runtime error:

  RuntimeError: Expected 4-dimensional input for 4-dimensional weight 1 1 33 33, but got 2-dimensional input of size [246, 248] instead

Any idea on how I can solve this? I specifically want to use the torch.nn.Conv2d() class/function.

Thanks in advance for any help!

Output:
pytorch's Conv2d expects its 2D inputs to actually have 4 dimensions: mini-batch dim, channel dim, and the two spatial dimensions.
Your input tensor has only two spatial dimensions and it lacks the mini-batch and channel dimensions. In your case these two dimensions are actually singelton dimensions (dimensions with size=1).

try:

conv1(tensor1[None, None, ...])


Instruction: Which ImageNet classes is PyTorch trained on?

Input: PyTorch lets you download pre-trained models for classification here, but it's not clear how to go from the output tensor of probabilities back to the actual class labels. Any ideas?

Output:
See example here: https://discuss.pytorch.org/t/imagenet-classes/4923
https://www.learnopencv.com/pytorch-for-beginners-image-classification-using-pre-trained-models/

You need to download imagenet_classes.txt