Instruction: How to change the axis on which 1 dimensional convolution is performed on embedding layer in PyTorch?

Input:
I've been playing around with text classification in PyTorch and I've encountered a problem with 1 dimensional convolutions.

I've set an embedding layer of dimesions (x, y, z) where:
x - denotes the batch size
y - denotes the length of a sentence (fixed with padding, so 40 words)
z - the dimensionality of pre-trained word embedding (for now 100)

For simplicity sake, let's assume I put in a matrix of (1,40, 100)

However, when to my knowledge once I perform torch.nn.conv1d(*args),
The resulting matrix becomes (batch size = 1, word size = 40, feature map size = 98) with kernel size of 3.

Basically, as I understand it convolves around y axis instead of x axis and it turn does not capture the spacial relationship between word embeddings.

Is there any way to change the convolutional layer so it calculates feature maps around different axis?

TL, DR:

Torch conv1d layer behaves this way on embedding layer:
enter image description here

But I want it to behave like this

enter image description here

Any help would be much appreciated.

Output:
conv1d expects the input's size to be (batch_size, num_channels, length) and there is no way to change that, so you have two possible ways ahead of you, you can either permute the output of embedding or you can use a conv1d instead of you embedding layer(in_channels = num_words, out_channels=word_embedding_size, and kernel_size=1) which is slower than embedding and not a good idea!

input = torch.randint(0, 10, (batch_size, sentence_length))
embeddings = word_embedding(input) #(batch_size, sentence_length, embedding_size)
embeddings_permuted = embeddings.permute(0, 2, 1) #(batch_size, embedding_size, sentence_length)
conv_out = convolution(embeddings_permuted) #(batch_size, conv_out_channels, changed_sentence_length)
#now you can either use the output as it is or permute it back (based on your upper layers)
#also note that I wrote changed_sentence_length because it is a fucntion of your padding and stride


Instruction: What's the difference between sum and torch.sum for a torch Tensor?

Input: I get the same results when using either the python sum or torch.sum so why did torch implement a sum function? Is there a difference between them?

Output:
nothing, torch.sum calls tensor.sum and python's sum calls __add__ (or __radd__ when needed) which calls tensor.sum again

so the only difference is in the number of function calls, and tensor.sum() should be the fastest (when you have small tensors and the function call's overhead is considerable)


Instruction: Import LSTM from Tensorflow to PyTorch by hand

Input:
I am trying to import a pretrained Model from tensorflow to PyTorch. It takes a single input and maps it onto a single output.
Confusion comes up, when I try to import the LSTM weights

I read the weights and their variables from the file with the following function:

def load_tf_model_weights():

    modelpath = 'models/model1.ckpt.meta'

    with tf.Session() as sess:
        tf.train.import_meta_graph(modelpath)
        init = tf.global_variables_initializer()
        sess.run(init)
        vars = tf.trainable_variables()
        W = sess.run(vars)

    return W,vars

W,V = load_tf_model_weights()

Then I am inspecting the shapes of the weights

In [33]:  [w.shape for w in W]
Out[33]: [(51, 200), (200,), (100, 200), (200,), (50, 1), (1,)]

furthermore the variables are defined as

In [34]:    V
Out[34]:
[<tf.Variable 'rnn/multi_rnn_cell/cell_0/lstm_cell/kernel:0' shape=(51, 200) dtype=float32_ref>,
<tf.Variable 'rnn/multi_rnn_cell/cell_0/lstm_cell/bias:0' shape=(200,) dtype=float32_ref>,
<tf.Variable 'rnn/multi_rnn_cell/cell_1/lstm_cell/kernel:0' shape=(100, 200) dtype=float32_ref>,
<tf.Variable 'rnn/multi_rnn_cell/cell_1/lstm_cell/bias:0' shape=(200,) dtype=float32_ref>,
<tf.Variable 'weight:0' shape=(50, 1) dtype=float32_ref>,
<tf.Variable 'FCLayer/Variable:0' shape=(1,) dtype=float32_ref>]

So I can say that the first element of  W defines the  Kernel of an LSTM and the second element define its bias. According to this post, the shape for the Kernel is defined as
[input_depth + h_depth, 4 * self._num_units]
and the bias as [4 * self._num_units]. We already know that input_depth is 1. So we get, that h_depth and _num_units both have the value 50.

In pytorch my LSTMCell, to which I want to assign the weights, looks like this:

In [38]: cell = nn.LSTMCell(1,50)
In [39]: [p.shape for p in cell.parameters()]
Out[39]:
[torch.Size([200, 1]),
torch.Size([200, 50]),
torch.Size([200]),
torch.Size([200])]

The first two entries can be covered by the first value of W which has the shape (51,200). But the LSTMCell from Tensorflow yields only one bias of shape (200) while pytorch wants two of them

And by leaving the bias out I have weights left over:

cell2 = nn.LSTMCell(1,50,bias=False)
[p.shape for p in cell2.parameters()]
Out[43]: [torch.Size([200, 1]), torch.Size([200, 50])]

Thanks!

Output:
pytorch uses CuDNN's LSTM underlayer(even when you don't have CUDA, it still uses something compatible) thus it has one extra bias term.

So you can pick two numbers with their sum equal to 1(0 and 1, 1/2 and 1/2 or anything else) and set your pytorch biases as those numbers times TF's bias.

pytorch_bias_1 = torch.from_numpy(alpha * tf_bias_data)
pytorch_bias_2 = torch.from_numpy((1.0-alpha) * tf_bias_data)


Instruction: pytorch: get number of classes given an ImageFolder dataset

Input:
If I have a dataset like:

image_datasets['train'] = datasets.ImageFolder(train_dir, transform=train_transforms)

How do I determine programatically the number of classes or unique labels in the dataset?

Output:
Use:

len(image_datasets['train'].classes)

.classes returns a list.


Instruction: How to open pretrained models in python

Input:
Hi I'm trying to load some pretrained models from .sav files and so far nothing is working. The models were originally made in pytorch and when I open the raw file in vs-code I can see that all the appropiate information was stored correctly.

I've tried the following libraries:

sklearn.externals.joblib

pickle

scipy.io

pyreadstat

Each library either gave me an error (such as wrong timestamp or signature mismatch) or just return an int instead of a python object.

The models can be downloaded from this link.

Output:
You need to use PyTorch to load the models. On top of this, you also need the original model definition, so you need to need the clone the authors repository. In your example this repo:

git clone https://github.com/tbepler/protein-sequence-embedding-iclr2019.git

Then you can open the model with torch.load(). Note that you need the model definition on your path (you can simply launch python from the repo directory).

Then it's straightforward to open a file:

import torch
model = torch.load('<downloaded models>/<model name>.sav')
print(model)

The last line prints the model definition. For example,  me_L1_100d_lstm3x512_lm_i512_mb64_tau0.5_p0.05_epoch100.sav produced the following output:

OrdinalRegression(
  (embedding): StackedRNN(
    (embed): LMEmbed(
      (lm): BiLM(
        (embed): Embedding(22, 21, padding_idx=21)
        (dropout): Dropout(p=0)
        (rnn): ModuleList(
          (0): LSTM(21, 1024, batch_first=True)
          (1): LSTM(1024, 1024, batch_first=True)
        )
        (linear): Linear(in_features=1024, out_features=21, bias=True)
      )
      (embed): Embedding(21, 512, padding_idx=20)
      (proj): Linear(in_features=4096, out_features=512, bias=True)
      (transform): ReLU()
    )
    (dropout): Dropout(p=0)
    (rnn): LSTM(512, 512, num_layers=3, batch_first=True, bidirectional=True)
    (proj): Linear(in_features=1024, out_features=100, bias=True)
  )
  (compare): L1()
)


Instruction: How to add parameters from self.some_dictionary_of_modules to self.parameters?

Input:
Consider this simple example:

import torch

class MyModule(torch.nn.Module):
    def __init__(self):
        super(MyModule, self).__init__()
        self.conv_0=torch.nn.Conv2d(3,32,3,stride=1,padding=0)
        self.blocks=torch.nn.ModuleList([
            torch.nn.Conv2d(3,32,3,stride=1,padding=0),
            torch.nn.Conv2d(32,64,3,stride=1,padding=0)])

        #the problematic part
        self.dict_block={"key_1": torch.nn.Conv2d(64,128,3,1,0),
                "key_2": torch.nn.Conv2d(56,1024,3,1,0)}

if __name__=="__main__":
    my_module=MyModule()
    print(my_module.parameters)

The output I get here is (notice that the parmeters from self.dict_block are missing)

<bound method Module.parameters of MyModule(
  (conv_0): Conv2d(3, 32, kernel_size=(3, 3), stride=(1, 1))
  (blocks): ModuleList(
    (0): Conv2d(3, 32, kernel_size=(3, 3), stride=(1, 1))
    (1): Conv2d(32, 64, kernel_size=(3, 3), stride=(1, 1))
  )
)>

This means that if I want the parameters of self.dict_block to be optimised, I will need to use something  like

my_optimiser.add_param_group({"params": params_from_self_dict})

before using my optimiser. However, I think there might be a more straight-forward alternative that will add the parameters of self.dict_block to the parameters of the my_module_object. Something that comes close is nn.Parameter(...) as explained here, but this requires the input to be a tensor.

Output:
Found the answer. Posting it in case someone runs into the same problem:

Looking into <torch_install>/torch/nn/modules/container.py it looks like there is a class torch.nn.ModuleDict that just does that. So in the example I gave in the question, the solution would be:

self.dict_block=torch.nn.ModuleDict({"key_1": torch.nn.Conv2d(64,128,3,1,0),
            "key_2": torch.nn.Conv2d(56,1024,3,1,0)})


Instruction: how to get torch.Size([1, 3, 16, 112, 112]) instead of torch.Size([1, 3, 16, 64, 64]) after ConvTranspose3d

Input:
I have a torch.Size([1, 64, 8, 32, 32]) which I want after my transpose 3d convolution to become torch.Size([1, 3, 16, 112, 112]).

Using this: nn.ConvTranspose3d(64, 3, kernel_size=4, stride=2, bias=False, padding=(1, 1, 1)) I get correct the output channels and the number of frames, but not the frame sizes:torch.Size([1, 3, 16, 64, 64])

What should I change in order to the right torch.Size?

Output:
you should use different strides and paddings for different dims.

ConvTranspose3d(64, 3, kernel_size=4, stride=(2, 4, 4), bias=False, padding=(1, 8, 8))


Instruction: Is Feature Scaling recommended for AutoEncoder?

Input:
Problem:

The Staked Auto Encoder is being applied to a dataset with 25K rows and 18 columns, all float values.
SAE is used for feature extraction with encoding & decoding.

When I train the model without feature scaling, the loss is around 50K, even after 200 epochs. But, when scaling is applied the loss is around 3 from the first epoch.

My questions:

Is it recommended to apply feature scaling when SAE is used for feature extraction
Does it impact accuracy during decoding?

Output:
With a few exceptions, you should always apply feature scaling in machine learning, especially when working with gradient descent as in your SAE. Scaling your features will ensure a much smoother cost function and thus faster convergence to global (hopefully) minima.

Also worth noting that your much smaller loss after 1 epoch with scaling should be a result of much smaller values used to compute the loss.

No


Instruction: PyTorch did not compute gradient and update parameters for 'masking' tensors?

Input:
I am coding a LeNet in PyTorch for MNIST dataset; I add a tensor self.mask_fc1\2\3 to mask some certain connections for the full connection layers. The code is like this:

import torch
import torchvision
import torchvision.transforms as transforms
import matplotlib.pyplot as plt
import numpy as np
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

def loadMNIST():
    transform = transforms.Compose([transforms.ToTensor()])

    trainset = torchvision.datasets.MNIST(root='./data', train=True,
                                          download=True, transform=transform)
    trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,
                                              shuffle=True, num_workers=2)

    testset = torchvision.datasets.MNIST(root='./data', train=False,
                                         download=True, transform=transform)
    testloader = torch.utils.data.DataLoader(testset, batch_size=4,
                                             shuffle=False, num_workers=2)
    return trainloader, testloader

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 6, 5, 1, 2)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

        self.mask_fc1 = torch.ones(16 * 5 * 5, 120, requires_grad=True)
        self.mask_fc2 = torch.ones(120, 84, requires_grad=True)
        self.mask_fc3 = torch.ones(84, 10, requires_grad=True)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, self.num_flat_features(x))

        # first layer
        x = x.matmul(self.fc1.weight.t() * self.mask_fc1)
        if self.fc1.bias is not None:
            x += torch.jit._unwrap_optional(self.fc1.bias)
        x = F.relu(x)

        # second layer
        x = x.matmul(self.fc2.weight.t() * self.mask_fc2)
        if self.fc2.bias is not None:
            x += torch.jit._unwrap_optional(self.fc2.bias)
        x = F.relu(x)

        # third layer
        x = x.matmul(self.fc3.weight.t() * self.mask_fc3)
        if self.fc3.bias is not None:
            x += torch.jit._unwrap_optional(self.fc3.bias)

        return x

    def num_flat_features(self, x):
        size = x.size()[1:]
        num_features = 1
        for s in size:
            num_features *= s
        return num_features

if __name__ == '__main__':
    trainloader, testloader = loadMNIST()

    net = Net()

    # train
    criterion = nn.CrossEntropyLoss()
    optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)

    for epoch in range(2):
        running_loss = 0.0
        for i, data in enumerate(trainloader, 0):
            inputs, labels = data

            optimizer.zero_grad()

            outputs = net(inputs)
            loss = criterion(outputs, labels)
            loss.backward()
            optimizer.step()

            # print
            running_loss += loss.item()
            if i % 2000 == 1999:
                # mean loss
                print('[%d, %5d] loss: %.3f' % (epoch + 1, i + 1, running_loss / 2000))
                running_loss = 0.0

    print('Finished Training')

    # print the mask
    print(net.mask_fc1)

I achieve the masking in the forward function and implement linear layer by myself rather than call x = F.relu(self.fc1(x)), and the model performs normally (at last in terms of the loss and accuracy).

However, when I print self.mask_fc1/2/3, the tensor does not change during the training. Since the tensor was set requires_grad=True in function __init__, I cannot understand why it does not change. Maybe it is because the tensor multiplication?

Output:
For training, you need to register mask_fc1/2/3 as module parameters:

self.mask_fc1 = nn.Parameter(torch.ones(16 * 5 * 5, 120))

You can print net.parameters() after that to confirm.


Instruction: Need help understanding the label input in a CGAN

Input:
I am trying to implement a CGAN. I understand that in convolutional generator and discriminator models, you add volume to the inputs by adding depth that represents the label. So if you have 10 classes in your data, your generator and discriminator would both have the base depth + 10 as its input volume.

However, I am reading various implementations online and I can't seem to find where they are actually acquiring this label. Surely CGANs can't be unsupervised because you need to obtain the label to input. e.g. in cifar10 if you are training the discriminator on a real image of a frog, you would need the 'frog' annotation.

Here is one of the pieces of code I am studying:

class CGAN(object):
def __init__(self, args):
    # parameters
    self.epoch = args.epoch
    self.batch_size = args.batch_size
    self.save_dir = args.save_dir
    self.result_dir = args.result_dir
    self.dataset = args.dataset
    self.log_dir = args.log_dir
    self.gpu_mode = args.gpu_mode
    self.model_name = args.gan_type
    self.input_size = args.input_size
    self.z_dim = 62
    self.class_num = 10
    self.sample_num = self.class_num ** 2

    # load dataset
    self.data_loader = dataloader(self.dataset, self.input_size, self.batch_size)
    data = self.data_loader.__iter__().__next__()[0]

    # networks init
    self.G = generator(input_dim=self.z_dim, output_dim=data.shape[1], input_size=self.input_size, class_num=self.class_num)
    self.D = discriminator(input_dim=data.shape[1], output_dim=1, input_size=self.input_size, class_num=self.class_num)
    self.G_optimizer = optim.Adam(self.G.parameters(), lr=args.lrG, betas=(args.beta1, args.beta2))
    self.D_optimizer = optim.Adam(self.D.parameters(), lr=args.lrD, betas=(args.beta1, args.beta2))

    if self.gpu_mode:
        self.G.cuda()
        self.D.cuda()
        self.BCE_loss = nn.BCELoss().cuda()
    else:
        self.BCE_loss = nn.BCELoss()

    print('---------- Networks architecture -------------')
    utils.print_network(self.G)
    utils.print_network(self.D)
    print('-----------------------------------------------')

    # fixed noise & condition
    self.sample_z_ = torch.zeros((self.sample_num, self.z_dim))
    for i in range(self.class_num):
        self.sample_z_[i*self.class_num] = torch.rand(1, self.z_dim)
        for j in range(1, self.class_num):
            self.sample_z_[i*self.class_num + j] = self.sample_z_[i*self.class_num]

    temp = torch.zeros((self.class_num, 1))
    for i in range(self.class_num):
        temp[i, 0] = i

    temp_y = torch.zeros((self.sample_num, 1))
    for i in range(self.class_num):
        temp_y[i*self.class_num: (i+1)*self.class_num] = temp

    self.sample_y_ = torch.zeros((self.sample_num, self.class_num)).scatter_(1, temp_y.type(torch.LongTensor), 1)
    if self.gpu_mode:
        self.sample_z_, self.sample_y_ = self.sample_z_.cuda(), self.sample_y_.cuda()

def train(self):
    self.train_hist = {}
    self.train_hist['D_loss'] = []
    self.train_hist['G_loss'] = []
    self.train_hist['per_epoch_time'] = []
    self.train_hist['total_time'] = []

    self.y_real_, self.y_fake_ = torch.ones(self.batch_size, 1), torch.zeros(self.batch_size, 1)
    if self.gpu_mode:
        self.y_real_, self.y_fake_ = self.y_real_.cuda(), self.y_fake_.cuda()

    self.D.train()
    print('training start!!')
    start_time = time.time()
    for epoch in range(self.epoch):
        self.G.train()
        epoch_start_time = time.time()
        for iter, (x_, y_) in enumerate(self.data_loader):
            if iter == self.data_loader.dataset.__len__() // self.batch_size:
                break

            z_ = torch.rand((self.batch_size, self.z_dim))
            y_vec_ = torch.zeros((self.batch_size, self.class_num)).scatter_(1, y_.type(torch.LongTensor).unsqueeze(1), 1)
            y_fill_ = y_vec_.unsqueeze(2).unsqueeze(3).expand(self.batch_size, self.class_num, self.input_size, self.input_size)
            if self.gpu_mode:
                x_, z_, y_vec_, y_fill_ = x_.cuda(), z_.cuda(), y_vec_.cuda(), y_fill_.cuda()

            # update D network
            self.D_optimizer.zero_grad()

            D_real = self.D(x_, y_fill_)
            D_real_loss = self.BCE_loss(D_real, self.y_real_)

            G_ = self.G(z_, y_vec_)
            D_fake = self.D(G_, y_fill_)
            D_fake_loss = self.BCE_loss(D_fake, self.y_fake_)

            D_loss = D_real_loss + D_fake_loss
            self.train_hist['D_loss'].append(D_loss.item())

            D_loss.backward()
            self.D_optimizer.step()

            # update G network
            self.G_optimizer.zero_grad()

            G_ = self.G(z_, y_vec_)
            D_fake = self.D(G_, y_fill_)
            G_loss = self.BCE_loss(D_fake, self.y_real_)
            self.train_hist['G_loss'].append(G_loss.item())

            G_loss.backward()
            self.G_optimizer.step()

It seems as if y_vec_ and y_fill_ are the labels for the images, but in the instance of y_fill_ which is used to label real images for the discriminator, it equals y_fill_ = y_vec_.unsqueeze(2).unsqueeze(3).expand(self.batch_size, self.class_num, self.input_size, self.input_size)

It doesn't seem like its getting any information on the label from the dataset?
How is it giving the discriminator the correct label?

Thanks!

Output: y_fill_ is based on y_vec_ which is based on y_, so they are reading label info from mini batches which is correct. you might be confused by the scatter operation, basically what the code's doing is transferring the label into one-hot encoding


Instruction: PyTorch preferred way to copy a tensor

Input:
There seems to be several ways to create a copy of a tensor in PyTorch, including
y = tensor.new_tensor(x) #a

y = x.clone().detach() #b

y = torch.empty_like(x).copy_(x) #c

y = torch.tensor(x) #d

b is explicitly preferred over a and d according to a UserWarning I get if I execute either a or d. Why is it preferred? Performance? I'd argue it's less readable.
Any reasons for/against using c?

Output:
TL;DR
Use .clone().detach() (or preferrably .detach().clone())

If you first detach the tensor and then clone it, the computation path is not copied, the other way around it is copied and then abandoned. Thus, .detach().clone() is very slightly more efficient.-- pytorch forums

as it's slightly fast and explicit in what it does.

Using perflot, I plotted the timing of various methods to copy a pytorch tensor.
y = tensor.new_tensor(x) # method a

y = x.clone().detach() # method b

y = torch.empty_like(x).copy_(x) # method c

y = torch.tensor(x) # method d

y = x.detach().clone() # method e

The x-axis is the dimension of tensor created, y-axis shows the time. The graph is in linear scale. As you can clearly see, the tensor() or new_tensor() takes more time compared to other three methods.

Note: In multiple runs, I noticed that out of b, c, e, any method can have lowest time. The same is true for a and d. But, the methods b, c, e consistently have lower timing than a and d.
import torch
import perfplot

perfplot.show(
    setup=lambda n: torch.randn(n),
    kernels=[
        lambda a: a.new_tensor(a),
        lambda a: a.clone().detach(),
        lambda a: torch.empty_like(a).copy_(a),
        lambda a: torch.tensor(a),
        lambda a: a.detach().clone(),
    ],
    labels=["new_tensor()", "clone().detach()", "empty_like().copy()", "tensor()", "detach().clone()"],
    n_range=[2 ** k for k in range(15)],
    xlabel="len(a)",
    logx=False,
    logy=False,
    title='Timing comparison for copying a pytorch tensor',
)


Instruction: Pytorch - why does preallocating memory cause "trying to backward through the graph a second time"

Input:
Suppose I have a simple one-hidden-layer network that I'm training in the typical way:

    for x,y in trainData:
        optimizer.zero_grad()
        out = self(x)
        loss = self.lossfn(out, y)
        loss.backward()
        optimizer.step()

This works as expected, but if I instead pre-allocate and update the output array, I get an error:

    out = torch.empty_like(trainData.tensors[1])
    for i,(x,y) in enumerate(trainData):
        optimizer.zero_grad()
        out[i] = self(x)
        loss = self.lossfn(out[i], y)
        loss.backward()
        optimizer.step()

  RuntimeError: Trying to backward through the graph a second time, but
  the buffers have already been freed. Specify retain_graph=True when
  calling backward the first time.

What's happening here that in the second version Pytorch attempts to backward through the graph again? Why is this not an issue in the first version? (Note that this error occurs even if I don't zero_grad())

Output:
The error implies that the program is trying to backpropagate through a set of operations a second time. The first time you backpropagate through a set of operations, pytorch deletes the computational graph to free memory. Therefore, the second time you try to backpropagate it fails as the graph has already been deleted.

Here's a detailed explanation of the same.

Short answer

Use loss.backward(retain_graph=True). This will not delete the computational graph.

Detailed answer

In the first version, in each loop iteration, a new computational graph is generated every time out = self(x) is run.

Every loop's graph
out = self(x) -> loss = self.lossfn(out, y)

In the second version, since out is declared outside the loop, the computational graphs in every loop have a parent node outside.

           - out[i] = self(x) -> loss = self.lossfn(out[i], y)
out[i] - | - out[i] = self(x) -> loss = self.lossfn(out[i], y)
           - out[i] = self(x) -> loss = self.lossfn(out[i], y)

Therefore, here's a timeline of what happens.

The first iteration runs
The computation graph is deleted including the parent node
The second iteration attempts to backpropagate but failed since it didn't find the the parent node


Instruction: pytorch linear regression given wrong results

Input:
I implemented a simple linear regression and I’m getting some poor results. Just wondering if these results are normal or I’m making some mistake.

I tried different optimizers and learning rates, I always get bad/poor results

Here is my code:

import torch
import torch.nn as nn
import numpy as np
import matplotlib.pyplot as plt
from torch.autograd import Variable

class LinearRegressionPytorch(nn.Module):

   def __init__(self, input_dim=1, output_dim=1):

       super(LinearRegressionPytorch, self).__init__()
       self.linear = nn.Linear(input_dim, output_dim)

   def forward(self,x):
       x = x.view(x.size(0),-1)
       y = self.linear(x)
       return y

input_dim=1
output_dim = 1
if torch.cuda.is_available():
   model = LinearRegressionPytorch(input_dim, output_dim).cuda()
else:
   model = LinearRegressionPytorch(input_dim, output_dim)

criterium = nn.MSELoss()
l_rate =0.00001
optimizer = torch.optim.SGD(model.parameters(), lr=l_rate)
#optimizer = torch.optim.Adam(model.parameters(),lr=l_rate)
epochs = 100

#create data
x = np.random.uniform(0,10,size = 100) #np.linspace(0,10,100);
y = 6*x+5
mu = 0
sigma = 5
noise =  np.random.normal(mu, sigma, len(y))
y_noise = y+noise

#pass it to pytorch
x_data = torch.from_numpy(x).float()
y_data = torch.from_numpy(y_noise).float()
if torch.cuda.is_available():
   inputs = Variable(x_data).cuda()
   target = Variable(y_data).cuda()
else:
   inputs = Variable(x_data)
   target = Variable(y_data)

for epoch in range(epochs):

   #predict data
   pred_y= model(inputs)

   #compute loss
   loss = criterium(pred_y, target)

   #zero grad and optimization
   optimizer.zero_grad()
   loss.backward()
   optimizer.step()

   #if epoch % 50 == 0:
   #   print(f'epoch = {epoch}, loss =  {loss.item()}')

#print params
for name, param in model.named_parameters():
   if param.requires_grad:
       print(name, param.data)

There are the poor results :

linear.weight tensor([[1.7374]], device='cuda:0')
linear.bias tensor([0.1815], device='cuda:0')

The results should be weight = 6 , bias = 5

Output:
Problem Solution

Actually your batch_size is problematic. If you have it set as one, your targetneeds the same shape as outputs (which you are, correctly, reshaping with view(-1, 1)).

Your loss should be defined like this:

loss = criterium(pred_y, target.view(-1, 1))

This network is correct

Results

Your results will not be bias=5 (yes, weight will go towards 6 indeed) as you are adding random noise to target (and as it's a single value for all your data points, only bias will be affected).

If you want bias equal to 5 remove addition of noise.

You should increase number of your epochs as well, as your data is quite small and network (linear regression in fact) is not really powerful. 10000 say should be fine and your loss should oscillate around 0 (if you change your noise to something sensible).

Noise

You are creating multiple gaussian distributions with different variations, hence your loss would be higher. Linear regression is unable to fit your data and find sensible bias (as the optimal slope is still approximately 6 for your noise, you may try to increase multiplication of 5 to 1000 and see what weight and bias will be learned).

Style (a little offtopic)

Please read documentation about PyTorch and keep your code up to date (e.g. Variable is deprecated in favor of Tensor and rightfully so).

This part of code:

x_data = torch.from_numpy(x).float()
y_data = torch.from_numpy(y_noise).float()
if torch.cuda.is_available():
    inputs = Tensor(x_data).cuda()
    target = Tensor(y_data).cuda()
else:
    inputs = Tensor(x_data)
    target = Tensor(y_data)

Could be written succinctly like this (without much thought):

inputs = torch.from_numpy(x).float()
target = torch.from_numpy(y_noise).float()
if torch.cuda.is_available():
    inputs = inputs.cuda()
    target = target.cuda()

I know deep learning has it's reputation for bad code and fatal practice, but please do not help spreading this approach.


Instruction: PyTorch - GPU is not used by tensors despite CUDA support is detected

Input:
As the title of the question clearly describes, even though torch.cuda.is_available() returns True, CPU is used instead of GPU by tensors. I have set the device of the tensor to GPU through the images.to(device) function call after defining the device. When I debug my code, I am able to see that the device is set to cuda:0; but the tensor's device is still set to cpu.

Defining the device:

use_cuda = torch.cuda.is_available()  # returns True
device = torch.device('cuda:0' if use_cuda else 'cpu')

Determining the device of the tensors:

for epoch in range(num_epochs):
    for i, (images, labels) in enumerate(train_loader):
        images.to(device)
        labels.to(device)
        # both of images and labels' devices are set to cpu

The software stack:

Python 3.7.1
torch 1.0.1
Windows 10 64-bit

p.s. PyTorch is installed with the option of Cuda 9.0 support.

Output:
tensor.to() does not modify the tensor inplace. It returns a new tensor that's stored
in the specified device.

Use the following instead.

 images = images.to(device)
 labels = labels.to(device)


Instruction: Determining the result of a convolution operation

Input:
Following guide from https://medium.com/mlreview/a-guide-to-receptive-field-arithmetic-for-convolutional-neural-networks-e0f514068807 I'm attempting to calculate the number of output features using code below :

The output of :

%reset -f

import torch
import torch.nn as nn

my_tensor = torch.randn((1, 16, 12, 12), requires_grad=False)
print(my_tensor.shape)

update_1 = nn.Conv2d(16, 16, kernel_size=3, stride=2, padding=1)
print(update_1(my_tensor).shape)

is :

torch.Size([1, 16, 12, 12])
torch.Size([1, 16, 6, 6])

How is torch.Size([1, 16, 6, 6]) calculated in the context of applying formula :

(taken from https://medium.com/mlreview/a-guide-to-receptive-field-arithmetic-for-convolutional-neural-networks-e0f514068807)

Attempting to calculate the number of output features manually by applying the formula :

stride = 2
padding = 1
kernel_size = 3

# 2304 as n_in = 1 * 16 * 16 * 12

n_out = ((2304 + (2 * padding) - kernel_size) / stride) + 1

print(n_out)

prints : 1152.5

But the produced number of output features is print(1 * 16 * 6 *6) = 576. I've taken the product of 1,16,6,6 as this is the shape of the result of print(update_1(my_tensor).shape)

Update :

Based on questions below I've updated code to :

%reset -f

import torch
import torch.nn as nn
from math import floor

stride_value = 2
padding_value = 1
kernel_size_value = 3

number_channels = 3
width = 10
height = 12

my_tensor = torch.randn((1, number_channels, width, height), requires_grad=False)
print(my_tensor.shape)

update_1 = nn.Conv2d(in_channels=number_channels,
                     out_channels=16,
                     kernel_size=kernel_size_value,
                     stride=stride_value,
                     padding=padding_value)

print(update_1(my_tensor).shape)

n_out = floor((number_channels + (2 * padding_value) - kernel_size_value) / stride_value) + 1
print(n_out)

print(my_tensor.shape) produces :
torch.Size([1, 3, 10, 12])

print(update_1(my_tensor).shape) produces :
torch.Size([1, 16, 5, 6])

print(update_1(n_out).shape) produces :
2

2 does not match the number of output features in each dimension. Have I implemented the calculation correctly ?

As the number of horizontal features produces is 5 and number of vertical features produces is 6 is this formula not applicable where the number of features differ as for an image it does not make sense to have differing x and y axis values length ?

Output:
I see where you confusion is coming from. The formula computes the linear number of outputs, whereas you assume that it operates on the whole tensor.

So the correct code is:

from math import floor

stride = 2
padding = 1
kernel_size = 3

n_out = floor((12 + (2 * padding) - kernel_size) / stride) + 1

print(n_out)

Therefore, it outputs 6 "horizontal" features.
Since the input tensor has the same "vertical" dimension (12), the formula will also produce 6 "vertical" features.
Finally, 16 is the number of output channels you have specified in Conv2d.

Putting it all together, the output is

1 image in a batch,
16 channels,
6 horizontal features, and
6 vertical features,

which totals in 576 features.

UPDATE

By convention, the number of output channels is not calculated by the formula, but provided manually as a second parameter to nn.Conv2d.

Therefore, to correct the second code above:

import torch
import torch.nn as nn
from math import floor

stride_value = 2
padding_value = 1
kernel_size_value = 3

number_channels = 3
width = 10
height = 12

my_tensor = torch.randn((1, number_channels, width, height), requires_grad=False)
print(my_tensor.shape)

update_1 = nn.Conv2d(in_channels=number_channels,
                     out_channels=16,
                     kernel_size=kernel_size_value,
                     stride=stride_value,
                     padding=padding_value)

print(update_1(my_tensor).shape)

n_out1 = floor((width + (2 * padding_value) - kernel_size_value) / stride_value) + 1
n_out2 = floor((height + (2 * padding_value) - kernel_size_value) / stride_value) + 1
print("(Expected: 5, 6)", n_out1, n_out2)


Instruction: Different methods for initializing embedding layer weights in Pytorch

Input:
There seem to be two ways of initializing embedding layers in Pytorch 1.0 using an uniform distribution.

For example you have an embedding layer:

self.in_embed = nn.Embedding(n_vocab, n_embed)

And you want to initialize its weights with an uniform distribution. The first way you can get this done is:

self.in_embed.weight.data.uniform_(-1, 1)

And another one would be:

nn.init.uniform_(self.in_embed.weight, -1.0, 1.0)

My question is: what is the difference between the first and second initialization form. Do both methods do the same thing?

Output:
Both are same

torch.manual_seed(3)
emb1 = nn.Embedding(5,5)
emb1.weight.data.uniform_(-1, 1)

torch.manual_seed(3)
emb2 = nn.Embedding(5,5)
nn.init.uniform_(emb2.weight, -1.0, 1.0)

assert torch.sum(torch.abs(emb1.weight.data - emb2.weight.data)).numpy() == 0

Every tensor has a uniform_ method which initializes it with the values from the uniform distribution. Also, the nn.init module has a method uniform_ which takes in a tensor and inits it with values from uniform distribution. Both are same expect first one is using the  member function and the second is using a general utility function.


Instruction: ValueError: operands could not be broadcast together with shapes (50,50,512) (3,) (50,50,512) while converting tensor to image in pytorch

Input:
I'm doing a neural style transfer. I'm trying to reconstruct the output of the convolutional layer conv4_2 of the VGG19 network.

def get_features(image, model):

    layers = {'0': 'conv1_1', '5': 'conv2_1',  '10': 'conv3_1',
              '19': 'conv4_1', '21': 'conv4_2', '28': 'conv5_1'}
    x = image
    features = {}

    for name, layer in model._modules.items():
        x = layer(x)

        if name in layers:
            features[layers[name]] = x

    return features

content_img_features = get_features(content_img, vgg)
style_img_features   = get_features(style_img, vgg)

target_content = content_img_features['conv4_2']

content_img_features is a dict that contains the output of every layer.
target_content is a tensor of shape torch.Size([1, 512, 50, 50])

This is the method I use to plot the image using the tensor. It works fine for the input image as well as the final output.

def tensor_to_image(tensor):
    image = tensor.clone().detach()
    image = image.numpy().squeeze()
    image = image.transpose(1, 2, 0)
    image *= np.array((0.22, 0.22, 0.22))+ np.array((0.44, 0.44, 0.44))
    image = image.clip(0, 1)
    return image

image = tensor_to_image(target_content)
fig = plt.figure()
plt.imshow(image)

But this throws the error,

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-188-a75a5f0743bb> in <module>()
      1
----> 2 image = tensor_to_image(target_content)
      3 fig = plt.figure()
      4 plt.imshow(image)

<ipython-input-186-e9385dbc4a85> in tensor_to_image(tensor)
      3     image = image.numpy().squeeze()
      4     image = image.transpose(1, 2, 0)
----> 5     image *= np.array((0.22, 0.22, 0.22))+ np.array((0.44, 0.44, 0.44))
      6     image = image.clip(0, 1)
      7     return image

ValueError: operands could not be broadcast together with shapes (50,50,512) (3,) (50,50,512)

This is the initial transformation I apply to the image before passing to the cnn layers,

def transformation(img):

    tasks = tf.Compose([tf.Resize(400), tf.ToTensor(),
            tf.Normalize((0.44,0.44,0.44),(0.22,0.22,0.22))])
    img = tasks(img)[:3,:,:].unsqueeze(0)

    return img

How do I fix this? Is there another way to reconstruct the image from the convolution layer?

Output:
Your tensor_to_image method only works for 3 channel images. Your input to the network is 3 channels, so is the final output, therefore it works fine there. But you cannot do the same at an internal high dimensional activation.

Essentially the problem is that you try to apply a channel-wise normalization, but you have parameters for only three channels, that's why that particular line fails. You would need a 512 element vector of means and standard deviations. So for example this would work:

image *= np.random.random([512]) + np.random.random([512])

However the fundamental problem is still that you try to visualize a high dimensional, 512 channel image, instead a traditional 3 channel (RGB) image. You may try to visualize channels separately, or in groups of 3, but still it might not be really useful.


Instruction: RuntimeError: Expected object of backend CUDA but got backend CPU for argument: ret = torch.addmm(torch.jit._unwrap_optional(bias), input, weight.t())

Input:
When the forward function of my neural network (after the training phase is completed) is being executed, I'm experiencing RuntimeError: Expected object of backend CUDA but got backend CPU for argument #4 'mat1'. The error trace indicates the error happens due to the call of output = self.layer1(x) command. I have tried to move all the data of the tensors to my GPU. It seems I miss something to be moved as well.

Here is the code I have tried:

use_cuda = torch.cuda.is_available()
device = torch.device('cuda:0' if use_cuda else 'cpu')

class NeuralNet(nn.Module):

    def __init__(self, input_size, hidden_size, output_size):
        super(NeuralNet, self).__init__()
        self.layer1 = nn.Linear(input_size, hidden_size).cuda(device)
        self.layer2 = nn.Linear(hidden_size, output_size).cuda(device)
        self.relu = nn.ReLU().cuda(device)

    def forward(self, x):
        x.cuda(device)
        output = self.layer1(x)  # throws the error
        output = self.relu(output)
        output = self.layer2(output)
        return output

def main():
    transform = transforms.Compose([
        transforms.ToTensor()
    ])

    mnist_trainset = datasets.MNIST(root='D:\\MNIST', train=True, download=False, transform=transform)
    mnist_testset = datasets.MNIST(root='D:\\MNIST', train=False, download=False, transform=transform)

    train_loader = DataLoader(dataset=mnist_trainset, batch_size=100, shuffle=True)
    test_loader = DataLoader(dataset=mnist_testset, batch_size=100, shuffle=False)

    input_size = 784
    hidden_size = 500
    output_size = 10
    num_epochs = 5

    learning_rate = 0.001

    model = NeuralNet(input_size, hidden_size, output_size)
    model.cuda(device)

    lossFunction = nn.CrossEntropyLoss()
    lossFunction.cuda(device)
    optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

    losses_in_epochs = []
    total_step = len(train_loader)
    for epoch in range(num_epochs):
        for i, (images, labels) in enumerate(train_loader):
            images = images.to(device)
            labels = labels.to(device)
            images = images.reshape(-1, 28 * 28)

            out = model(images)
            loss = lossFunction(out, labels)

            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

            if (i + 1) % 100 == 0:
                print('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}'.format(epoch + 1, num_epochs, i + 1, total_step,
                                                                         loss.item()))

            if (i % 600) == 0:
                losses_in_epochs.append(loss.item())

    with torch.no_grad():
        correct = 0
        total = 0
        for images, labels in test_loader:
            images = images.reshape(-1, 28 * 28)
            out = model(images)
            _, predicted = torch.max(out.data, 1)
            total += labels.size(0)
            correct += (predicted == labels).sum().item()
            print('Accuracy of the network on the 10000 test images: {} %'.format(100 * correct / total))

if __name__ == '__main__':
    main()

The software stack:

Python 3.7.1
torch 1.0.1 (with Cuda 9.0)
Windows 10 64-bit

Output:
The error only happens only at the testing step, when you try calculating the accuracy, this might already give you a hint. The training loop runs without a problem.

The error is simply that you don't send the images and labels to the GPU at this step. This is your corrected evaluation loop:

with torch.no_grad():
    correct = 0
    total = 0
    for images, labels in test_loader:
        images = images.to(device)  # missing line from original code
        labels = labels.to(device)  # missing line from original code
        images = images.reshape(-1, 28 * 28)
        out = model(images)
        _, predicted = torch.max(out.data, 1)
        total += labels.size(0)
        correct += (predicted == labels).sum().item()

BTW you don't need to send all your layers to the GPU separately (at your class __init__()). It's better to just send the whole instantiated model to the gpu at once.


Instruction: Pytorch RNN HTML Generation

Input:
I’ m stuck for a couple of days trying to make and RNN network to learn a basic HTML template.
I tried different approaches and I even overfit on the following data:

<!DOCTYPE html>

<title>Page Title</title>

This is a Heading

This is a paragraph.

Obtaining 100% accuracy on training and validation using Adam Optimizer and CrossEntropyLoss.

The problem is that when I try to sample from the network, the results are completely random and I don’t know whats the problem:

..<a<a<a<a<aa<ttp11111b11111b11111111b11b1bbbb<btttn111

My sampling function is the following:

def sample_sentence():
    words = list()
    count = 0
    modelOne.eval()
    with torch.no_grad():
        # Setup initial input state, and input word (we use "the").
        previousWord = torch.LongTensor(1, 1).fill_(trainData.vocabulary['letter2id']['[START]'])
        hidden =  Variable(torch.zeros(6, 1, 100).to(device))

        while True:
            # Predict the next word based on the previous hidden state and previous word.
            inputWord = torch.autograd.Variable(previousWord.to(device))

            predictions, newHidden = modelOne(inputWord, hidden)

            hidden = newHidden

            pred = torch.nn.functional.softmax(predictions.squeeze()).data.cpu().numpy().astype('float64')

            pred = pred/np.sum(pred)

            nextWordId = np.random.multinomial(1, pred, 1).argmax()

            if nextWordId == 0:
                continue

            words.append(trainData.vocabulary['id2letter'][nextWordId])
            # Setup the inputs for the next round.
            previousWord.fill_(nextWordId)

            # Keep adding words until the [END] token is generated.
            if nextWordId == trainData.vocabulary['letter2id']['[END]']:
                break

            if count>20000:
                break
            count += 1
        words.insert(0, '[START]')

        return words

And my network architecture is here:

class ModelOne(Model) :
    def __init__(self,
                vocabulary_size,
                hidden_size,
                num_layers,
                rnn_dropout,
                embedding_size,
                dropout,
                num_directions):
        super(Model, self).__init__()

        self.vocabulary_size = vocabulary_size
        self.hidden_size = hidden_size
        self.num_layers = num_layers
        self.rnn_dropout = rnn_dropout
        self.dropout = dropout
        self.num_directions = num_directions
        self.embedding_size = embedding_size

        self.embeddings = nn.Embedding(self.vocabulary_size, self.embedding_size)
        self.rnn = nn.GRU(self.embedding_size,
                          self.hidden_size,
                          num_layers=self.num_layers,
                          bidirectional=True if self.num_directions==2 else False,
                          dropout=self.rnn_dropout,
                          batch_first=True)
        self.linear = nn.Linear(self.hidden_size*self.num_directions, self.vocabulary_size)

    def forward(self, paddedSeqs, hidden):

        batchSequenceLength = paddedSeqs.size(1)

        batchSize = paddedSeqs.size(0)

        lengths = paddedSeqs.ne(0).sum(dim=1)

        embeddingVectors = self.embeddings(paddedSeqs)

        x = torch.nn.utils.rnn.pack_padded_sequence(embeddingVectors, lengths, batch_first=True)

        self.rnn.flatten_parameters()

        x,hid = self.rnn(x, hidden)

        output, _ = torch.nn.utils.rnn.pad_packed_sequence(x, batch_first=True, padding_value=0, total_length=batchSequenceLength)

        predictions = self.linear(output)

        return predictions.view(batchSize, self.vocabulary_size, batchSequenceLength), hid

    def init_hidden(self, paddedSeqs):
        hidden = Variable(torch.zeros(self.num_layers*self.num_directions,
                                    1,
                                    self.hidden_size).to(device))
        return hidden

modelOne =ModelOne(vocabulary_size=vocabularySize,
                 hidden_size=100,
                 embedding_size=50,
                 num_layers=3,
                 rnn_dropout=0.0,
                 dropout=0,
                 num_directions=2).to(device)

If you have any idea of what needs to be changed, please let me know.
I added all the code to github repository here: https://github.com/OverclockRo/HTMLGeneration/blob/SamplingTestTemplate/Untitled.ipynb

Output:
First of all for a GRU (RNN) to be efficient, you may need more data to train.

Second, it seems that you have a problem with the embedding. It looks like, the mapping vocabulary['id2letter'] does not work, otherwise you would obtain
sequences of tags like <head><title><title><title>, instead of p111.

EDIT

I have trained this character-level GRU network on the html source code of this page for 1700 epochs. And here an example 2000-character excerpt of what it generates:

A+Implexementation--nope bande that shoos</td></tr><tr><td class="line-number" value="296"></td><td class="line-content">              </td></tr><tr><td class="line-number" value="1437"></td><td class="line-content">                    <span class="html-tag">&lt;/a&gt;</span></td></tr><tr><td class="line-number" value="755"></td><td class="line-content">                </td></tr><tr><td class="line-number" value="584"></td><td class="line-content">    <span class="html-tag">&lt;a <span class="html-attribute-name">data-controller</span>="<span class="html-attribute-value">footer__menu__link</span>"&gt;</span><span class="html-tag">&lt;div <span class="html-attribute-name">data-target</span>="<span class="html-attribute-value">release__line</span>"&gt;</span><span class="html-tag">&lt;a <span class="html-attribute-name">class</span>="<span class="html-attribute-value">/hase__version-date</span>"&gt;</span></td></tr><tr><td class="line-number" value="174"></td><td class="line-content"><br></td></tr><tr><td class="line-number" value="1315"></td><td class="line-content">Bule and the use the twith a hoas suiecode excess ardates</td></tr><tr><td class="line-number" value="1003"></td><td class="line-content"><span class="html-tag">&lt;/a&gt;</span></td></tr><tr><td class="line-number" value="129"></td><td class="line-content">              </td></tr><tr><td class="line-number" value="269"></td><td class="line-content">              <span class="html-tag">&lt;/ul&gt;</span></td></tr><tr><td class="line-number" value="591"></td><td class="line-content">                <span class="html-tag">&lt;/div&gt;</span></td></tr><tr><td class="line-number" value="553"></td><td class="line-content">    <span class="html-tag">&lt;div <span class="html-attribute-name">href</span>="<a class="html-attribute-value html-external-link" target__link</td><td class="line-content">              </td></tr><tr><td class="line-number" value="103"></td><td class="line-content">    </td></tr><tr><td cla

I hope, this helps.


Instruction: not able to predict using pytorch [MNIST]

Input:
pytorch noob here,trying to learn.

link to my notebook:
https://gist.github.com/jagadeesh-kotra/412f371632278a4d9f6cb31a33dfcfeb

I get validation accuracy of 95%.

i use the following to predict:

m.eval()

testset_predictions = []
for batch_id,image in enumerate(test_dataloader):
    image = torch.autograd.Variable(image[0])
    output = m(image)
    _, predictated = torch.max(output.data,1)
    for prediction in predicted:
        testset_predictions.append(prediction.item())

len(testset_predictions)

The problem is i get only 10% accuracy when i submit the result to kaggle competition,which is as good as random prediction.I cant figure out what i'm doing wrong.

please help :)

Output:
Most probably it is due to a typo; while you want to use the newly created predictated outcomes, you actually use predicted:

_, predictated = torch.max(output.data,1)
    for prediction in predicted:

which predicted comes from earlier in your linked code, and it contains predictions from the validation set instead of your test set:

#validation

# ...

for batch_idx, (data, target) in enumerate(val_dataloader):

    data, target = Variable(data), Variable(target)

    output = m.forward(data)

    _, predicted = torch.max(output.data,1)

So, you don't even get an error message, because predicted indeed exists - it's just not what you actually want to use... You end up submitting the results for the validation set instead of the test one (it certainly doesn't help that both consist of 10,000 samples), hence you expectedly get a random guessing accuracy of ~ 10%...


Instruction: int8 data type in Pytorch

Input:
What is the best way to run a qauntized model using int8 data types in Pytorch? I know in pytorch I can define tensors as int8, however, when I actually want to use int8, I get:

RuntimeError: _thnn_conv2d_forward is not implemented for type torch.CharTensor

So I am confused, how to run quantized model in pytorch that uses for instance int8 when the datatype is not supported for computational blocks such as convolutions? I am using pytorch version 1.0.1.post2.

Output:
Depends what your goals are.

If you want to simulate your quantized model:

You may stick to existing float data type and only introduce truncation as needed, i.e.:

 x = torch.floor(x * 2**8) / 2**8

assuming x is a float tensor.

If you want to simulate your quantized model efficiently:

Then, I am afraid PyTorch will be not very useful since the low-level convolutional operator is implemented only for float type.


Instruction: Loss over pixels

Input:
During backpropagation, will these cases have different effect:-

sum up loss over all pixels then backpropagate.
average loss over all pixels then backpropagate
backpropagate individuallyover all pixels.

My main doubts in regarding the numerical value but the effect all these would be having.

Output:
The difference between no 1 and 2 is basically : since sum will result in bigger than mean, the magnitude of gradients from sum operation will be bigger, but direction will be same.

Here's a little demonstration, lets first declare necessary variables:

x = torch.tensor([4,1,3,7],dtype=torch.float32,requires_grad=True)
target = torch.tensor([4,2,5,4],dtype=torch.float32)

Now lets compute gradient for x using L2 loss with sum:

loss = ((x-target)**2).sum()
loss.backward()
print(x.grad)

This outputs: tensor([ 0., -2., -4.,  6.])

Now using mean: (after resetting x grad)

loss = ((x-target)**2).mean()
loss.backward()
print(x.grad)

And this outputs: tensor([ 0.0000, -0.5000, -1.0000,  1.5000])
Notice how later gradients are exactly 1/4th of that of sum, that's because the tensors here contain 4 elements.

About third option, if I understand you correctly, that's not possible. You can not backpropagate before aggregating individual pixel errors to a scalar, using sum, mean or anything else.


Instruction: Pytorch nn embeddings dimension size?

Input:
What is the correct dimension size for nn embeddings in Pytorch? I'm doing batch training.

I'm just a little confused with what the dimensions of "self.embeddings" in the code below are supposed to be when I get "shape"?

self.embeddings = nn.Embedding(vocab_size, embedding_dim)

Output:
The shape of the self.embedding will be [sentence_length, batch_size, embedding_dim]

where sentence_length is the length of inputs in each batch.


Instruction: pip install horovod fails on conda + OSX 10.14

Input:
Running pip install horovod in a conda environment with pytorch installed resulted in

error: None of TensorFlow, PyTorch, or MXNet plugins were built. See errors above.

where the root problem near the top of stdout is

ld: library not found for -lstdc++
  clang: error: linker command failed with exit code 1 (use -v to see invocation)
  INFO: Unable to build PyTorch plugin, will skip it.

Output:
CFLAGS=-mmacosx-version-min=10.9 pip install horovod, inspired from this seemingly unrelated Horovod issue.
This issue thread from pandas has a nice explanation:

The compiler standard library defaults to either libstdc++ or libc++, depending on the targetted macOS version - libstdc++ for 10.8 and below, and libc++ for 10.9 and above. This is determined by the environment variable MACOSX_DEPLOYMENT_TARGET or the compiler option-mmacosx-version-min, defaulting to the system version otherwise.
When distuils builds extensions on macOS, it setsMACOSX_DEPLOYMENT_TARGET to the version that python was compiled with, even if the host system / Xcode is newer.
Recent macOS versions of python have a 64-bit only variant built for 10.9 (python.org), and a universal 64/32-bit variant built for 10.6 (python.org) or 10.7 (conda). I am running the conda universal variant, so distutils targets macOS 10.7, despite my system being 10.14, with Xcode 10 which doesn't install libstdc++.


Instruction: Creating a torch tensor from a generator

Input:
I attempt to construct a tensor from a generator as follows:

>>> torch.tensor(i**2 for i in range(10))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: Could not infer dtype of generator

Currently I just do:

>>> torch.tensor([i**2 for i in range(10)])
tensor([ 0,  1,  4,  9, 16, 25, 36, 49, 64, 81])

Is there a way to avoid needing this intermediate list?

Output:
As @blue-phoenox already points out, it is preferred to use the built-in PyTorch functions to create the tensor directly. But if you have to deal with generator, it can be advisable to use numpy as a intermediate stage. Since PyTorch avoid to copy the numpy array, it should be quite performat (compared to the simple list comprehension)

>>> import torch
>>> import numpy as np
>>> torch.from_numpy(np.fromiter((i**2 for i in range(10)), int))
tensor([ 0,  1,  4,  9, 16, 25, 36, 49, 64, 81])


Instruction: Loss not decreasing - Pytorch

Input:
I am using dice loss for my implementation of a Fully Convolutional Network(FCN) which involves hypernetworks. The model has two inputs and one output which is a binary segmentation map. The model is updating weights but loss is constant.
It is not even overfitting on only three training examples

I have used other loss functions as well like dice+binarycrossentropy loss, jacard loss and MSE loss but the loss is almost constant.
I have also tried almost every activation function like ReLU, LeakyReLU, Tanh. Moreover I have to use sigmoid at the the output because I need my outputs to be in range [0,1]
Learning rate is 0.01. Moreover, I have tried different learning rates as well like 0.0001, 0.001, 0.1. And no matter what loss the training starts at, it always comes at this value

This shows gradients for three training examples. And overall loss

tensor(0.0010, device='cuda:0')
tensor(0.1377, device='cuda:0')
tensor(0.1582, device='cuda:0')
Epoch 9, Overall loss = 0.9604763123724196, mIOU=0.019766070265581623
tensor(0.0014, device='cuda:0')
tensor(0.0898, device='cuda:0')
tensor(0.0455, device='cuda:0')
Epoch 10, Overall loss = 0.9616242945194244, mIOU=0.01919178702228237
tensor(0.0886, device='cuda:0')
tensor(0.2561, device='cuda:0')
tensor(0.0108, device='cuda:0')
Epoch 11, Overall loss = 0.960331304506822, mIOU=0.01983801422510155

I expect the loss to converge in few epochs.
What should I do?

Output:
@Muhammad Hamza Mughal

You got to add code of at least your forward and train functions for us to pinpoint the issue, @Jatentaki is right there could be so many things that could mess up a ML / DL code. Even I moved recently to pytorch from Keras, took some time to get used to it. But, here are the things I'd do:

1) As you're dealing with images, try to pre-process them a bit ( rotation, normalization, Gaussian Noise etc).

2) Zero gradients of your optimizer at the beginning of each batch you fetch and also step optimizer after you calculated loss and called loss.backward().

3) Add a weight decay term to your optimizer call, typically L2, as you're dealing with Convolution networks have a decay term of 5e-4 or 5e-5.

4) Add a learning rate scheduler to your optimizer, to change learning rates if there's no improvement over time.

We really can't include code in our answers. It's up to the practitioner to scout for how to implement all this stuff. Hope this helps.


Instruction: PyTorch 1.0 loading VGGFace2 weights in Python3.7

Input:
I am using Python3.7 and PyTorch 1.0 to develop a face recognition system. I want to use VGGFace2 Resnet50 pretrained model as described here as a feature extractor. I have downloaded the model and weights.
I run the following codes as project readme says:

MainModel = imp.load_source('MainModel', 'resnet50_128_pytorch.py')
model = torch.load('resnet50_128_pytorch.pth')

First line executed as expected but in the second line i got

  'ascii' codec can't decode byte 0xc3 in position 1124: ordinal not in
  range(128)

I searched in the Stackoverflow and Google and i saw that it may be about this model saved with Python2 and loading from Python3 creates a problem. Is there any way that i can solve this?

Thank you.

Output:
I found a solution which currently looks like it's working. It basically changes the pickle load with latin1 encoding.

from functools import partial
import pickle
pickle.load = partial(pickle.load, encoding="latin1")
pickle.Unpickler = partial(pickle.Unpickler, encoding="latin1")
MainModel = imp.load_source('MainModel', 'resnet50_ft_pytorch.py')
model = torch.load('resnet50_ft_pytorch.pth', map_location=lambda storage, loc: storage, pickle_module=pickle)


Instruction: BatchNorm1d needs 2d input?

Input:
I want to fix problem in PyTorch.
I wrote the following code that is learning sine functions as tutorial.

import torch
from torch import nn
from torch import optim
from torch.autograd import Variable as V
from torch.utils.data import TensorDataset, DataLoader
import numpy as np

# y=sin(x1)
numTrain = 512
numTest = 128
noiseScale = 0.01
PI2 = 3.1415 * 2
X_train = np.random.rand(numTrain,1) * PI2
y_train = np.sin(X_train) + np.random.randn(numTrain,1) * noiseScale + 1.5
X_test  = np.random.rand(numTest,1) * PI2
y_test  = np.sin(X_test) + np.random.randn(numTest,1) * noiseScale

# Construct DataSet
X_trainT = torch.Tensor(X_train)
y_trainT = torch.Tensor(y_train)
X_testT = torch.Tensor(X_test)
y_testT = torch.Tensor(y_test)
ds_train = TensorDataset(X_trainT, y_trainT)
ds_test = TensorDataset(X_testT, y_testT)

# Construct DataLoader
loader_train = DataLoader(ds_train, batch_size=64, shuffle=True)
loader_test = DataLoader(ds_test, batch_size=64, shuffle=False)

# Construct network
net = nn.Sequential(
    nn.Linear(1,10),
    nn.ReLU(),
    nn.BatchNorm1d(10),
    nn.Linear(10,5),
    nn.ReLU(),
    nn.BatchNorm1d(5),
    nn.Linear(5,1),
)
optimizer = optim.Adam(net.parameters())
loss_fn = nn.SmoothL1Loss()

# Training
losses = []
net.train()
for epoc in range(100):
    for data, target in loader_train:
        y_pred = net(data)
        loss = loss_fn(target,y_pred)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        losses.append(loss.data)

# evaluation
%matplotlib inline
from matplotlib import pyplot as plt

#plt.plot(losses)
plt.scatter(X_train, y_train)

net.eval()
sinsX = []
sinsY = []
for t in range(128):
    x = t/128 * PI2
    output = net(V(torch.Tensor([x])))
    sinsX.append(x)
    sinsY.append(output.detach().numpy())
plt.scatter(sinsX,sinsY)

Training is done without error, But the next line caused an error, "expected 2D or 3D input (got 1D input)"

output = net(V(torch.Tensor([x])))

This error doesn't occur if it is without BatchNorm1d().
I feel strange because the input is 1D.

How to fix it?

Thanks.

Update: How did I fix

arr = np.array([x])
output = net(V(torch.Tensor(arr[None,...])))

Output:
When working with 1D signals, pyTorch actually expects a 2D tensors: the first dimension is the "mini-batch" dimension. Therefore, you should evaluate your net on a batch with one 1D signal:

output - net(V(torch.Tensor([x[None, ...]]))

Make sure you set your net to "eval" mode before evaluating it:

net.eval()


Instruction: How to clear CUDA memory in PyTorch

Input:
I am trying to get the output of a neural network which I have already trained. The input is an image of the size 300x300. I am using a batch size of 1, but I still get a CUDA error: out of memory error after I have successfully got the output for 25 images.
I tried torch.cuda.empty_cache(), but this still doesn't seem to solve the problem. Code:
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

train_x = torch.tensor(train_x, dtype=torch.float32).view(-1, 1, 300, 300)
train_x = train_x.to(device)
dataloader = torch.utils.data.DataLoader(train_x, batch_size=1, shuffle=False)

right = []
for i, left in enumerate(dataloader):
    print(i)
    temp = model(left).view(-1, 1, 300, 300)
    right.append(temp.to('cpu'))
    del temp
    torch.cuda.empty_cache()

This for loop runs for 25 times every time before giving the memory error.
Every time, I am sending a new image in the network for computation. So, I don't really need to store the previous computation results in the GPU after every iteration in the loop. Is there any way to achieve this?

Output:
I figured out where I was going wrong. I am posting the solution as an answer for others who might be struggling with the same problem.

Basically, what PyTorch does is that it creates a computational graph whenever I pass the data through my network and stores the computations on the GPU memory, in case I want to calculate the gradient during backpropagation. But since I only wanted to perform a forward propagation, I simply needed to specify torch.no_grad() for my model.

Thus, the for loop in my code could be rewritten as:

for i, left in enumerate(dataloader):
    print(i)
    with torch.no_grad():
        temp = model(left).view(-1, 1, 300, 300)
    right.append(temp.to('cpu'))
    del temp
    torch.cuda.empty_cache()

Specifying no_grad() to my model tells PyTorch that I don't want to store any previous computations, thus freeing my GPU space.


Instruction: Unity ML-Agents Running Very Slowly

Input: Using the Python API, whether I run using a build or in the Editor the simulation is much slower than when using the provided mlagents-learn methods. I'm running something similar to this, using a PyTorch implementation of DDPG and CUDA 9.0. Is this expected behaviour?

Output:
Got to the Academy's proprieties and make sure that the value of Time Scale in the Inference Configuration is equals to the Training Configuration's one.

For other info check the official documentation of ML-Agents at Learning-Environment-Design-Academy


Instruction: Visualizing the output of intermediate layers of cnn in pytorch

Input:
I'm trying to visualize the output of the intermediate layers of the VGG19 network, from the torchvision module, specifically the layer, conv4_2.
I've extracted the output in a tensor of shape [1, 512, 50, 50].
But how do I visualize an image with 512 channels?

Output:
Feature visualization is a very complex subject.
If you want to have a visual idea what each filter (of the 512) of the trained net is responding to, you can use methods like these: propagating gradients from conv4_2's output to the input image, and change the image to maximize the feature response. You will have to work your way through regularization etc. to get smooth visually pleasing results.

Alternatively, you can see the specific responses of each filter (out of the 512) to each location (overlapping receptive fields). In that case you have 512 different 50-by-50 intensity images, each showing the response map of each neuron to the input image.


Instruction: Why there are different output between model.forward(input) and model(input)

Input:
I'm using pytorch to build a simple model like VGG16,and I have overloaded the function forward in my model.

I found everyone tends to use model(input) to get the output rather than model.forward(input), and I am interested in the difference between them. I try to input the same data, but the result is different. I'm confused.

I have output the layer_weight before I input data, the weight not be changed, and I know when we using model(input) it using __call__ function, and this function will call model.forward.

   vgg = VGG()
   vgg.double()
   for layer in vgg.modules():
      if isinstance(layer,torch.nn.Linear):
         print(layer.weight)
   print("   use model.forward(input)     ")
   result = vgg.forward(array)

   for layer in vgg.modules():
     if isinstance(layer,torch.nn.Linear):
       print(layer.weight)
   print("   use model(input)     ")
   result_2 = vgg(array)
   print(result)
   print(result_2)

output:

    Variable containing:1.00000e-02 *
    -0.2931  0.6716 -0.3497 -2.0217 -0.0764  1.2162  1.4983 -1.2881
    [torch.DoubleTensor of size 1x8]

    Variable containing:
    1.00000e-02 *
    0.5302  0.4494 -0.6866 -2.1657 -0.9504  1.0211  0.8308 -1.1665
    [torch.DoubleTensor of size 1x8]

Output:
model.forward just calls the forward operations as you mention but __call__ does a little extra.

If you dig into the code of nn.Module class you will see __call__ ultimately calls forward but internally handles the forward or backward hooks and manages some states that pytorch allows. When calling a simple model like just an MLP, it may not be really needed but more complex model like spectral normalization layers have hooks and therefore you should use model(.) signature as much as possible unless you explicitly just want to call model.forward

Also see Calling forward function without .forward()

In this case, however, the difference may be due to some dropout layer, you should call vgg.eval() to make sure all the stochasticity in network is turned off before comparing the outputs.


Instruction: How to do parallel processing in pytorch

Input:
I am working on a deep learning problem. I am solving it using pytorch. I have two GPU's which are on the same machine (16273MiB,12193MiB). I want to use both the GPU's for my training (video dataset).

I get a warning:

There is an imbalance between your GPUs. You may want to exclude GPU 1 which
    has less than 75% of the memory or cores of GPU 0. You can do so by setting
    the device_ids argument to DataParallel, or by setting the CUDA_VISIBLE_DEVICES
    environment variable.
  warnings.warn(imbalance_warn.format(device_ids[min_pos], device_ids[max_pos]))

I also get an error:

raise TypeError('Broadcast function not implemented for CPU tensors')
TypeError: Broadcast function not implemented for CPU tensors

if __name__ == '__main__':

    opt.scales = [opt.initial_scale]
    for i in range(1, opt.n_scales):
        opt.scales.append(opt.scales[-1] * opt.scale_step)
    opt.arch = '{}-{}'.format(opt.model, opt.model_depth)
    opt.mean = get_mean(opt.norm_value)
    opt.std = get_std(opt.norm_value)
    print("opt",opt)
    with open(os.path.join(opt.result_path, 'opts.json'), 'w') as opt_file:
        json.dump(vars(opt), opt_file)

    torch.manual_seed(opt.manual_seed)

    model, parameters = generate_model(opt)
    #print(model)

    pytorch_total_params = sum(p.numel() for p in model.parameters() if p.requires_grad)
    print("Total number of trainable parameters: ", pytorch_total_params)

    # Define Class weights
    if opt.weighted:
        print("Weighted Loss is created")
        if opt.n_finetune_classes == 2:
            weight = torch.tensor([1.0, 3.0])
        else:
            weight = torch.ones(opt.n_finetune_classes)
    else:
        weight = None

    criterion = nn.CrossEntropyLoss()
    if not opt.no_cuda:

        criterion = nn.DataParallel(criterion.cuda())

    if opt.no_mean_norm and not opt.std_norm:
        norm_method = Normalize([0, 0, 0], [1, 1, 1])
    elif not opt.std_norm:
        norm_method = Normalize(opt.mean, [1, 1, 1])
    else:
        norm_method = Normalize(opt.mean, opt.std)

        train_loader = torch.utils.data.DataLoader(
            training_data,
            batch_size=opt.batch_size,
            shuffle=True,
            num_workers=opt.n_threads,
            pin_memory=True)
        train_logger = Logger(
            os.path.join(opt.result_path, 'train.log'),
            ['epoch', 'loss', 'acc', 'precision','recall','lr'])
        train_batch_logger = Logger(
            os.path.join(opt.result_path, 'train_batch.log'),
            ['epoch', 'batch', 'iter', 'loss', 'acc', 'precision', 'recall', 'lr'])

        if opt.nesterov:
            dampening = 0
        else:
            dampening = opt.dampening
        optimizer = optim.SGD(
            parameters,
            lr=opt.learning_rate,
            momentum=opt.momentum,
            dampening=dampening,
            weight_decay=opt.weight_decay,
            nesterov=opt.nesterov)
        # scheduler = lr_scheduler.ReduceLROnPlateau(
        #     optimizer, 'min', patience=opt.lr_patience)
    if not opt.no_val:
        spatial_transform = Compose([
            Scale(opt.sample_size),
            CenterCrop(opt.sample_size),
            ToTensor(opt.norm_value), norm_method
        ])

    print('run')
    for i in range(opt.begin_epoch, opt.n_epochs + 1):
        if not opt.no_train:
            adjust_learning_rate(optimizer, i, opt.lr_steps)
            train_epoch(i, train_loader, model, criterion, optimizer, opt,
                        train_logger, train_batch_logger)

I have also made changes in my train file:

      model = nn.DataParallel(model(),device_ids=[0,1]).cuda()
        outputs = model(inputs)

It does not seem to work properly and is giving error. Please advice, I am new to pytorch.

Thanks

Output:
As mentioned in this link, you have to do model.cuda() before passing it to nn.DataParallel.

net = nn.DataParallel(model.cuda(), device_ids=[0,1])

https://github.com/pytorch/pytorch/issues/17065


Instruction: How to make a dataset from video datasets(tensorflow first)

Input:
everyone.

Now I have an object classification task, and I have a dataset containing a large number of videos. In every video, some frames(not every frame, about 160 thousand frames) have its labels, since a frame may have multiple objects.

I have some confusion about creating the dataset. My idea is to convert videos to frames firstly, then every frame only with labels will be made as tfrecord or hdf5 format. Finally, I would write every frame's path into csv files (training and validation) using for my task.

My question is :
1. Is there efficient enough(tfrecord or hdf5)? Should I preprocess every frame such as compression for save the storage space before creating tfrecord or hdf5 files?
2. Is there a way to handle the video dataset directly in tensorflow or pytorch?

I want to find an efficient and conventional way to handle video datasets. Really looking forward to every answer.

Output:
I am no TensorFlow guy, so my answer won't cover that, sorry.

Video formats generally gain compression at the cost of longer random-access times thanks to exploiting temporal correlations in the data. It makes sense because one usually accesses video frames sequentially, but if your access is entirely random I suggest you convert to hdf5. Otherwise, if you access sub-sequences of video, it may make sense to stay with video formats.

PyTorch does not have any "blessed" approaches to video AFAIK, but I use imageio to read videos and seek particular frames. A short wrapper makes it follow the PyTorch Dataset API. The code is rather simple but has a caveat, which is necessary to allow using it with multiprocessing DataLoader.

import imageio, torch

class VideoDataset:
    def __init__(self, path):
        self.path = path

        # explained in __getitem__
        self._reader = None

        reader = imageio.get_reader(self.path, 'ffmpeg')
        self._length = reader.get_length()

    def __getitem__(self, ix):
        # Below is a workaround to allow using `VideoDataset` with
        # `torch.utils.data.DataLoader` in multiprocessing mode.
        # `DataLoader` sends copies of the `VideoDataset` object across
        # processes, which sometimes leads to bugs, as `imageio.Reader`
        # does not support being serialized. Since our `__init__` set
        # `self._reader` to None, it is safe to serialize a
        # freshly-initialized `VideoDataset` and then, thanks to the if
        # below, `self._reader` gets initialized independently in each
        # worker thread.

        if self._reader is None:
            self._reader = imageio.get_reader(self.path, 'ffmpeg')

        # this is a numpy ndarray in [h, w, channel] format
        frame = self._reader.get_data(ix)

        # PyTorch standard layout [channel, h, w]
        return torch.from_numpy(frame.transpose(2, 0, 1))

     def __len__(self):
        return self.length

This code can be adapted to support multiple video files as well as to output the labels as you would like to have them.


Instruction: Pooling over channels in pytorch

Input:
In tensorflow, I can pool over the depth dimension which would reduce the channels and leave the spatial dimensions unchanged. I'm trying to do the same in pytorch but the documentation seems to say pooling can only be done over the height and width dimensions. Is there a way I can pool over channels in pytorch?
I've a tensor of shape [1,512,50,50] I'm trying to use pooling to bring the number of channels down to 3.
I saw this question but did not find the answer helpful.

Output:
The easiest way to reduce the number of channels is using a 1x1 kernel:

import torch

x = torch.rand(1, 512, 50, 50)
conv = torch.nn.Conv2d(512, 3, 1)
y = conv(x)

print(y.size())
# torch.Size([1, 3, 50, 50])

If you really need to perform pooling along the channels dimension due to some reason, you may want to permute the dimensions so that the channels dimension is swapped with some other dimension (e.g. width).
This idea was referenced here.


Instruction: Character level CNN - 1D or 2D

Input:
I want to implement a character-level CNN in Pytorch.

My input has 4 dimensions:

  (batch_size, seq_length, padded_character_length, embedding_dim)

I'm wondering whether I should merge two dimensions and use a Conv1D-layer or instead use a Conv2D-layer on the existing dimensions.

Given the dimensions of the input both would technically work fine, I also have seen implementations for both versions. So I'm wondering which method to prefer.

Does one of the two methods have particular advantages over the other?

Output:
I agree with Venkatesh that 1D might make more sense for your implementation. Instead of merging, I typically use the TimeDistributed layers that are found in Keras. This takes one layer and applies it across time dimensions. The advantage is that you keep the features from each dimension separate until you want to merge them.

If you are using padding (as you mentioned) then it makes more sense to apply the same layer across the time dimensions instead of merging the layers and creating awkward padding space in between actual characters. TimeDistributed layers solves this.

I googled for just a moment and found someone who has tried implementing this feature in PyTorch, that might at least get you started.

PyTorch TimeDistributed

To expand on my usage a bit.. My latest model has 5, 12 hour periods, each 12 hour period has sporadic activity so it is padded to reach a standard length of 30, so the final input shape is (?, 5, 30, embedding_size). I generate features within a single period using a TimeDistributed 1D CNN, then I max pool those features and concatenate to create a new shape of (?, 5, n_feats) where there are now 5 sets of feature maps. I again run that data over with a different 1D CNN layer which looks across the 5, 12 hour periods. The padding in each period is independent of each other, so I can't simply use a 2D CNN as elements at the same index wouldn't represent the same time across periods.

Edit:

I think the Keras implementation is a little more sophisticated but that should be close. Their documentation says "This wrapper applies a layer to every temporal slice of an input." If getting there requires to merge then restore after then their may be some considerations around the feature map. For example, if the filter size is 2 then the last item in the first feature map (after reshaping) will include the last feature and the first feature between two time slices.

Here is one more link on discussion around this capability in PyTorch that might be helpful.


Instruction: installed pytorch1.0.1 for OS X with pip3 but cannot import, what can I do?

Input:
I have already installed pytorch for MacOS 10.14 with pip3, but I can not import it in the python script. What should I do?

System: MacOS 10.14

Python3: v3.7

➜  ~ pip3 list
Package     Version
----------- -----------
numpy       1.16.2
Pillow      5.4.1
pip         18.1
pycairo     1.17.1
pygobject   3.28.3
setuptools  40.5.0
six         1.12.0
torch       1.0.1.post2
torchvision 0.2.2.post3
virtualenv  16.1.0
wheel       0.32.2
➜  ~ python3
Python 3.7.0 (v3.7.0:1bf9cc5093, Jun 26 2018, 23:26:24)
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'torch'
>>>

Output:
To expand upon my comment:

There's no strict guarantee that a pip3 wrapper script somewhere on your system is related to the pip package manager/module for your python3 binary. That wrapper may be created by a different installation of Python – maybe your system's own, maybe something else. (You can see where the script is located with which pip3 and see which interpreter it uses with less $(which pip3) and looking at the shebang line at the top.)

Each version of Python you have installed has its own site-packages directory, which contains the globally (as far as that version is concerned) installed packages. Fortunately, pip can be run exactly equally as the wrapper script would with the -m switch, so to be sure Torch and Torchvision get installed into your python3 (which appears to be Python 3.7.0 at this time),

python3 -m pip install torch torchvision

should do the trick.

However, globally (well, interpreter-globally, as discussed above) installed packages should be avoided, since you can easily get into hairy conflicts when you're working on multiple projects. You should instead use virtualenvs to separate your library installations from each other – the venv module is included in Python these days, and the official documentation has a guide on it. (Other options are pipenv and poetry, but it's worth knowing the lower-level tooling.)


Instruction: Why the pytorch implementation is so inefficient?

Input:
I have implemented a paper about a CNN architecture in both Keras and Pytorch but keras implementation is much more efficient it takes 4 gb of gpu for training with 50000 samples and 10000 validation samples but pytorch one takes all the 12 gb of gpu and i cant even use a validation set !
Optimizer for both of them is sgd with momentum and same settings for both.
more info about the paper:[architecture]:https://github.com/Moeinh77/Lightweight-Deep-Convolutional-Network-for-Tiny-Object-Recognition/edit/master/train.py

pytorch code :

class SimpleCNN(torch.nn.Module):

    def __init__(self):
        super(SimpleCNN, self).__init__()

        self.conv2d_11 = torch.nn.Conv2d(3, 64, kernel_size = 3, stride = 1, padding = 1)
        self.conv2d_12 = torch.nn.Conv2d(64, 64, kernel_size = 3, stride = 1, padding = 1)

        self.conv2d_21 = torch.nn.Conv2d(64, 128, kernel_size = 3, stride = 1, padding = 1)
        self.conv2d_22 = torch.nn.Conv2d(128, 128, kernel_size = 3, stride = 1, padding = 1)

        self.conv2d_31 = torch.nn.Conv2d(128, 256, kernel_size = 3, stride = 1, padding = 1)
        self.conv2d_32 = torch.nn.Conv2d(256, 256, kernel_size = 3, stride = 1, padding = 1)
        self.conv2d_33 = torch.nn.Conv2d(256, 256, kernel_size = 3, stride = 1, padding = 1)

        self.conv2d_41 = torch.nn.Conv2d(256, 512, kernel_size = 3, stride = 1, padding = 1)
        self.conv2d_42 = torch.nn.Conv2d(512, 512, kernel_size = 3, stride = 1, padding = 1)

        self.conv2d_51 = torch.nn.Conv2d(512, 512, kernel_size = 3, stride = 1, padding = 1)

        self.Batchnorm_1=torch.nn.BatchNorm2d(64)
        self.Batchnorm_2=torch.nn.BatchNorm2d(128)
        self.Batchnorm_3=torch.nn.BatchNorm2d(256)
        self.Batchnorm_4=torch.nn.BatchNorm2d(512)

        self.dropout2d_1=torch.nn.Dropout2d(p=0.3)
        self.dropout2d_2=torch.nn.Dropout2d(p=0.4)
        self.dropout2d_3=torch.nn.Dropout2d(p=0.5)

        self.dropout1d=torch.nn.Dropout(p=0.5)

        self.maxpool2d = torch.nn.MaxPool2d(kernel_size = 2, stride = 2, padding = 0)

        self.avgpool2d = torch.nn.AvgPool2d(kernel_size = 2, stride = 2, padding = 0)

        self.fc = torch.nn.Linear(512, 10)

    def forward(self, x):

        ############################# Phase 1
        #print(x.size())
        x = F.relu(self.conv2d_11(x))
        x = self.dropout2d_1(x) #rate =0.3
        x = self.Batchnorm_1(x) #input 64
        #print(x.size())

        x = F.relu(self.conv2d_12(x))
        x = self.dropout2d_1(x) #rate=0.3
        x = self.Batchnorm_1(x) #input 64
        #print(x.size())

        x = self.maxpool2d(x)
        #print(x.size())
        ############################# Phase 2
        x = F.relu(self.conv2d_21(x))
        x = self.dropout2d_1(x) #rate=0.3
        x = self.Batchnorm_2(x) #input 128
        #print(x.size())

        x = F.relu(self.conv2d_22(x))
        x = self.dropout2d_1(x) #rate=0.3
        x = self.Batchnorm_2(x) #input 128
        #print(x.size())

        x = self.maxpool2d(x)
        #print(x.size())
        ############################# Phase 3
        x = F.relu(self.conv2d_31(x))
        x = self.dropout2d_2(x) #rate=0.4
        x = self.Batchnorm_3(x) #input 256
        #print(x.size())

        x = F.relu(self.conv2d_32(x))
        x = self.dropout2d_2(x) #rate=0.4
        x = self.Batchnorm_3(x) #input 256
        #print(x.size())

        x = F.relu(self.conv2d_33(x))
        x = self.dropout2d_2(x) #rate=0.4
        x = self.Batchnorm_3(x) #input 256
        #print(x.size())

        x = self.maxpool2d(x)
        #print(x.size())
        ############################# Phase 4
        x = F.relu(self.conv2d_41(x))
        x = self.dropout2d_2(x)
        x = self.Batchnorm_4(x)
        #print(x.size())

        x = F.relu(self.conv2d_42(x))
        x = self.dropout2d_2(x)
        x = self.Batchnorm_4(x)
        #print(x.size())

        x = self.maxpool2d(x)
        #print(x.size())
        ############################# Phase 5
        x = F.relu(self.conv2d_51(x))
        x = self.dropout2d_3(x)
        x = self.Batchnorm_4(x)
        #print(x.size())

        x = self.avgpool2d(x)
        #print(x.size())
        x = x.view(x.size(0), -1)
        #print(x.size())
        x = self.dropout1d(x)
        x = F.relu(self.fc(x))
        x = self.dropout1d(x)
        #print(x.size())
        x = F.softmax(x)
        ###############################

        return(x)

import time
from torch.optim.lr_scheduler import ReduceLROnPlateau

def trainNet(model, batch_size, n_epochs, learning_rate):

    lr=learning_rate

    #Print all of the hyperparameters of the training iteration:
    print("======= HYPERPARAMETERS =======")
    print("Batch size=", batch_size)
    print("Epochs=", n_epochs)
    print("Base learning_rate=", learning_rate)
    print("=" * 30)

    #Get training data
    n_batches = len(train_loader)

    #Time for printing
    training_start_time = time.time()

    #Loss function"
    loss = torch.nn.CrossEntropyLoss()
    optimizer = createOptimizer(model, lr)

    scheduler = ReduceLROnPlateau(optimizer, 'min'
                                  ,patience=3,factor=0.9817
                                 ,verbose=True,)

    #Loop for n_epochs
    for epoch in range(n_epochs):

        #save the weightsevery 10 epochs
        if epoch % 10 == 0 :
            torch.save(model.state_dict(), 'model.ckpt')

        #print('learning rate : {:.3f} '.format(lr))
        #Create our loss and optimizer functions

        running_loss = 0.0
        print_every = n_batches // 10
        start_time = time.time()
        total_train_loss = 0
        total_train_acc = 0
        epoch_time = 0

        for i, data in enumerate(train_loader, 0):

            #free up the cuda memory
            inputs=None
            labels=None

            inputs, labels = data

            inputs, labels = Variable(inputs.to(device)), Variable(labels.to(device))

            optimizer.zero_grad()

            outputs = model(inputs)

            score, predictions = torch.max(outputs.data, 1)
            acc = (labels==predictions).sum()
            total_train_acc += acc

            loss_size = loss(outputs, labels)
            loss_size.backward()
            optimizer.step()

            running_loss += loss_size.item()
            total_train_loss += loss_size.item()

            #Print every 10th batch of an epoch
            if (i + 1) % (print_every + 1) == 0:
                print("Epoch {}, {:d} % \t | train_loss: {:.3f} | train_acc:{}% | took: {:.2f}s".format(
                        epoch+1, int(100 * (i+1) / n_batches), running_loss / print_every
                        ,int(acc), time.time() - start_time))

                epoch_time += (time.time() - start_time)

                #Reset running loss and time
                running_loss = 0.0
                start_time = time.time()

        scheduler.step(total_train_loss)
        torch.cuda.empty_cache()
        #At the end of the epoch, do a pass on the validation set
        total_val_loss = 0

        for inputs, labels in val_loader:

            #Wrap tensors in Variables
            inputs, labels = Variable(inputs.to(device)), Variable(labels.to(device))

            #Forward pass
            val_outputs = model(inputs)
            val_loss_size = loss(val_outputs, labels)
            total_val_loss += val_loss_size.item()

        print("-"*30)
        print("Train loss = {:.2f} | Train acc = {:.1f}% | Val loss={:.2f} | took: {:.2f}s".format(
            total_train_loss / len(train_loader),total_train_acc/ len(train_loader)
            ,total_val_loss/len(val_loader),epoch_time))
        print("="*60)

    print("Training finished, took {:.2f}s".format(time.time() - training_start_time))
CNN = SimpleCNN().to(device)
CNN.eval()

trainNet(CNN, batch_size=64, n_epochs=250, learning_rate=0.1)

Keras:

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout, Flatten,Activation
from tensorflow.keras.layers import Conv2D, MaxPool2D,BatchNormalization,GlobalAveragePooling2D

model = Sequential()
#####################################################
# Phase 1
model.add(Conv2D(64,(3,3),input_shape=(32,32,3),padding='same'))
model.add(Activation('relu'))
model.add(Dropout(rate=0.3))
model.add(BatchNormalization())

#(32,32,3)

model.add(Conv2D(64,(3,3),padding='same'))
model.add(Activation('relu'))
model.add(Dropout(rate=0.3))
model.add(BatchNormalization())
#(32,32,3)

model.add(MaxPool2D((2,2)))
#(16,16,3)

#####################################################
#Phase 2
model.add(Conv2D(128, (3,3),padding='same'))
model.add(Activation('relu'))
model.add(Dropout(rate=0.3))
model.add(BatchNormalization())
#(16,16,3)

model.add(Conv2D(128, (3,3),padding='same'))
model.add(Activation('relu'))
model.add(Dropout(rate=0.3))
model.add(BatchNormalization())
#(16,16,3)

model.add(MaxPool2D((2,2),padding='same'))
#(8,8,3)

#####################################################
#Phase 3
model.add(Conv2D(256, (3,3),padding='same'))
model.add(Activation('relu'))
model.add(Dropout(rate=0.4))
model.add(BatchNormalization())
#(8,8,3)

model.add(Conv2D(256, (3,3),padding='same'))
model.add(Activation('relu'))
model.add(Dropout(rate=0.4))
model.add(BatchNormalization())
#(8,8,3)

model.add(Conv2D(256, (3,3),padding='same'))
model.add(Activation('relu'))
model.add(Dropout(rate=0.4))
model.add(BatchNormalization())
#(8,8,3)

model.add(MaxPool2D((2,2)))
#(4,4,3)

#####################################################
#Phase 4
model.add(Conv2D(512, (3,3),padding='same'))
model.add(Activation('relu'))
model.add(Dropout(rate=0.4))
model.add(BatchNormalization())
#(4,4,3)

model.add(Conv2D(512, (3,3),padding='same'))
model.add(Activation('relu'))
model.add(Dropout(rate=0.4))
model.add(BatchNormalization())
#(4,4,3)

model.add(MaxPool2D((2,2)))
#(2,2,3)

#####################################################
#Phase 5
model.add(Conv2D(512, (3,3),padding='same'))
model.add(Activation('relu'))
model.add(Dropout(rate=0.5))
model.add(BatchNormalization())
#(2,2,3)

model.add(GlobalAveragePooling2D(data_format='channels_last'))
model.add(Flatten())
model.add(Dropout(rate=0.5))

model.add(Dense(10,activation='relu'))
model.add(Dropout(rate=0.5))
model.add(Dense(10, activation='softmax'))
model.compile(optimizer=sgd_optimizer,loss='categorical_crossentropy',metrics=['accuracy'])
history=model.fit(x=x_train,y=y_train,batch_size=64,
                  epochs=250,verbose=1,callbacks=[checkpoint],validation_data=(x_test,y_test))

Output:
Edit: on a closer look, acc doesn't seem to require gradient, so this paragraph probably doesn't apply.
It looks like the most significant issue is that total_train_acc accumulates history across the training loop (see https://pytorch.org/docs/stable/notes/faq.html for details).
Changing total_train_acc += acc to total_train_acc += acc.item() should fix this.

Another thing you should use with torch.no_grad() for the validation loop.

Not really about speed, but model.train() and model.eval() should be used for training/evaluation to make batchnorm and dropout layers work in correct mode.


Instruction: How to save grayscale image in Pytorch?

Input:
I want to save grayscale image in Pytorch, each image has four gray values, 0, 60, 120 and 180. I try the following way to save images, but the saved image is not I expected.

for i, (inputs) in enumerate(test_generator):
    pred = modelPl(inputs.float()).detach()
    fig,ax = plt.subplots(1,1,figsize = (5,5))
    ax.imshow(pred[0,:,:], cmap = "gray")
    print(pred.shape)
    torchvision.utils.save_image(pred, saveTestPath + 'img_{0}.png'.format(i))

Output:
torch.Size([400, 400])

Expected image:

However, the saved picture is not correct as follows:

Output:
It might be that torchvision.utils.save_image requires values to be in range 0 to 1. Your images have values which are greater than 1 and hence the problem.

You can check this by dividing the tensor by 255 (or some appropriate number). You can also try to set normalize=True and see if it can automatically normalize the data for you.


Instruction: PyTorch doesn't seem to be optimizing correctly

Input:
I have posted this question on Data Science StackExchange site since StackOverflow does not support LaTeX. Linking it here because this site is probably more appropriate.

The question with correctly rendered LaTeX is here: https://datascience.stackexchange.com/questions/48062/pytorch-does-not-seem-to-be-optimizing-correctly

The idea is that I am considering sums of sine waves with different phases. The waves are sampled with some sample rate s in the interval [0, 2pi]. I need to select phases in such a way, that the sum of the waves at any sample point is minimized.

Below is the Python code. Optimization does not seem to be computed correctly.

import numpy as np
import torch

def phaseOptimize(n, s = 48000, nsteps = 1000):
    learning_rate = 1e-3

    theta = torch.zeros([n, 1], requires_grad=True)
    l = torch.linspace(0, 2 * np.pi, s)
    t = torch.stack([l] * n)
    T = t + theta

    for jj in range(nsteps):
        loss = T.sin().sum(0).pow(2).sum() / s
        loss.backward()
        theta.data -= learning_rate * theta.grad.data

    print('Optimal theta: \n\n', theta.data)
    print('\n\nMaximum value:', T.sin().sum(0).abs().max().item())

Below is a sample output.

phaseOptimize(5, nsteps=100)

Optimal theta:

 tensor([[1.2812e-07],
        [1.2812e-07],
        [1.2812e-07],
        [1.2812e-07],
        [1.2812e-07]], requires_grad=True)

Maximum value: 5.0

I am assuming this has something to do with broadcasting in

T = t + theta

and/or the way I am computing the loss function.

One way to verify that optimization is incorrect, is to simply evaluate the loss function at random values for the array $\theta_1, \dots, \theta_n$, say uniformly distributed in $[0, 2\pi]$. The maximum value in this case is almost always much lower than the maximum value reported by phaseOptimize(). Much easier in fact is to consider the case with $n = 2$, and simply evaluate at $\theta_1 = 0$ and $\theta_2 = \pi$. In that case we get:

phaseOptimize(2, nsteps=100)

Optimal theta:

 tensor([[2.8599e-08],
        [2.8599e-08]])

Maximum value: 2.0

On the other hand,

theta = torch.FloatTensor([[0], [np.pi]])
l = torch.linspace(0, 2 * np.pi, 48000)
t = torch.stack([l] * 2)
T = t + theta

T.sin().sum(0).abs().max().item()

produces

3.2782554626464844e-07

Output:
You have to move computing T inside the loop, or it will always have the same constant value, thus constant loss.

Another thing is to initialize theta to different values at indices, otherwise because of the symmetric nature of the problem the gradient is the same for every index.

Another thing is that you need to zero gradient, because backward just accumulates them.

This seems to work:

def phaseOptimize(n, s = 48000, nsteps = 1000):
    learning_rate = 1e-1

    theta = torch.zeros([n, 1], requires_grad=True)
    theta.data[0][0] = 1
    l = torch.linspace(0, 2 * np.pi, s)
    t = torch.stack([l] * n)

    for jj in range(nsteps):
        T = t + theta
        loss = T.sin().sum(0).pow(2).sum() / s
        loss.backward()
        theta.data -= learning_rate * theta.grad.data
        theta.grad.zero_()


Instruction: Converting Keras (Tensorflow) convolutional neural networks to PyTorch convolutional networks?

Input:
Keras and PyTorch use different arguments for padding: Keras requires a string to be input, while PyTorch works with numbers. What is the difference, and how can one be translated into another (what code gets the equivalent result in either framework)?

PyTorch also takes the args in_channels, out_chanels while keras only takes an argument called filters. What does 'filters' mean?

Output:
Regarding padding,

Keras => 'valid' - no padding;
         'same'  - input is padded so that the output shape is same as input shape

Pytorch => you explicitly specify the padding

Valid padding

>>> model = keras.Sequential()
>>> model.add(keras.layers.Conv2D(filters=10, kernel_size=3, padding='valid', input_shape=(28,28,3)))
>>> model.layers[0].output_shape
(None, 26, 26, 10)

>>> x = torch.randn((1,3,28,28))
>>> conv = torch.nn.Conv2d(in_channels=3, out_channels=10, kernel_size=3)
>>> conv(x).shape
torch.Size([1, 10, 26, 26])

Same padding

>>> model = keras.Sequential()
>>> model.add(keras.layers.Conv2D(filters=10, kernel_size=3, padding='same', input_shape=(28,28,3)))
>>> model.layers[0].output_shape
(None, 28, 28, 10)

>>> x = torch.randn((1,3,28,28))
>>> conv = torch.nn.Conv2d(in_channels=3, out_channels=10, kernel_size=3, padding=1)
>>> conv(x).shape
torch.Size([1, 10, 28, 28])

W - Input Width, F - Filter(or kernel) size, P - padding, S - Stride, Wout - Output width

Wout = ((W−F+2P)/S)+1

Similarly for Height. With this formula, you can calculate the amount of padding required to retain the input width or height in the output.

http://cs231n.github.io/convolutional-networks/

Regarding in_channels, out_chanels and filters,

filters is the same as out_channels. In Keras, the in_channels is automatically inferred from the previous layer shape or input_shape(in case of first layer).