Instruction: PyTorch set_grad_enabled(False) vs with no_grad():

Input:
Assuming autograd is on (as it is by default), is there any difference (besides  indent) between doing:

with torch.no_grad():

and

torch.set_grad_enabled(False)

torch.set_grad_enabled(True)

Output:
Actually no, there no difference in the way used in the question. When you take a look at the source code of no_grad. You see that it is actually using torch.set_grad_enabled to archive this behaviour:

class no_grad(object):
    r"""Context-manager that disabled gradient calculation.

    Disabling gradient calculation is useful for inference, when you are sure
    that you will not call :meth:`Tensor.backward()`. It will reduce memory
    consumption for computations that would otherwise have `requires_grad=True`.
    In this mode, the result of every computation will have
    `requires_grad=False`, even when the inputs have `requires_grad=True`.

    Also functions as a decorator.

    Example::

        >>> x = torch.tensor([1], requires_grad=True)
        >>> with torch.no_grad():
        ...   y = x * 2
        >>> y.requires_grad
        False
        >>> @torch.no_grad()
        ... def doubler(x):
        ...     return x * 2
        >>> z = doubler(x)
        >>> z.requires_grad
        False
    """

    def __init__(self):
        self.prev = torch.is_grad_enabled()

    def __enter__(self):
        torch._C.set_grad_enabled(False)

    def __exit__(self, *args):
        torch.set_grad_enabled(self.prev)
        return False

    def __call__(self, func):
        @functools.wraps(func)
        def decorate_no_grad(*args, **kwargs):
            with self:
                return func(*args, **kwargs)
        return decorate_no_grad

However there is an additional functionality of torch.set_grad_enabled  over torch.no_grad when used in a with-statement which lets you control to switch on or off gradient computation:

    >>> x = torch.tensor([1], requires_grad=True)
    >>> is_train = False
    >>> with torch.set_grad_enabled(is_train):
    ...   y = x * 2
    >>> y.requires_grad

https://pytorch.org/docs/stable/_modules/torch/autograd/grad_mode.html

Edit:

@TomHale Regarding your comment. I just made a short test with PyTorch 1.0 and it turned out that the gradient will be active:

import torch
w = torch.rand(5, requires_grad=True)
print('Grad Before:', w.grad)
torch.set_grad_enabled(False)
with torch.enable_grad():
    scalar = w.sum()
    scalar.backward()
    # Gradient tracking will be enabled here.
torch.set_grad_enabled(True)

print('Grad After:', w.grad)

Output:

Grad Before: None
Grad After: tensor([1., 1., 1., 1., 1.])

So gradients will be computed in this setting.

The other setting you posted in your answer also yields to the same result:

import torch
w = torch.rand(5, requires_grad=True)
print('Grad Before:', w.grad)
with torch.no_grad():
    with torch.enable_grad():
        # Gradient tracking IS enabled here.
        scalar = w.sum()
        scalar.backward()

print('Grad After:', w.grad)

Output:

Grad Before: None
Grad After: tensor([1., 1., 1., 1., 1.])


Instruction: understanding seed of a ByteTensor in PyTorch

Input:
I understand that a seed is a number used to initialize pseudo-random number generator. in pytorch, torch.get_rng_state documentation states as follows "Returns the random number generator state as a torch.ByteTensor.". and when i print it i get a 1-d tensor of size 5048 whose values are as shown below

  tensor([ 80,  78, 248,  ...,   0,   0,   0], dtype=torch.uint8)

why does a seed have 5048 values and how is this different from usual seed which we can get using  torch.initial_seed

Output:
It sounds like you're thinking of the seed and the state as equivalent.  For older pseudo-random number generators (PRNGs) that was true, but with more modern PRNGs tend to work as described here. (The answer in the link was written with respect to Mersenne Twister, but the concepts apply equally to other generators.)

Why is it a good idea to not have a 32- or 64-bit state space and report the state as the generator's output?  Because if you do that, as soon as you see any value repeat the entire sequence will repeat.  PRNGs were designed to be "full cycle," i.e., to iterate through the maximum number of values possible before repeating.  This paper showed that the birthday problem could quickly (O(sqrt(cycle-length)) identify such PRNGs as non-random.  This meant, for instance, that with 32-bit integers you shouldn't use more than ~50000 values before a statistician could call you out with a better than 99% level of confidence.  The solution, used by many modern PRNGs, is to have a larger state space and collapse it down to output a 32- or 64-bit result.  Since multiple states can produce the same output, duplicates will occur in the output stream without the entire stream being replicated.  It looks like that's what PyTorch is doing.

Given the larger state space, why allow seeding with a single integer?  Convenience.  For instance, Mersenne Twister has a 19,937 bit state space, but most people don't want to enter that much info to kick-start it.  You can if you want to, but most people use the front-end which populates the full state space from a single integer input.


Instruction: Load huge image dataset and some data augmentation using Pytorch

Input:
I have a question. I have about 2 million images (place365-standard dataset) and I want to do some data augmentation like transforming, cropping etc. Also, I have to make my own target image (y) based on some color model algorithms (CMYK) for example.

So Actually, my preprocessing step includes augmentation and making terget image (y). And then I should feed these images to a deep network. When should I do this based on Dataset class? Should I do my processing step in __getitem__()? If yes, would it be parallel and fast?

Here is my template of Dataset(data.Dataset) class:

import torch
from torch.utils import data

class Dataset(data.Dataset):
    """
    Return Dataset class representing our data set
    """
    def __int__(self, list_IDs, labels):
        """
        Initialize data set as a list of IDs corresponding to each item of data set and labels of each data

        Args:
            list_IDs: a list of IDs for each data point in data set
            labels: label of an item in data set with respect to the ID
        """

        self.labels = labels
        self.list_IDs = list_IDs

    def __len__(self):
        """
        Return the length of data set using list of IDs

        :return: number of samples in data set
        """
        return len(self.list_IDs)

    def __getitem__(self, item):
        """
        Generate one item of data set. Here we apply our preprocessing things like halftone styles and subtractive color process using CMYK color model etc. (See the paper for operations)

        :param item: index of item in IDs list

        :return: a sample of data
        """
        ID = self.list_IDs[item]

        # Code to load data
        X = None #

        # code to apply your custom function to make y image (time consuming task - some algorithms)
        y = None #

        return X, y

Thanks for any advice

Best regards

Output:
If you look at e.g., torchvision.dataset.ImageFolder you'll see that it works quite similar to your design: the class has transform member that lists all sorts of augmentations (resizing, cropping, flipping etc.) and these are carried out on the images in the __getitem__ method.
Regarding parallelism, the Dataset itself is not parallel, but the DataLoader can be (see num_workers argument), so if you use your dataset inside a parallel dataloader you get the parallelism for free, Cool!


Instruction: Pytorch LSTM: Target Dimension in Calculating Cross Entropy Loss

Input:
I've been trying to get an LSTM (LSTM followed by a linear layer in a custom model), working in Pytorch, but was getting the following error when calculating the loss:

Assertion cur_target >= 0 && cur_target < n_classes' failed.

I defined the loss function with:

criterion = nn.CrossEntropyLoss()

and then called with

loss += criterion(output, target)

I was giving the target with dimensions [sequence_length, number_of_classes], and output has dimensions [sequence_length, 1, number_of_classes].

The examples I was following seemed to be doing the same thing, but it was different on the Pytorch docs on cross entropy loss.

The docs say the target should be of dimension (N), where each value is  0 ≤ targets[i] ≤ C−1 and C is the number of classes. I changed the target to be in that form, but now I'm getting an error saying (The sequence length is 75, and there are 55 classes):

Expected target size (75, 55), got torch.Size([75])

I've tried looking at solutions for both errors, but still can't get this working properly. I'm confused as to the proper dimensions of target, as well as the actual meaning behind the first error (different searches gave very different meanings for the error, none of the fixes worked).

Thanks

Output:
You can use squeeze() on your output tensor, this returns a tensor with all the dimensions of size 1 removed.

This short code uses the shapes you mentioned in your question:

sequence_length   = 75
number_of_classes = 55
# creates random tensor of your output shape
output = torch.rand(sequence_length, 1, number_of_classes)
# creates tensor with random targets
target = torch.randint(55, (75,)).long()

# define loss function and calculate loss
criterion = nn.CrossEntropyLoss()
loss = criterion(output, target)
print(loss)

Results in the error you described:

ValueError: Expected target size (75, 55), got torch.Size([75])

So using squeeze() on your output tensor solves your problem by getting it to correct shape.

Example with corrected shape:

sequence_length   = 75
number_of_classes = 55
# creates random tensor of your output shape
output = torch.rand(sequence_length, 1, number_of_classes)
# creates tensor with random targets
target = torch.randint(55, (75,)).long()

# define loss function and calculate loss
criterion = nn.CrossEntropyLoss()

# apply squeeze() on output tensor to change shape form [75, 1, 55] to [75, 55]
loss = criterion(output.squeeze(), target)
print(loss)

Output:

tensor(4.0442)

Using squeeze() changes your tensor shape from [75, 1, 55] to [75, 55] so it that output and target shape matches!

You can also use other methods to reshape your tensor, it is just important that you have the shape of [sequence_length, number_of_classes] instead of [sequence_length, 1, number_of_classes].

Your targets should be a LongTensor resp. a tensor of type torch.long containing the classes. Shape here is [sequence_length].

Edit:
Shapes from above example when passing to cross-entropy function:

Outputs: torch.Size([75, 55])
Targets: torch.Size([75])

Here is a more general example what outputs and targets should look like for CE. In this case we assume we have 5 different target classes, there are three examples for sequences of length 1, 2 and 3:

# init CE Loss function
criterion = nn.CrossEntropyLoss()

# sequence of length 1
output = torch.rand(1, 5)
# in this case the 1th class is our target, index of 1th class is 0
target = torch.LongTensor([0])
loss = criterion(output, target)
print('Sequence of length 1:')
print('Output:', output, 'shape:', output.shape)
print('Target:', target, 'shape:', target.shape)
print('Loss:', loss)

# sequence of length 2
output = torch.rand(2, 5)
# targets are here 1th class for the first element and 2th class for the second element
target = torch.LongTensor([0, 1])
loss = criterion(output, target)
print('\nSequence of length 2:')
print('Output:', output, 'shape:', output.shape)
print('Target:', target, 'shape:', target.shape)
print('Loss:', loss)

# sequence of length 3
output = torch.rand(3, 5)
# targets here 1th class, 2th class and 2th class again for the last element of the sequence
target = torch.LongTensor([0, 1, 1])
loss = criterion(output, target)
print('\nSequence of length 3:')
print('Output:', output, 'shape:', output.shape)
print('Target:', target, 'shape:', target.shape)
print('Loss:', loss)

Output:

Sequence of length 1:
Output: tensor([[ 0.1956,  0.0395,  0.6564,  0.4000,  0.2875]]) shape: torch.Size([1, 5])
Target: tensor([ 0]) shape: torch.Size([1])
Loss: tensor(1.7516)

Sequence of length 2:
Output: tensor([[ 0.9905,  0.2267,  0.7583,  0.4865,  0.3220],
        [ 0.8073,  0.1803,  0.5290,  0.3179,  0.2746]]) shape: torch.Size([2, 5])
Target: tensor([ 0,  1]) shape: torch.Size([2])
Loss: tensor(1.5469)

Sequence of length 3:
Output: tensor([[ 0.8497,  0.2728,  0.3329,  0.2278,  0.1459],
        [ 0.4899,  0.2487,  0.4730,  0.9970,  0.1350],
        [ 0.0869,  0.9306,  0.1526,  0.2206,  0.6328]]) shape: torch.Size([3, 5])
Target: tensor([ 0,  1,  1]) shape: torch.Size([3])
Loss: tensor(1.3918)

I hope this helps!


Instruction: Pytorch: "Model Weights not Changing"

Input:
Can someone help me understand why the weights are not updating?

    unet = Unet()
    optimizer = torch.optim.Adam(unet.parameters(), lr=0.001)
    loss_fn = torch.nn.MSELoss()
    input =  Variable(torch.randn(32, 1, 64, 64, 64 ), requires_grad=True)
    target = Variable(torch.randn(32, 1, 64, 64, 64), requires_grad=False)

    optimizer.zero_grad()
    y_pred = unet(input)
    y = target[: , : , 20:44, 20:44, 20:44]

    loss = loss_fn(y_pred, y)
    print(unet.conv1.weight.data[0][0]) # weights of the first layer in the unet
    loss.backward()
    optimizer.step()
    print(unet.conv1.weight.data[0][0]) # weights havent changed

The model is defined like:

class Unet(nn.Module):

def __init__(self):
  super(Unet, self).__init__()

  # Down hill1
  self.conv1 = nn.Conv3d(1, 2, kernel_size=3,  stride=1)
  self.conv2 = nn.Conv3d(2, 2, kernel_size=3,  stride=1)

  # Down hill2
  self.conv3 = nn.Conv3d(2, 4, kernel_size=3,  stride=1)
  self.conv4 = nn.Conv3d(4, 4, kernel_size=3,  stride=1)

  #bottom
  self.convbottom1 = nn.Conv3d(4, 8, kernel_size=3,  stride=1)
  self.convbottom2 = nn.Conv3d(8, 8, kernel_size=3,  stride=1)

  #up hill1
  self.upConv0 = nn.Conv3d(8, 4, kernel_size=3,  stride=1)
  self.upConv1 = nn.Conv3d(4, 4, kernel_size=3,  stride=1)
  self.upConv2 = nn.Conv3d(4, 2, kernel_size=3,  stride=1)

  #up hill2
  self.upConv3 = nn.Conv3d(2, 2, kernel_size=3, stride=1)
  self.upConv4 = nn.Conv3d(2, 1, kernel_size=1, stride=1)

  self.mp = nn.MaxPool3d(kernel_size=3, stride=2, padding=1)
  # some more irrelevant properties...

The forward function looks like:

def forward(self, input):
    # Use U-net Theory to Update the filters.
    # Example Approach...
    input = F.relu(self.conv1(input))
    input = F.relu(self.conv2(input))

    input = self.mp(input)

    input = F.relu(self.conv3(input))
    input = F.relu(self.conv4(input))

    input = self.mp(input)

    input = F.relu(self.convbottom1(input))
    input = F.relu(self.convbottom2(input))

    input = F.interpolate(input, scale_factor=2, mode='trilinear')

    input = F.relu(self.upConv0(input))
    input = F.relu(self.upConv1(input))

    input = F.interpolate(input, scale_factor=2, mode='trilinear')

    input = F.relu(self.upConv2(input))
    input = F.relu(self.upConv3(input))

    input = F.relu(self.upConv4(input))

    return input

I have followed the approach of any example and documentation i could find and it is beyound me why that doesn't work?

I can figure out as much that y_pred.grad after the backward call is none which it shouldn't be. If we have no gradient then ofcourse the optimizer can't change the weights in any direction but why is there no gradient?

Output:
I identified this problem to be of "The Dying ReLu Problem" Due to the data being Hounsfield units and Pytorch uniform distribution of initial weights meant that many neurons would start out in ReLu's zero region leaving them paralyzed and dependable on other neurons to produce a gradient that could pull them out of the zero region. This unlikely to happen as training progresses all neurons gets pushed into ReLu's zero region.

There are several solutions to this problem. You can use Leaky_relu or other activation functions that do not have a zero region.

You can also normalize the input data using Batch Normalization and initialize the weights to only be of the positive kind.

Solution number two is probably the most optimal solution since both will solve the problem but leaky_relu will prolong training whereas Batch normalization will do the opposite and increase accuracy. On the other hand, Leaky_relu is an easy fix whereas the other solution requires a little extra work.

For Hounsfield data, one could also add a constant of 1000 to the input eliminating negative units from the data. This still requires a different weight initialization than Pytorch's standard initialization.


Instruction: Shape of pytorch model.parameter is inconsistent with how it's defined in the model

Input:
I'm attempting to extract the weights and biases from a simple network built in PyTorch. My entire network is composed of nn.Linear layers. When I create a layer by calling nn.Linear(in_dim, out_dim), I expect the parameters that I get from calling model.parameters() for that model to be of shape (in_dim, out_dim) for the weight and (out_dim) for the bias. However, the weights that come out of model.parameters() are instead of shape (out_dim, in_dim).

The intention of my code is to be able to use matrix multiplication to perform a forward pass using only numpy, not any PyTorch. Because of the shape inconsistency, matrix multiplications throw an error. How can I fix this?

Here is my exact code:

class RNN(nn.Module):

    def __init__(self, dim_input, dim_recurrent, dim_output):

        super(RNN, self).__init__()

        self.dim_input = dim_input
        self.dim_recurrent = dim_recurrent
        self.dim_output = dim_output

        self.dense1 = nn.Linear(self.dim_input, self.dim_recurrent)
        self.dense2 = nn.Linear(self.dim_recurrent, self.dim_recurrent, bias = False)
        self.dense3 = nn.Linear(self.dim_input, self.dim_recurrent)
        self.dense4 = nn.Linear(self.dim_recurrent, self.dim_recurrent, bias = False)
        self.dense5 = nn.Linear(self.dim_recurrent, self.dim_output)

#There is a defined forward pass

model = RNN(12, 100, 6)

for i in model.parameters():
    print(i.shape())

The output is:

torch.Size([100, 12])
torch.Size([100])
torch.Size([100, 100])
torch.Size([100, 12])
torch.Size([100])
torch.Size([100, 100])
torch.Size([6, 100])
torch.Size([6])

The output should, if I'm correct, be:

torch.Size([12, 100])
torch.Size([100])
torch.Size([100, 100])
torch.Size([12, 100])
torch.Size([100])
torch.Size([100, 100])
torch.Size([100, 6])
torch.Size([6])

What is my issue?

Output:
What you see there is not the (out_dim, in_dim), it is just the shape of the weight matrix. When you call print(model) you can see that input and output features are correct:

RNN(
  (dense1): Linear(in_features=12, out_features=100, bias=True)
  (dense2): Linear(in_features=100, out_features=100, bias=False)
  (dense3): Linear(in_features=12, out_features=100, bias=True)
  (dense4): Linear(in_features=100, out_features=100, bias=False)
  (dense5): Linear(in_features=100, out_features=6, bias=True)
)

You can check the source code to see that the weights are actually transposed before calling matmul.

nn.Linear is define here:
https://pytorch.org/docs/stable/_modules/torch/nn/modules/linear.html#Linear

You can check the forward, it looks like this:

def forward(self, input):
    return F.linear(input, self.weight, self.bias)

F.linear is define here:
https://pytorch.org/docs/stable/_modules/torch/nn/functional.html

The respective line for multiplying the weights is:

output = input.matmul(weight.t())

As mentioned above you can see that the weights are transposed before applying matmul and therefore the shape of the weights is different than you expected.

So if you want to do the matrix multiplication manually, you do:

# dummy input of length 5
input = torch.rand(5, 12)
# apply layer dense1 (without bias, for bias just add + model.dense1.bias)
output_first_layer = input.matmul(model.dense1.weight.t())
print(output_first_layer.shape)

Just as you would expect from your dense1 it returns:

torch.Size([5, 100])

I hope this explains your observations with the shape :)


Instruction: Sparse tensors to decrease training time

Input:
I've learning about PyTorch sparse tensors : https://pytorch.org/docs/stable/sparse.html

From the docs (https://pytorch.org/docs/stable/sparse.html) : "Torch supports sparse tensors in COO(rdinate) format, which can efficiently store and process tensors for which the majority of elements are zeros."

Is one of the intended use of sparse tensors instead of regular PyTorch tensors to decrease training time ?

Output:
Yes but indirectly.

sparse tensors can reduce the complexity of computations and hence training/inference time. Complexity of matrix multiplication depends on number of elements in matrix whereas complexity of sparse matrix multiplication would depend on the number of non-zero elements which are less (due to sparsity)


Instruction: Auto updating custom layer parameters while backpropagating in pytorch

Input:
I have a pytorch custom layer defined as:

class MyCustomLayer(nn.Module):
  def __init__(self):
    super(MyCustomLayer, self).__init__()

    self.my_parameter = torch.rand(1, requires_grad = True)

    # the following allows the previously defined parameter to be recognized as a network parameter when instantiating the model
    self.my_registered_parameter = nn.ParameterList([nn.Parameter(self.my_parameter)])

  def forward(self, x):
    return x*self.my_parameter

I then define my network where the custom layer is used:

class MyNet(nn.Module):
  def __init__(self):
    super(MyNet, self).__init__()
    self.layer1 = MyCustomLayer()

  def forward(self, x):
    x = self.layer1(x)
    return x

Now Let's instantiate MyNet and observe the issue:

# instantiate MyNet and run it over one input value
model = MyNet()
x = torch.tensor(torch.rand(1))
output = model(x)
criterion = nn.MSELoss()
loss = criterion(1, output)
loss.backward()

Iterating through model parameters shows None for custom layer parameter:

for p in model.parameters():
    print (p.grad)

None

while directly accessing that parameter shows the correct grad value:

print(model.layer1.my_parameter.grad)

tensor([-1.4370])

This, in turn, prevents the optim step from updating the inner parameters automatically and leaves me with the hassle of having to update those manually. Anyone knows how I can address this issue?

Output:
What you did i.e. return x*self.my_registered_parameter[0] worked because you use the registered param for calculating the gradient.

When you call nn.Parameter it returns a new object and hence self.my_parameter that you use for the operation and the one registered are not same.

You can fix this by declaring the my_parameter as nn.Parameter

self.my_parameter = nn.Parameter(torch.rand(1, requires_grad = True))
self.my_registered_parameter= nn.ParameterList([self.some_parameter])

or you don't need to create my_registered_parameter variable at all. When you declare self.my_parameter as nn.Parameter it gets registered as a parameter.


Instruction: How does `images, labels = dataiter.next() ` work in PyTorch Tutorial?

Input:
From the tutorial cifar10_tutorial, how is images, labels assigned?

trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
                                        download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,
                                          shuffle=True, num_workers=2)

def imshow(img):
    img = img / 2 + 0.5     # unnormalize
    npimg = img.numpy()
    plt.imshow(np.transpose(npimg, (1, 2, 0)))

# get some random training images
dataiter = iter(trainloader)

images, labels = dataiter.next()

How does the last line know how to automatically assign images, label in images, labels = dataiter.next()?

I checked the DataLoader class and the DataLoaderIter class, but think I need a bit more knowledge of iters in general.

Output:
I think it is crucial to understand the difference between an iterable and an iterator. An iterable is an object that you can iterate over. An Iterator is an object which is used to iterate over an iterable object using the __next__ method, which returns the next item of the object.

A simple example is the following. Consider an iterable and use the next method to call the next item in the list. This will print the next item until the end of the list is reached. If the end is reached it will raise a StopIteration error.

test = (1,2,3)
tester = iter(test)

while True:
    nextItem = next(tester)
    print(nextItem)

The class you refer to above probably has an implementation similar to this, however it returns a tuple containing the image and the label.


Instruction: PyTorch - shape of nn.Linear weights

Input:
Yesterday I came across this question and for the first time noticed that the weights of the linear layer nn.Linear need to be transposed before applying matmul.

Code for applying the weights:

output = input.matmul(weight.t())

What is the reason for this?

Why are the weights not in the transposed shape just from the beginning, so they don't need to be transposed every time before applying the layer?

Output:
I found an answer here:
Efficient forward pass in nn.Linear #2159

It seems like there is no real reasoning behind this. However the transpose operation doesn't seem to be slowing down the computation.

According to the issue mentioned above, during the forward pass the transpose operation is (almost) free in terms of computation. While during the backward pass leaving out the transpose operation would actually make computation less efficient with the current implementation.

The last post in that issue sums it up quite nicely:

  It's historical weight layout, changing it is backward-incompatible.
  Unless there is some BIG benefit in terms of speed or convenience, we
  wont break userland.

https://github.com/pytorch/pytorch/issues/2159#issuecomment-390068272


Instruction: PyTorch - multiplying tensor with scalar results in zero vector

Input:
I have no idea why the result is all 0 with tensor. Anything wrong here?

>>> import torch
>>> import numpy as np
>>> import math

>>> torch.__version__
'0.4.1'
>>> np.__version__
'1.15.4'

>>> torch.arange(0, 10, 2) *-(math.log(10000.0) / 10)
tensor([0, 0, 0, 0, 0])
>>> np.arange(0, 10, 2) *-(math.log(10000.0) / 10)
array([-0.        , -1.84206807, -3.68413615, -5.52620422, -7.3682723 ])

>>> torch.arange(0, 10, 2)
tensor([0, 2, 4, 6, 8])
>>> np.arange(0, 10, 2)
array([0, 2, 4, 6, 8])

Output:
As written in the comment when using 0.4.0 get the same results as with numpy:

tensor([-0.0000, -1.8421, -3.6841, -5.5262, -7.3683])

However with 0.4.1 I'm getting a zero vector too.

The reason for this is that torch.arange(0, 10, 2) returns a tensor of type float for 0.4.0 while it returns a tensor of type long for 0.4.1.

So casting your tensor to float should work for you:

torch.arange(0, 10, 2).float() *-(math.log(10000.0) / 10)

Multiplying long and float works by heavy rounding, as the result is still a tensor of type long. So when converting a FloatTensor to a LongTensor values between -1 and 1 will be rounded to 0.

Since -(math.log(10000.0) / 10) results in -0.9210340371976183 your result is 0. So effectively -0.9210340371976183 is converted to type long before multiplying. But when converting it will be round down to 0, see this example:

t = torch.tensor((-(math.log(10000.0) / 10)))
print('FloatTensor:', t)
print('Converted to Long:', t.long())

Outout:

FloatTensor: tensor(-0.9210)
Converted to Long: tensor(0)

Thus:

torch.arange(0, 10, 2).float() *-(math.log(10000.0) / 10)

becomes:

torch.arange(0, 10, 2).float() * 0

Therefore you get a tensor of zeros as result.

Some more examples:

If you multiply it with a value between 1 and 2, lets say 1.7, it will always been rounded down to 1:

t = torch.tensor(range(5), dtype=torch.long)
print(t)
print(t * 1.7)

Output:

tensor([ 0,  1,  2,  3,  4])
tensor([ 0,  1,  2,  3,  4])

And similarly when multiplying with 2.7 results in an effective multiplication of 2:

t = torch.tensor(range(5), dtype=torch.long)
print(t)
print(t * 2.7)

Output:

tensor([ 0,  1,  2,  3,  4])
tensor([ 0,  2,  4,  6,  8])


Instruction: Index pytorch 4d tensor by values in 2d tensor

Input:
I have two pytorch tensors:

X with shape (A, B, C, D)
I with shape (A, B)

Values in I are integers in range [0, C).

What is the most efficient way to get tensor Y with shape (A, B, D), such that:

Y[i][j][k] = X[i][j][ I[i][j] ][k]

Output:
You probably want to use torch.gather for the indexing and expand to adjust I to the required size:

eI = I[..., None, None].expand(-1, -1, 1, X.size(3))  # make eI the same for the last dimension
Y = torch.gather(X, dim=2, index=eI).squeeze()

testing the code:

A = 3
B = 4
C = 5
D = 7

X = torch.rand(A, B, C, D)
I = torch.randint(0, C, (A, B), dtype=torch.long)

eI = I[..., None, None].expand(-1, -1, 1, X.size(3))
Y = torch.gather(X, dim=2, index=eI).squeeze()

# manually gather
refY = torch.empty(A, B, D)
for i in range(A):
    for j in range(B):
        refY[i, j, :] = X[i, j, I[i,j], :]

(refY == Y).all()
# Out[]: tensor(1, dtype=torch.uint8)


Instruction: How to convert 3D tensor to 2D tensor in pytorch?

Input:
I am new to pytorch. I have 3D tensor (32,10,64) and I want a 2D tensor (32, 64).
I tried view() and used after passing to linear layer squeeze() which converted it to (32,10).

Output:
Try this

t = torch.rand(32, 10, 64).permute(0, 2, 1)[:, :, -1]

or, as pointed out by Shai, you could also

t = torch.rand(32, 10, 64)[:, -1, :]

print(t.size()) # torch.Size([32, 64])


Instruction: Understanding the softmax output layer of RNN

Input:
Here's a simple LSTM model in Keras:

input = Input(shape=(max_len,))
model = Embedding(input_dim=input_dim, output_dim=embed_dim, input_length=max_len)(input)
model = Dropout(0.1)(model)
model = Bidirectional(LSTM(units=blstm_dim, return_sequences=True, recurrent_dropout=0.1))(model)
out =Dense(label_dim, activation="softmax")(model)

Here's my attempt to translate it to a Pytorch model:

class RNN(nn.Module):
    def __init__(self, input_dim, embed_dim, blstm_dim, label_dim):
        super(RNN, self).__init__()
        self.embed = nn.Embedding(input_dim, embed_dim)
        self.blstm = nn.LSTM(embed_dim, blstm_dim,  bidirectional=True, batch_first=True)
        self.fc = nn.Linear(2*blstm_dim, label_dim)

    def forward(self, x):
        h0 = torch.zeros(2, x.size(0), blstm_dim).to(device)
        c0 = torch.zeros(2, x.size(0), blstm_dim).to(device)
        x = self.embed(x)
        x = F.dropout(x, p=0.1, training=self.training)
        x,_ = self.blstm(x, (h0, c0))
        x = self.fc(x)
        return F.softmax(x, dim=1)
#        return x

Now running the Keras model gives the following:

Epoch 5/5
38846/38846 [==============================] - 87s 2ms/step - loss: 0.0374 - acc: 0.9889 - val_loss: 0.0473 - val_acc: 0.9859

But running the PyTorch model gives this:

Train Epoch: 10/10 [6400/34532 (19%)]   Loss: 2.788933
Train Epoch: 10/10 [12800/34532 (37%)]  Loss: 2.788880
Train Epoch: 10/10 [19200/34532 (56%)]  Loss: 2.785547
Train Epoch: 10/10 [25600/34532 (74%)]  Loss: 2.796180
Train Epoch: 10/10 [32000/34532 (93%)]  Loss: 2.790446
Validation: Average loss: 0.0437, Accuracy: 308281/431600 (71%)

I've made sure the loss & optimiser are the same (cross entropy & RMSprop). Now interestingly if I remove the softmax from the PyTorch model (i.e. use the hashed output in the code, I get what seems to be right:

Train Epoch: 10/10 [32000/34532 (93%)]  Loss: 0.022118
Validation: Average loss: 0.0009, Accuracy: 424974/431600 (98%)

So here are my questions:

1) Are the two models I've printed above equivalent (let's ignore the recurrent_dropout since I haven't figure out how to do that in PyTorch)?

2) What am I doing wrong with the softmax output layer in PyTorch?

Thanks a lot!

Output:
Are the two models I've printed above equivalent (let's ignore the recurrent_dropout since I haven't figure out how to do that in PyTorch)?

Besides the dropout I can see no difference. So they should be completely equivalent in terms of structure.

One note: You don't have to initialize the states if you use it this way (if you're not reusing the states). You can just forward the LSTM with x,_ = self.blstm(x) - it will automatically initialize states with zeros.

What am I doing wrong with the softmax output layer in PyTorch?

PyTorch torch.nn.CrossEntropyLoss already includes softmax:

  This criterion combines nn.LogSoftmax() and nn.NLLLoss() in one
  single class.

So it is actually a CE with logits. I guess this makes it more efficient. So you can just leave out the softmax activation at the end.


Instruction: AttributeError: 'tuple' object has no attribute 'log_softmax'

Input:
while trying to finetune inception_V3 for my own dataset by changing the last fc layer like

last_layer =nn.Linear(n_inputs, len(classes))
inception_v3.fc = last_layer

after that when I train it got this error on this position

    # on training loop
    output = inception_v3(data)
    # calculate the batch loss
    loss = criterion(output, target)

Error is

 AttributeError: 'tuple' object has no attribute 'log_softmax'

Output:
This is a well known problem.

Try one the following solutions:

disable aux_logits when the model is created here by also passing aux_logits=False to the inception_v3 function.
edit your train function to accept and unpack the returned tuple to be something like: output, aux = model(input_var)

Check the following link for more info.


Instruction: Pytorch DataLoader multiple data source

Input:
I am trying to use Pytorch dataloader to define my own dataset, but I am not sure how to load multiple data source:

My current code:

class MultipleSourceDataSet(Dataset):
    def __init__ (self, json_file, root_dir, transform = None):
        with open(root_dir + 'block0.json') as f:
            self.result = torch.Tensor(json.load(f))

    self.root_dir = root_dir
    self.transform = transform

    def __len__(self):
        return len(self.result[0])

    def __getitem__ (self):
        None

The data source is 50 blocks under root_dir = ~/Documents/blocks/

I split them and avoid to combine them directly before since this is a very big dataset.

How can I load them into a single dataloader?

Output:
For DataLoader you need to have a single Dataset, your problem is that you have multiple 'json' files and you only know how to create a Dataset from each 'json' separately.
What you can do in this case is to use ConcatDataset that contains all the single-'json' datasets you create:

import os
import torch.utils.data as data

class SingeJsonDataset(data.Dataset):
    # implement a single json dataset here...

list_of_datasets = []
for j in os.path.listdir(root_dir):
    if not j.endswith('.json'):
        continue  # skip non-json files
    list_of_datasets.append(SingeJsonDataset(json_file=j, root_dir=root_dir, transform=None))
# once all single json datasets are created you can concat them into a single one:
multiple_json_dataset = data.ConcatDataset(list_of_datasets)

Now you can feed the concatenated dataset into data.DataLoader.


Instruction: Munging PyTorch's tensor shape from (C, B, H) to (B, C*H)

Input:
Given an input tensor of shape (C, B, H) torch.Size([2, 5, 32]) of some neural net layers, where

channels = 2
batch_size = 5
hidden_size = 32

The goal is to flatten the channels and manipulate the input tensor to the shape (B, C*H) torch.Size([5, 2 * 32]), where:

batch_size = 5
hidden_size = 32 * 2

I've tried to do the following:

import torch

t = torch.rand([2, 5, 32])

# Changed from (channels, batch_size, hidden_size)
# -> (batch_size, channels, hidden_size)
t = t.permute(1, 0, 2)

# Reshape using view(), where batch_size is t.size(0)
# and -1 is to flatten the left over values to the other dimension.
z = t.contiguous().view(t.size(0), -1)

print(z.shape)
print(z)

[out]:

torch.Size([5, 64])
tensor([[0.3911, 0.9586, 0.2104, 0.3937, 0.9976, 0.3378, 0.0630, 0.6676, 0.0806,
         0.9311, 0.5219, 0.1697, 0.7442, 0.5162, 0.2555, 0.0826, 0.5502, 0.9700,
         0.3375, 0.5012, 0.9025, 0.8176, 0.1465, 0.1848, 0.3460, 0.9999, 0.7892,
         0.7577, 0.6615, 0.2620, 0.6868, 0.2003, 0.4840, 0.8354, 0.9253, 0.3172,
         0.9516, 0.8962, 0.1272, 0.2268, 0.6510, 0.5166, 0.6772, 0.9616, 0.9826,
         0.5254, 0.9191, 0.4378, 0.7048, 0.8808, 0.0299, 0.1102, 0.9710, 0.8714,
         0.7256, 0.9684, 0.6117, 0.1957, 0.8663, 0.4742, 0.2843, 0.6548, 0.9592,
         0.1559],
        [0.2333, 0.0858, 0.5284, 0.2965, 0.3863, 0.3370, 0.6940, 0.3387, 0.3513,
         0.1022, 0.3731, 0.3575, 0.7095, 0.0053, 0.7024, 0.4091, 0.3289, 0.5808,
         0.5640, 0.8847, 0.7584, 0.8878, 0.9873, 0.0525, 0.7731, 0.2501, 0.9926,
         0.5226, 0.0925, 0.0300, 0.4176, 0.0456, 0.4643, 0.4497, 0.5920, 0.9519,
         0.6647, 0.2379, 0.4927, 0.9666, 0.1675, 0.9887, 0.7741, 0.5668, 0.7376,
         0.4452, 0.7449, 0.1298, 0.9065, 0.3561, 0.5813, 0.1439, 0.2115, 0.5874,
         0.2038, 0.1066, 0.3843, 0.6179, 0.8321, 0.9428, 0.1067, 0.5045, 0.9324,
         0.3326],
        [0.6556, 0.1479, 0.9288, 0.9238, 0.1324, 0.0718, 0.6620, 0.2659, 0.7162,
         0.7559, 0.7564, 0.2120, 0.3943, 0.9497, 0.7520, 0.8455, 0.4444, 0.4708,
         0.8371, 0.6365, 0.3616, 0.0326, 0.1581, 0.4973, 0.6701, 0.9245, 0.8274,
         0.3464, 0.7044, 0.5376, 0.0441, 0.5210, 0.8603, 0.7396, 0.2544, 0.3514,
         0.5686, 0.3283, 0.7248, 0.4303, 0.9531, 0.5587, 0.8703, 0.1585, 0.9161,
         0.9043, 0.9778, 0.4489, 0.9463, 0.8655, 0.5576, 0.1135, 0.1268, 0.3424,
         0.1504, 0.2265, 0.1734, 0.1872, 0.3995, 0.1191, 0.0532, 0.6109, 0.1662,
         0.6937],
        [0.6342, 0.1922, 0.1758, 0.4625, 0.7654, 0.6509, 0.2908, 0.1546, 0.4768,
         0.3779, 0.2490, 0.0086, 0.6170, 0.5425, 0.6953, 0.4730, 0.5834, 0.8326,
         0.0165, 0.8236, 0.0023, 0.7479, 0.5621, 0.9894, 0.5957, 0.0857, 0.6087,
         0.5667, 0.5478, 0.8197, 0.9228, 0.7329, 0.4434, 0.5894, 0.9860, 0.6133,
         0.2395, 0.4718, 0.8830, 0.6361, 0.6104, 0.6630, 0.5084, 0.7604, 0.7591,
         0.3601, 0.6888, 0.6767, 0.9178, 0.5291, 0.0591, 0.4320, 0.7875, 0.5038,
         0.4419, 0.0319, 0.3719, 0.5843, 0.0334, 0.3525, 0.0023, 0.1205, 0.4040,
         0.7908],
        [0.0989, 0.8436, 0.0425, 0.6247, 0.6091, 0.4778, 0.2692, 0.4785, 0.9217,
         0.9604, 0.6355, 0.4686, 0.9414, 0.7722, 0.8013, 0.1660, 0.6578, 0.6414,
         0.6814, 0.6212, 0.4124, 0.7102, 0.7416, 0.7404, 0.9842, 0.6542, 0.0106,
         0.3826, 0.5529, 0.8079, 0.9855, 0.3012, 0.2341, 0.9353, 0.6597, 0.7177,
         0.8214, 0.1438, 0.4729, 0.6747, 0.9310, 0.4167, 0.3689, 0.8464, 0.9395,
         0.9407, 0.8419, 0.5486, 0.1786, 0.1423, 0.9900, 0.9365, 0.3996, 0.1862,
         0.6232, 0.7547, 0.7779, 0.4767, 0.6218, 0.9079, 0.6153, 0.1488, 0.5960,
         0.4015]])

Although the permute() + view() achieve the desired output, are there other ways to perform the same operation? Is there a better way that can directly rehape without first permutating the order of the shape?

Output:
Let's look "behind the curtain" and see why one must have both permute/transpose and view in order to go from a C-B-H to B-C*H:

Elements of tensors are stored as a long contiguous vector in memory. For instance, if you look at a 2-3-4 tensor it has 24 elements stored at 24 consecutive places in memory. This tensor also has a "header" that tells pytorch to treat these 24 values as a 2-by-3-by-4 tensor. This is done by storing not only the size of the tensor, but also "strides": what is the "stride" one need to jump in order to get to the next element along each dimension. In our example, size=(2,3,4) and strides=(12, 4, 1) (you can check this out yourself, and you can see more about it here).

Now, if you only want to change the size to 2-(3*4) you do not need to move any item of the tensor in memory, only to update the "header" of the tensor. By setting size=(2, 12) and strides=(12, 1) you are done!

Alternatively, if you want to "transpose" the tensor to 3-2-4 that's a bit more tricky, but you can still do that by manipulating the strides. Setting size=(3, 2, 4) and strides=(4, 12, 1) gives you exactly what you want without moving any of the real tensor elements in memory.

However, once you manipulated the strides, you cannot trivially change the size of the tensor - because now you will need to have two different "stride" values for one (or more) dimensions. This is why you must call contiguous() at this point.

Summary
If you want to move from shape (C, B, H) to (B, C*H) you must have permute, contiguous and view operations, otherwise you just scramble the entries of your tensor.

A small example with 2-3-4 tensor:

a =
array([[[ 0,  1,  2,  3],
        [ 4,  5,  6,  7],
        [ 8,  9, 10, 11]],

       [[12, 13, 14, 15],
        [16, 17, 18, 19],
        [20, 21, 22, 23]]])

If you just change the view of the tensor you get

a.view(3,8)
array([[ 0,  1,  2,  3,  4,  5,  6,  7],
       [ 8,  9, 10, 11, 12, 13, 14, 15],
       [16, 17, 18, 19, 20, 21, 22, 23]])

Which is not what you want!
You need to have

a.permute(1,0,2).contiguous().view(3, 8)
array([[ 0,  1,  2,  3, 12, 13, 14, 15],
       [ 4,  5,  6,  7, 16, 17, 18, 19],
       [ 8,  9, 10, 11, 20, 21, 22, 23]])


Instruction: Matrix multiplication in pyTorch

Input:
I'm writing a simple neural network in pyTorch, where features and weights both are (1, 5) tensors. What are the differences between the two methods that I mention below?

y = activation(torch.sum(features*weights) + bias)

and

yy = activation(torch.mm(features, weights.view(5,1)) + bias)

Output:
Consider it step by step:

x = torch.tensor([[10, 2], [3,5]])
y = torch.tensor([[1,3], [5,6]])

x * y
# tensor([[10,  6],
#         [15, 30]])

torch.sum(x*y)

#tensor(61)

x = torch.tensor([[10, 2], [3,5]])
y = torch.tensor([[1,3], [5,6]])

np.matmul(x, y)
# array([[20, 42],
#       [28, 39]])

So there is a difference betweeen matmul and * operator. Furthermore, torch.sum makes an entire sum from the tensor, not row or columnwisely.


Instruction: ValueError: expected 2D or 3D input (got 1D input) PyTorch

Input:
class VAE(torch.nn.Module):

 def __init__(self, input_size, hidden_sizes, batch_size):

    super(VAE, self).__init__()

    self.input_size = input_size
    self.hidden_sizes = hidden_sizes
    self.batch_size = batch_size
    self.fc = torch.nn.Linear(input_size, hidden_sizes[0])
    self.BN = torch.nn.BatchNorm1d(hidden_sizes[0])
    self.fc1 = torch.nn.Linear(hidden_sizes[0], hidden_sizes[1])
    self.BN1 = torch.nn.BatchNorm1d(hidden_sizes[1])
    self.fc2 = torch.nn.Linear(hidden_sizes[1], hidden_sizes[2])
    self.BN2 = torch.nn.BatchNorm1d(hidden_sizes[2])
    self.fc3_mu = torch.nn.Linear(hidden_sizes[2], hidden_sizes[3])
    self.fc3_sig = torch.nn.Linear(hidden_sizes[2], hidden_sizes[3])

    self.fc4 = torch.nn.Linear(hidden_sizes[3], hidden_sizes[2])
    self.BN4 = torch.nn.BatchNorm1d(hidden_sizes[2])
    self.fc5 = torch.nn.Linear(hidden_sizes[2], hidden_sizes[1])
    self.BN5 = torch.nn.BatchNorm1d(hidden_sizes[1])
    self.fc6 = torch.nn.Linear(hidden_sizes[1], hidden_sizes[0])
    self.BN6 = torch.nn.BatchNorm1d(hidden_sizes[0])
    self.fc7 = torch.nn.Linear(hidden_sizes[0], input_size)

def sample_z(self, x_size, mu, log_var):

     eps = torch.randn(x_size, self.hidden_sizes[-1])
     return(mu + torch.exp(log_var/2) * eps)

 def forward(self, x):

    ###########
    # Encoder #
    ###########

    out1 = self.fc(x)
    out1 = nn.relu(self.BN(out1))
    out2 = self.fc1(out1)
    out2 = nn.relu(self.BN1(out2))
    out3 = self.fc2(out2)
    out3 = nn.relu(self.BN2(out3))
    mu = self.fc3_mu(out3)
    sig = nn.softplus(self.fc3_sig(out3))

    ###########
    # Decoder  #
    ###########

    # sample from the distro
    sample = self.sample_z(x.size(0), mu, sig)
    out4 = self.fc4(sample)
    out4 = nn.relu(self.BN4(out4))
    out5 = self.fc5(out4)
    out5 = nn.relu(self.BN5(out5))
    out6 = self.fc6(out5)
    out6 = nn.relu(self.BN6(out6))
    out7 = nn.sigmoid(self.fc7(out6))

    return(out7, mu, sig)

vae = VAE(input_size, hidden_sizes, batch_size)

vae.eval()

x_sample, z_mu, z_var = vae(X)

The error is:

File "VAE_LongTensor.py", line 200, in <module>
x_sample, z_mu, z_var = vae(X)
ValueError: expected 2D or 3D input (got 1D input)

Output:
When you build a nn.Module in pytorch for processing 1D signals, pytorch actually expects the input to be 2D: first dimension is the "mini batch" dimension.
Thus you need to add a singleton dimesion to your X:

x_sample, z_mu, z_var = vae(X[None, ...])


Instruction: Pytorch RuntimeError: size mismatch, m1: [1 x 7744], m2: [400 x 120]

Input:
In a simple CNN that classifies 5 objects, I get a size mis-match error:

"RuntimeError: size mismatch, m1: [1 x 7744], m2: [400 x 120]" in the convolutional layer .

my model.py file:

import torch.nn as nn
import torch.nn.functional as F

class FNet(nn.Module):

    def __init__(self,device):
        # make your convolutional neural network here
        # use regularization
        # batch normalization
        super(FNet, self).__init__()
        num_classes = 5
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.conv2 = nn.Conv2d(6, 16, 5)
        # an affine operation: y = Wx + b
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 5)

    def forward(self, x):

        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))

        x = F.max_pool2d(F.relu(self.conv2(x)), 2)
        x = x.view(-1, self.num_flat_features(x))
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

    def num_flat_features(self, x):
        size = x.size()[1:]  # all dimensions except the batch dimension
        num_features = 1
        for s in size:
            num_features *= s
        return num_features

if __name__ == "__main__":
    net = FNet()

Complete Error:

Traceback (most recent call last):
  File "main.py", line 98, in <module>
    train_model('../Data/fruits/', save=True, destination_path='/home/mitesh/E yantra/task1#hc/Task 1/Task 1B/Data/fruits')
  File "main.py", line 66, in train_model
    outputs = model(images)
  File "/home/mitesh/anaconda3/envs/HC#850_stage1/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/mitesh/E yantra/task1#hc/Task 1/Task 1B/Code/model.py", line 28, in forward
    x = F.relu(self.fc1(x))
  File "/home/mitesh/anaconda3/envs/HC#850_stage1/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/mitesh/anaconda3/envs/HC#850_stage1/lib/python3.6/site-packages/torch/nn/modules/linear.py", line 55, in forward
    return F.linear(input, self.weight, self.bias)
  File "/home/mitesh/anaconda3/envs/HC#850_stage1/lib/python3.6/site-packages/torch/nn/functional.py", line 1024, in linear
    return torch.addmm(bias, input, weight.t())
RuntimeError: size mismatch, m1: [1 x 7744], m2: [400 x 120] at /opt/conda/conda-bld/pytorch-cpu_1532576596369/work/aten/src/TH/generic/THTensorMath.cpp:2070

Output:
If you have a nn.Linear layer in your net, you cannot decide "on-the-fly" what the input size for this layer would be.
In your net you compute num_flat_features for every x and expect your self.fc1 to handle whatever size of x you feed the net. However, self.fc1 has a fixed size weight matrix of size 400x120 (that is expecting input of dimension 16*5*5=400 and outputs 120 dim feature). In your case the size of x translated to 7744 dim feature vector that self.fc1 simply cannot handle.

If you do want your network to be able to handle any size x, you can have a parameter-free interpolation layer resizing all x to the right size before self.fc1:

x = F.max_pool2d(F.relu(self.conv2(x)), 2)  # output of conv layers
x = F.interpolate(x, size=(5, 5), mode='bilinear')  # resize to the size expected by the linear unit
x = x.view(x.size(0), 5 * 5 * 16)
x = F.relu(self.fc1(x))  # you can go on from here...

See torch.nn.functional.interpolate for more information.


Instruction: PyTorch - parameters not changing

Input:
In an effort to learn how pytorch works, I am trying to do maximum likelihood estimation of some of the parameters in a multivariate normal distribution. However it does not seem to work for any of the covariance related parameters.

So my question is: why does this code not work?

import torch

def make_covariance_matrix(sigma, rho):
    return torch.tensor([[sigma[0]**2, rho * torch.prod(sigma)],
                         [rho * torch.prod(sigma), sigma[1]**2]])

mu_true = torch.randn(2)
rho_true = torch.rand(1)
sigma_true = torch.exp(torch.rand(2))

cov_true = make_covariance_matrix(sigma_true, rho_true)
dist_true = torch.distributions.MultivariateNormal(mu_true, cov_true)

samples = dist_true.sample((1_000,))

mu = torch.zeros(2, requires_grad=True)
log_sigma = torch.zeros(2, requires_grad=True)
atanh_rho = torch.zeros(1, requires_grad=True)

lbfgs = torch.optim.LBFGS([mu, log_sigma, atanh_rho])

def closure():
    lbfgs.zero_grad()
    sigma = torch.exp(log_sigma)
    rho = torch.tanh(atanh_rho)
    cov = make_covariance_matrix(sigma, rho)
    dist = torch.distributions.MultivariateNormal(mu, cov)
    loss = -torch.mean(dist.log_prob(samples))
    loss.backward()
    return loss

lbfgs.step(closure)

print("mu: {}, mu_hat: {}".format(mu_true, mu))
print("sigma: {}, sigma_hat: {}".format(sigma_true, torch.exp(log_sigma)))
print("rho: {}, rho_hat: {}".format(rho_true, torch.tanh(atanh_rho)))

output:

mu: tensor([0.4168, 0.1580]), mu_hat: tensor([0.4127, 0.1454], requires_grad=True)
sigma: tensor([1.1917, 1.7290]), sigma_hat: tensor([1., 1.], grad_fn=<ExpBackward>)
rho: tensor([0.3589]), rho_hat: tensor([0.], grad_fn=<TanhBackward>)

>>> torch.__version__
'1.0.0.dev20181127'

In other words, why have the estimates of log_sigma and atanh_rho not moved from their initial value?

Output:
The way you create your covariance matrix is not backprob-able:

def make_covariance_matrix(sigma, rho):
    return torch.tensor([[sigma[0]**2, rho * torch.prod(sigma)],
                         [rho * torch.prod(sigma), sigma[1]**2]])

When creating a new tensor from (multiple) tensors, only the values of your input tensors will be kept. All additional information from the input tensors is stripped away, thus all graph-connection to your parameters is cut from this point, therefore backpropagation cannot get through.

Here is a short example to illustrate this:

import torch

param1 = torch.rand(1, requires_grad=True)
param2 = torch.rand(1, requires_grad=True)
tensor_from_params = torch.tensor([param1, param2])

print('Original parameter 1:')
print(param1, param1.requires_grad)
print('Original parameter 2:')
print(param2, param2.requires_grad)
print('New tensor form params:')
print(tensor_from_params, tensor_from_params.requires_grad)

Output:

Original parameter 1:
tensor([ 0.8913]) True
Original parameter 2:
tensor([ 0.4785]) True
New tensor form params:
tensor([ 0.8913,  0.4785]) False

As you can see the tensor, created from the parameters param1 and param2, does not keep track of the gradients of param1 and param2.

So instead you can use this code that keeps the graph connection and is backprob-able:

def make_covariance_matrix(sigma, rho):
    conv = torch.cat([(sigma[0]**2).view(-1), rho * torch.prod(sigma), rho * torch.prod(sigma), (sigma[1]**2).view(-1)])
    return conv.view(2, 2)

The values are concatenated to a flat tensor using torch.cat. Then they are brought into right shape using view(). This results in the same matrix output as in your function, but it keeps the connection to your parameters log_sigma and atanh_rho.

Here is an output before and after the step with the changed make_covariance_matrix. As you can see, now you can optimize your parameters and the values do change:

Before:
mu: tensor([ 0.1191,  0.7215]), mu_hat: tensor([ 0.,  0.])
sigma: tensor([ 1.4222,  1.0949]), sigma_hat: tensor([ 1.,  1.])
rho: tensor([ 0.2558]), rho_hat: tensor([ 0.])

After:
mu: tensor([ 0.1191,  0.7215]), mu_hat: tensor([ 0.0712,  0.7781])
sigma: tensor([ 1.4222,  1.0949]), sigma_hat: tensor([ 1.4410,  1.0807])
rho: tensor([ 0.2558]), rho_hat: tensor([ 0.2235])

Hope this helps!


Instruction: Row-wise Element Indexing in PyTorch for C++

Input:
I am using the C++ frontend for PyTorch and am struggling with a relatively basic indexing problem.

I have an 8 by 6 Tensor such as the one below:

[ Variable[CUDAFloatType]{8,6} ]
                 0           1           2           3           4           5
0       1.7107e-14  4.0448e-17  4.9708e-06  1.1664e-08  9.9999e-01  2.1857e-20
1       1.8288e-14  5.9356e-17  5.3042e-06  1.2369e-08  9.9999e-01  2.4799e-20
2       2.6828e-04  9.0390e-18  1.7517e-02  1.0529e-03  9.8116e-01  6.7854e-26
3       5.7521e-10  3.1037e-11  1.5021e-03  1.2304e-06  9.9850e-01  1.4888e-17
4       1.7811e-13  1.8383e-15  1.6733e-05  3.8466e-08  9.9998e-01  5.2815e-20
5       9.6191e-06  2.6217e-23  3.1345e-02  2.3024e-04  9.6842e-01  2.9435e-34
6       2.2653e-04  8.4642e-18  1.6085e-02  9.7405e-04  9.8271e-01  6.3059e-26
7       3.8951e-14  2.9903e-16  8.3518e-06  1.7974e-08  9.9999e-01  3.6993e-20

I have another Tensor with just 8 elements in it such as:

[ Variable[CUDALongType]{8} ]
 0
 3
 4
 4
 4
 4
 4
 4

I would like to index the rows of my first tensor using the second to produce:

        0
0       1.7107e-14
1       1.2369e-08
2       9.8116e-01
3       9.9850e-01
4       9.9998e-01
5       9.6842e-01
6       9.8271e-01
7       9.9999e-01

I have tried a few different approaches including index_select but it seems to produce an output that has the same dimensions as the input (8x6).

In Python I think I could index with Python's built-in indexing as discussed here: https://github.com/pytorch/pytorch/issues/1080

Unfortunately, in C++ I can only index a Tensor with a scalar (zero-dimensional Tensor) so I don't think that approach works for me here.

How can I achieve my desired result without resorting to loops?

Output:
It turns out you can do this in a couple different ways. One with gather and one with index. From the PyTorch discussions where I asked the same question:

Using torch::gather

auto x = torch::randn({8, 6});
int64_t idx_data[8] = { 0, 3, 4, 4, 4, 4, 4, 4 };
auto idx = x.type().toScalarType(torch::kLong).tensorFromBlob(idx_data, 8);
auto result = x.gather(1, idx.unsqueeze(1));

Using the C++ specific torch::index

auto x = torch::randn({8, 6});
int64_t idx_data[8] = { 0, 3, 4, 4, 4, 4, 4, 4 };
auto idx = x.type().toScalarType(torch::kLong).tensorFromBlob(idx_data, 8);
auto rows = torch::arange(0, x.size(0), torch::kLong);
auto result = x.index({rows, idx});


Instruction: How make customised dataset in Pytorch for images and their masks?

Input:
I have two dataset folder of tif images, one is a folder called BMMCdata, and the other one is the mask of BMMCdata images called BMMCmasks(the name of images are corresponds). I am trying to make a customised dataset and also split the data randomly to train and test. at the moment I am getting an error

self.filenames.append(fn)
AttributeError: 'CustomDataset' object has no attribute 'filenames'

Any comment will be appreciated a lot.

import torch
from torch.utils.data.dataset import Dataset  # For custom data-sets
from torchvision import transforms
from PIL import Image
import os.path as osp
import glob

folder_data = "/Users/parto/PycharmProjects/U-net/BMMCdata/data"

class CustomDataset(Dataset):
def __init__(self, root):

    self.filename = folder_data
    self.root = root
    self.to_tensor = transforms.ToTensor()
    filenames = glob.glob(osp.join(folder_data, '*.tif'))
    for fn in filenames:
        self.filenames.append(fn)
    self.len = len(self.filenames)
    print(fn)

def __getitem__(self, index):
    image = Image.open(self.filenames[index])
    return self.transform(image)

def __len__(self):

    return self.len
custom_img = CustomDataset(folder_data)
# total images in set
print(custom_img.len)

train_len = int(0.6*custom_img.len)
test_len = custom_img.len - train_len
train_set, test_set = CustomDataset.random_split(custom_img, lengths=[train_len, test_len])
# check lens of subset
len(train_set), len(test_set)

train_set = CustomDataset(folder_data)
train_set = torch.utils.data.TensorDataset(train_set, train=True, batch_size=4)
train_loader = torch.utils.data.DataLoader(train_set, batch_size=4, shuffle=True, num_workers=1)
print(train_set)
print(train_loader)

test_set = torch.utils.data.DataLoader(Dataset, batch_size=4, sampler= train_sampler)
test_loader = torch.utils.data.DataLoader(Dataset, batch_size=4)

Output:
answer given by @ptrblck in pytorch community. thank you

 # get all the image and mask path and number of images
 folder_data = glob.glob("D:\\Neda\\Pytorch\\U-net\\BMMCdata\\data\\*.tif")
 folder_mask = glob.glob("D:\\Neda\\Pytorch\\U-net\\BMMCmasks\\masks\\*.tif")

 # split these path using a certain percentage
 len_data = len(folder_data)
 print(len_data)
 train_size = 0.6

 train_image_paths = folder_data[:int(len_data*train_size)]
 test_image_paths = folder_data[int(len_data*train_size):]

 train_mask_paths = folder_mask[:int(len_data*train_size)]
 test_mask_paths = folder_mask[int(len_data*train_size):]

 class CustomDataset(Dataset):
    def __init__(self, image_paths, target_paths, train=True):   # initial logic
      happens like transform

         self.image_paths = image_paths
         self.target_paths = target_paths
         self.transforms = transforms.ToTensor()

    def __getitem__(self, index):

        image = Image.open(self.image_paths[index])
        mask = Image.open(self.target_paths[index])
        t_image = self.transforms(image)
     return t_image, mask

def __len__(self):  # return count of sample we have

    return len(self.image_paths)

train_dataset = CustomDataset(train_image_paths, train_mask_paths, train=True)
train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=4, shuffle=True, num_workers=1)

test_dataset = CustomDataset(test_image_paths, test_mask_paths, train=False)
test_loader = torch.utils.data.DataLoader(test_dataset, batch_size=4, shuffle=False, num_workers=1)


Instruction: How do I split the training dataset into training, validation and test datasets?

Input: I have a custom data set of images and its target. I have created a training data set in PyTorch. I want to split it into 3 parts: training, validation and test. How do I do it?

Output:
Once you have the "master" dataset you can use data.Subset to split it.
Here's an example for random split

import torch
from torch.utils import data
import random

master = data.Dataset( ... )  # your "master" dataset
n = len(master)  # how many total elements you have
n_test = int( n * .05 )  # number of test/val elements
n_train = n - 2 * n_test
idx = list(range(n))  # indices to all elements
random.shuffle(idx)  # in-place shuffle the indices to facilitate random splitting
train_idx = idx[:n_train]
val_idx = idx[n_train:(n_train + n_test)]
test_idx = idx[(n_train + n_test):]

train_set = data.Subset(master, train_idx)
val_set = data.Subset(master, val_idx)
test_set = data.Subset(master, test_idx)

This can also be achieved using data.random_split:

train_set, val_set, test_set = data.random_split(master, (n_train, n_val, n_test))


Instruction: Save and load checkpoint pytorch

Input:
i make a model and save the configuration as:

def checkpoint(state, ep, filename='./Risultati/checkpoint.pth'):
    if ep == (n_epoch-1):
        print('Saving state...')
        torch.save(state,filename)
checkpoint({'state_dict':rnn.state_dict()},ep)

and then i want load this configuration :

state_dict= torch.load('./Risultati/checkpoint.pth')
    rnn.state_dict(state_dict)

when i try, this is the error:

Traceback (most recent call last):
File "train.py", line 288, in <module>
rnn.state_dict(state_dict)
File "/home/marco/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 593, in state_dict
destination._metadata[prefix[:-1]] = dict(version=self._version)
AttributeError: 'dict' object has no attribute '_metadata'

where i do wrong?

thx in advance

Output:
You need to load rnn.state_dict() stored in the dictionary you loaded:

rnn.load_state_dict(state_dict['state_dict'])

Look at load_state_dict method for more information.


Instruction: What is the backward process of max operation in deep learning?

Input:
I know that the backward process of deep learning follows the gradient descent algorithm. However, there is never a gradient concept for max operation.

How does deep learning frameworks like tensorflow, pytorch deal with the backward of 'max' operation like maxpooling?

Output:
You have to think of  what the max operator actually does? That is:

It returns or lets better say it propagates the maximum.

And that's exactly what it does here - it takes two or more tensors and propagates forward (only) the maximum.

It is often helpful to take a look at a short example:

t1 = torch.rand(10, requires_grad=True)
t2 = torch.rand(10, requires_grad=True)

s1 = torch.sum(t1)
s2 = torch.sum(t2)
print('sum t1:', s1, 'sum t2:', s2)
m = torch.max(s1, s2)
print('max:', m, 'requires_grad:', m.requires_grad)
m.backward()
print('t1 gradients:', t1.grad)
print('t2 gradients:', t2.grad)

This code creates two random tensors sums them up and puts them through a max function. Then backward() is called upon the result.

Lets take a look at the two possible outcomes:

Outcome 1 - sum of t1 is larger:

sum t1: tensor(5.6345) sum t2: tensor(4.3965)
max: tensor(5.6345) requires_grad: True
t1 gradients: tensor([ 1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.])
t2 gradients: tensor([ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.])

Outcome 2 - sum of t2 is larger:

sum t1: tensor(3.3263) sum t2: tensor(4.0517)
max: tensor(4.0517) requires_grad: True
t1 gradients: tensor([ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.])
t2 gradients: tensor([ 1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.])

As you would expect in the case s1 represents the maximum gradients will be calculated for t1. Similarly when s2 is the maximum gradients will be calculated for t2.

Similarly like in the forward step, back propagation is propagating backwards through the maximum.

One thing worth mentioning is that the other tensors which do not represent the maximum are still part of the graph. Only the gradients are set to zero then. If they wouldn't be part of the graph you would get None as gradient, instead of a zero vector.

You can check what happens if you use python-max instead of torch.max:

t1 = torch.rand(10, requires_grad=True)
t2 = torch.rand(10, requires_grad=True)

s1 = torch.sum(t1)
s2 = torch.sum(t2)
print('sum t1:', s1, 'sum t2:', s2)
m = max(s1, s2)
print('max:', m, 'requires_grad:', m.requires_grad)
m.backward()
print('t1 gradients:', t1.grad)
print('t2 gradients:', t2.grad)

Output:

sum t1: tensor(4.7661) sum t2: tensor(4.4166)
max: tensor(4.7661) requires_grad: True
t1 gradients: tensor([ 1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.])
t2 gradients: None


Instruction: Pytorch: Normalize Image data set

Input: I want to normalize custom dataset of images. For that i need to compute mean and standard deviation by iterating over the dataset. How can I normalize my entire dataset before creating the data set?

Output:
Well, let's take this image as an example:

The first thing you need to do is decide which library you want to use: Pillow or OpenCV. In this example I'll use Pillow:

from PIL import Image
import numpy as np

img = Image.open("test.jpg")
pix = np.asarray(img.convert("RGB")) # Open the image as RGB

Rchan = pix[:,:,0]  # Red color channel
Gchan = pix[:,:,1]  # Green color channel
Bchan = pix[:,:,2]  # Blue color channel

Rchan_mean = Rchan.mean()
Gchan_mean = Gchan.mean()
Bchan_mean = Bchan.mean()

Rchan_var = Rchan.var()
Gchan_var = Gchan.var()
Bchan_var = Bchan.var()

And the results are:

Red Channel Mean: 134.80585625
Red Channel Variance: 3211.35843945
Green Channel Mean: 81.0884125
Green Channel Variance: 1672.63200823
Blue Channel Mean: 68.1831375
Blue Channel Variance: 1166.20433566

Hope it helps for your needs.


Instruction: How to mask weights in PyTorch weight parameters?

Input:
I am attempting to mask (force to zero) specific weight values in PyTorch. The weights I am trying to mask are defined as so in the def __init__

class LSTM_MASK(nn.Module):
        def __init__(self, options, inp_dim):
            super(LSTM_MASK, self).__init__()
            ....
            self.wfx = nn.Linear(input_dim, curernt_output, bias=add_bias)

The mask is also defined in def __init__ as

self.mask_use = torch.Tensor(curernt_output, input_dim)

The mask is a constant and the .requires_grad_() is False for the mask parameter. Now in the def forward part of the class I attempt to do an element-wise multiplication of the weight parameter and the mask before the linear operation is completed

def forward(self, x):
    ....
    self.wfx.weight = self.wfx.weight * self.mask_use
    wfx_out = self.wfx(x)

I get ann error message:

self.wfx.weight = self.wfx.weight * self.mask_use
  File "/home/xyz/anaconda2/lib/python2.7/site-packages/torch/nn/modules/module.py", line 537, in __setattr__
    .format(torch.typename(value), name))
TypeError: cannot assign 'torch.cuda.FloatTensor' as parameter 'weight' (torch.nn.Parameter or None expected)

But when I check on the both parameters with .type() both of them come up as torch.cuda.FloatTensor. I am not sure why there is an error here.

Output:
The element-wise operation always returns a FloatTensor. It is not possible to assign normal tensors as weight of layers.

There are two possible options to deal with it. You can assign it to the data attribute of your weight, there it is possible assign normal tensors.

Or alternatively you convert your result to an nn.Parameter itself, then you can assign it to wfx.weight.

Here is an example which shows both ways:

import torch
import torch.nn as nn

wfx = nn.Linear(10, 10)
mask_use = torch.rand(10, 10)
#wfx.weight = wfx.weight * mask_use #your example - this raises an error

# Option 1: write directly to data
wfx.weight.data = wfx.weight * mask_use

# Option 2: convert result to nn.Parameter and write to weight
wfx.weight = nn.Parameter(wfx.weight * mask_use)

Disclaimer: When using an = (assignment) on the weights you are replacing the weights tensor of your parameter. This may have unwanted effects on the graph resp. the optimization step.


Instruction: Plot the derivative of a function with PyTorch?

Input:
I have this code:

import torch
import matplotlib.pyplot as plt
x=torch.linspace(-10, 10, 10, requires_grad=True)
y = torch.sum(x**2)
y.backward()
plt.plot(x.detach().numpy(), y.detach().numpy(), label='function')
plt.legend()

But, I got this error:

ValueError: x and y must have same first dimension, but have shapes (10,) and (1,)

Output:
I think the main problem is that your dimensions do not match. Why do you wan't to use torch.sum?

This should work for you:

# %matplotlib inline added this line only for jupiter notebook
import torch
import matplotlib.pyplot as plt
x = torch.linspace(-10, 10, 10, requires_grad=True)

y = x**2      # removed the sum to stay with the same dimensions
y.backward(x) # handing over the parameter x, as y isn't a scalar anymore
# your function
plt.plot(x.detach().numpy(), y.detach().numpy(), label='x**2')
# gradients
plt.plot(x.detach().numpy(), x.grad.detach().numpy(), label='grad')
plt.legend()

You get a nicer picture though with more steps, I also changed the interval a bit to torch.linspace(-2.5, 2.5, 50, requires_grad=True).

Edit regarding comment:

This version plots you the gradients with torch.sum included:

# %matplotlib inline added this line only for jupiter notebook
import torch
import matplotlib.pyplot as plt
x = torch.linspace(-10, 10, 10, requires_grad=True)

y = torch.sum(x**2)
y.backward()
print(x.grad)
plt.plot(x.detach().numpy(), x.grad.detach().numpy(), label='grad')
plt.legend()

Output:

tensor([-20.0000, -15.5556, -11.1111,  -6.6667,  -2.2222,   2.2222,
      6.6667,  11.1111,  15.5556,  20.0000])

Plot:


Instruction: Pytorch Loading two Images from Dataloader

Input:
I'm trying to make a GAN which takes a lo-res image, and tries to create a hi-res image from it. To do this, I need to user a Dataloader which has both the hi-res and low-res training images stored in it.

     data_transform = transforms.Compose([transforms.Resize(imageSize),
                                         transforms.Grayscale(num_output_channels=1),
                                         transforms.ToTensor()])

    dataset_hi = "./hi-res-train"
    dataset_lo = "./low-res-train"

    img_data_hi = dset.ImageFolder(root=dataset_hi,transform=data_transform)
    img_data_lo = dset.ImageFolder(root=dataset_lo,transform=data_transform)

    dataloader_hi = torch.utils.data.DataLoader(img_data_hi, batch_size = batchSize, shuffle = True, num_workers = 2)
    dataloader_lo = torch.utils.data.DataLoader(img_data_lo, batch_size = batchSize, shuffle = True, num_workers = 2)

I've tried using two separate data loaders (shown above) but when they are shuffled, I cant enumerate through them both because the hi-res and low-res images are not matched up. How can I make it so I can enumerate and shuffle both with pytorch?

Output:
Assuming you have similar names for hi & low resolution images (say img01_hi & img01_low), one option is to create a custom Dataloader that returns both images by overriding __getitem__ method.

As both images are returned in one call, you can make sure they match by appending _hi & _low to the filename.

You may need to create a "cue" text file containing list of all your image file names to make sure you are processing each image file only once.


Instruction: What transformation do I need to do in order to run dataset through neural network?

Input:
I'm new to deep learning and Pytorch, but I hope someone can help me out with this. My dataset contains images from different sizes. I'm trying to create a simple neural network that can classify images. However, I'm getting mismatch errors.

Neural network

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 32, 3)
        self.conv2 = nn.Conv2d(32, 32, 3)
        self.fc1 = nn.Linear(32 * 3 * 3, 200)
        self.fc2 = nn.Linear(200, 120)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        x = F.relu(self.conv2(x))
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        return x
net = Net()

My first convolution layer has 1 input channel, because I transform the images to grayscale images. 32 output channels was an arbitrary decision. The final fully-connected layer has 120 output channels, because there are 120 different classes.

Determine transformations and assign training set and validation set

transform = transforms.Compose(
    [transforms.Grayscale(1),
     transforms.RandomCrop((32,32)),
     transforms.ToTensor(),
     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

data_dir = 'dataset'
full_dataset = datasets.ImageFolder(os.path.join(data_dir, 'train'), transform = transform)

train_size = int(0.8 * len(full_dataset))
val_size = len(full_dataset) - train_size
trainset, valset = torch.utils.data.random_split(full_dataset, [train_size, val_size])

trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,
                                           shuffle=True, num_workers=2)
valloader = torch.utils.data.DataLoader(valset, batch_size=4,
                                           shuffle=False, num_workers=2)
classes = full_dataset.classes

I transform the images to grayscale, because they are gray anyway. I crop the images to 32, because the images have different sizes and I figured that they must all be the same size when putting it through the neural network. Everything is working fine so far.

Train neural network

for epoch in range(2):  # loop over the dataset multiple times

    running_loss = 0.0
    for i, data in enumerate(trainloader, 0):
        # get the inputs
        inputs, labels = data

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = net(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()
        if i % 2000 == 1999:    # print every 2000 mini-batches
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 2000))
            running_loss = 0.0

print('Finished Training')

When running this last piece of code, I get the following error: size mismatch, m1: [3584 x 28], m2: [288 x 200] at /Users/soumith/miniconda2/conda-bld/pytorch_1532623076075/work/aten/src/TH/generic/THTensorMath.cpp:2070 when the following line is being executed: outputs = net(inputs)

My code is a variation of the code provided in this Pytorch tutorial. Can someone tell me what I'm doing wrong?

UPDATE

I updated the neural network class to this:

class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()
        # 1 input image channel, 6 output channels, 5x5 square convolution
        # kernel
        self.conv1 = nn.Conv2d(1, 6, 5)
        self.conv2 = nn.Conv2d(6, 16, 5)
        # an affine operation: y = Wx + b
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        # Max pooling over a (2, 2) window
        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
        # If the size is a square you can only specify a single number
        x = F.max_pool2d(F.relu(self.conv2(x)), 2)
        x = x.view(-1, self.num_flat_features(x))
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

    def num_flat_features(self, x):
        size = x.size()[1:]  # all dimensions except the batch dimension
        num_features = 1
        for s in size:
            num_features *= s
        return num_features

net = Net()

But now I get an error at loss = criterion(outputs, labels):
Assertion cur_target >= 0 && cur_target < n_classes' failed.  at /Users/soumith/miniconda2/conda-bld/pytorch_1532623076075/work/aten/src/THNN/generic/ClassNLLCriterion.c:93

Output: In your first configuration, you have configured self.fc1 incorrectly. You need the input to be of dimensions 32 * 28 * 28 instead of 32 * 3 * 3 as your images are 32 * 32 and kernel and stride are 3 and 1 respectively. See this video for a simpler explanation. Try adjusting your second configuration yourself now, if you can't, comment below.


Instruction: How to convert a pytorch tensor of ints to a tensor of booleans?

Input:
I would like to cast a tensor of ints to a tensor of booleans.

Specifically I would like to be able to have a function which transforms tensor([0,10,0,16]) to tensor([0,1,0,1])

This is trivial in Tensorflow by just using tf.cast(x,tf.bool).

I want the cast to change all ints greater than 0 to a 1 and all ints equal to 0 to a 0. This is the equivalent of !! in most languages.

Since pytorch does not seem to have a dedicated boolean type to cast to, what is the best approach here?

Edit: I am looking for a vectorized solution opposed to looping through each element.

Output:
What you're looking for is to generate a boolean mask for the given integer tensor. For this, you can simply check for the condition: "whether the values in the tensor are greater than 0" using simple comparison operator (>) or using torch.gt(), which would then give us the desired result.
# input tensor
In [76]: t
Out[76]: tensor([ 0, 10,  0, 16])

# generate the needed boolean mask
In [78]: t > 0
Out[78]: tensor([0, 1, 0, 1], dtype=torch.uint8)

# sanity check
In [93]: mask = t > 0

In [94]: mask.type()
Out[94]: 'torch.ByteTensor'

Note: In PyTorch version 1.4+, the above operation would return 'torch.BoolTensor'
In [9]: t > 0
Out[9]: tensor([False,  True, False,  True])

# alternatively, use `torch.gt()` API
In [11]: torch.gt(t, 0)
Out[11]: tensor([False,  True, False,  True])

If you indeed want single bits (either 0s or 1s), cast it using:
In [14]: (t > 0).type(torch.uint8)
Out[14]: tensor([0, 1, 0, 1], dtype=torch.uint8)

# alternatively, use `torch.gt()` API
In [15]: torch.gt(t, 0).int()
Out[15]: tensor([0, 1, 0, 1], dtype=torch.int32)

The reason for this change has been discussed in this feature-request issue: issues/4764 - Introduce torch.BoolTensor ...

TL;DR: Simple one liner
t.bool().int()


Instruction: pytorch, How can i make same size of tensor model(x) and answer(x)?

Input:
I'm try to make a simple linear model to predict parameters of formula.

y = 3*x1 + x2 - 2*x3

Unfortunately, there are some problem when i try to compute loss.

def answer(x):
return 3 * x[:,0] + x[:,1] - 2 * x[:,2]

def loss_f(x):

y = answer(x)
y_hat = model(x)
loss = ((y - y_hat).pow(2)).sum() / x.size(0)

return loss

When i set batch_size = 3, the size of each result is different

x = torch.randn(3,3)
answer(x)
tensor([ 2.0201, -3.8354,  2.0059])

model(x)
tensor([[ 0.2085],
    [-0.0670],
    [-1.3635]], grad_fn=<ThAddmmBackward>)

answer(x.data).size()
torch.Size([3])
model(x.data).size()
torch.Size([3, 1])

I think the broadcast applied automatically.

loss = ((y - y_hat).pow(2)).sum() / x.size(0)

How can i make same size of two tensors? Thanks
This is my code

import torch
import torch.nn as nn
import torch.optim as optim

class model(nn.Module):

    def __init__(self, input_size, output_size):
        super(model, self).__init__()

        self.linear = nn.Linear(input_size, output_size)

        def forward(self, x):

        y = self.linear(x)

        return y

model = model(3,1)
optimizer = optim.SGD(model.parameters(), lr = 0.001, momentum=0.1)

print('Parameters : ')
for p in model.parameters():
    print(p)

print('')
print('Optimizer : ')
print(optimizer)

def generate_data(batch_size):
    x = torch.randn(batch_size, 3)

    return x

def answer(x):

    return 3 * x[:,0] + x[:,1] - 2 * x[:,2]

def loss_f(x):

    y = answer(x)
    y_hat = model(x)

    loss = ((y - y_hat).pow(2)).sum() / x.size(0)

    return loss

x = torch.randn(3,3)
print(x)
x = torch.FloatTensor(x)

batch_size = 3
epoch_n = 1000
iter_n = 100

for epoch in range(epoch_n):
    avg_loss = 0

    for i in range(iter_n):
        x = torch.randn(batch_size, 3)

        optimizer.zero_grad()
        loss = loss_f(x.data)
        loss.backward()
        optimizer.step()

        avg_loss += loss

    avg_loss = avg_loss / iter_n

    x_valid = torch.FloatTensor([[1,2,3]])
    y_valid = answer(x_valid)

    model.eval()
    y_hat = model(x_valid)
    model.train()

    print(avg_loss, y_valid.data[0], y_hat.data[0])

    if avg_loss < 0.001:
        break

Output:
You can use Tensor.view

https://pytorch.org/docs/stable/tensors.html#torch.Tensor.view

So something like

answer(x.data).view(-1, 1)

should do the trick.


Instruction: Get single random example from PyTorch DataLoader

Input:
How do I get a single random example from a PyTorch DataLoader?
If my DataLoader gives minbatches of multiple images and labels, how do I get a single random image and label?
Note that I don't want a single image and label per minibatch, I want a total of one example.

Output:
If you want to choose specific images from your Trainloader/Testloader, you should check out the Subset function from master:
Here's an example how to use it:
testset = ImageFolderWithPaths(root="path/to/your/Image_Data/Test/", transform=transform)
subset_indices = [0] # select your indices here as a list
subset = torch.utils.data.Subset(testset, subset_indices)
testloader_subset = torch.utils.data.DataLoader(subset, batch_size=1, num_workers=0, shuffle=False)

This way you can use exactly one image and label. However, you can of course use more than just one index in your subset_indices.
If you want to use a specific image from your DataFolder, you can use dataset.sample and build a dictionary to get the index of the image you want to use.


Instruction: pytorch Crossentropy error in simple example of NN

Input:
H1, I am try to make NN model that satisfy simple formula.
y = X1^2 + X2^2

But when i use CrossEntropyLoss for loss function, i get two different error message.
First, when i set code like this

x = torch.randn(batch_size, 2)
y_hat = model(x)
y = answer(x).long()

optimizer.zero_grad()
loss = loss_func(y_hat, y)
loss.backward()
optimizer.step()

i get this message

RuntimeError: Assertion `cur_target >= 0 && cur_target < n_classes' failed.  at
c:\programdata\miniconda3\conda-bld\pytorch_1533090623466\work\aten\src\thnn\generic/Cl

assNLLCriterion.c:93

Second, I change code like this

x = torch.randn(batch_size, 2)
y_hat = model(x)
y = answer(x).long().view(batch_size,1,1)

optimizer.zero_grad()
loss = loss_func(y_hat, y)
loss.backward()
optimizer.step()

then i get message like

RuntimeError: multi-target not supported at c:\programdata\miniconda3\conda-bld\pytorch_1533090623466\work\aten\src\thnn\generic/ClassNLLCriterion.c:21

How can i solve this problem? Thanks.(sorry for my English)
This is my code

import torch
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F

def answer(x):

    y = x[:,0].pow(2) + x[:,1].pow(2)

    return y

class Model(nn.Module):

    def __init__(self, input_size, output_size):
        super(Model, self).__init__()

        self.linear1 = nn.Linear(input_size, 10)
        self.linear2 = nn.Linear(10, 1)

    def forward(self, x):

        y = F.relu(self.linear1(x))
        y = F.relu(self.linear2(y))

        return y

model = Model(2,1)
print(model, '\n')

loss_func = nn.CrossEntropyLoss()
optimizer = optim.SGD(model.parameters(), lr = 0.001)

batch_size = 3
epoch_n = 100
iter_n = 100

for epoch in range(epoch_n):
    loss_avg = 0

    for i in range(iter_n):

        x = torch.randn(batch_size, 2)
        y_hat = model(x)
        y = answer(x).long().view(batch_size,1,1)

        optimizer.zero_grad()
        loss = loss_func(y_hat, y)
        loss.backward()
        optimizer.step()

        loss_avg += loss

    loss_avg = loss_avg / iter_n

    if epoch % 10 == 0:
        print(loss_avg)

    if loss_avg < 0.001:
        break

Can i make those dataset using dataloader in pytorch? Thanks for your help.

Output: You are using the wrong loss function. CrossEntropyLoss is used for classification problems generally wheread your problem is that of regression. So you should use losses which are meant for regression like tasks like Mean Squared Error Loss, L1 Loss etc. Take a look at this, this, this and this.


Instruction: Most efficient way to use a large data set for PyTorch?

Input:
Perhaps this question has been asked before, but I'm having trouble finding relevant info for my situation.

I'm using PyTorch to create a CNN for regression with image data. I don't have a formal, academic programming background, so many of my approaches are ad-hoc and just terribly inefficient. May times I can go back through my code and clean things up later because the inefficiency is not so drastic that performance is significantly affected. However, in this case, my method for using the image data takes a long time, uses a lot of memory, and it is done every time I want to test a change in the model.

What I've done is essentially loaded the image data into numpy arrays, saved those arrays in an .npy file, and then when I want to use said data for the model I import all of the data in that file. I don't think the data set is really THAT large, as it is comprised of 5000, 3 color channel images of size 64x64. Yet my memory usage shoots up to 70%-80% (out of 16gb) when it is being loaded, and it takes 20-30 seconds to load in every time.

My guess is that I'm being dumb about the way I'm loading it in, but frankly I'm not sure what the standard is. Should I, in some way, put the image data somewhere before I need it, or should the data be loaded directly from the image files? And in either case, what is the best, most efficient way to do that, independent of file structure?

I would really appreciate any help on this.

Output:
Here is a concrete example to demonstrate what I meant. This assumes that you've already dumped the images into an hdf5 file (train_images.hdf5) using h5py.
import h5py
hf = h5py.File('train_images.hdf5', 'r')

group_key = list(hf.keys())[0]
ds = hf[group_key]

# load only one example
x = ds[0]

# load a subset, slice (n examples)
arr = ds[:n]

# should load the whole dataset into memory.
# this should be avoided
arr = ds[:]

In simple terms, ds can now be used as an iterator which gives images on the fly (i.e. it doesn't load anything in memory). This should make the whole run time blazing fast.
for idx, img in enumerate(ds):
   # do something with `img`


Instruction: How to get two scalars on same chart with tensorboardX?

Input:
The docs seem to indicate that add_custom_scalars_multilinechart does it but it is not working.

Have something like this:

from tensorboardX import SummaryWriter
writer = SummaryWriter(comment='test')

writer.add_custom_scalars_multilinechart(['loss/train', 'loss/test'], title='losses')

for blahblah:
    ...
    writer.add_scalar('loss/train', loss.item(), epoch)
    writer.add_scalar('loss/test', loss_test.item(), epoch)

Output:
Plot two scalars on same chart with tensorboardX:
from tensorboardX import SummaryWriter

Create two summaryWriter for two scalars
writer_train = SummaryWriter('runs/train_0')
writer_test = SummaryWriter('runs/test_0')

Add scalars instances to the summaryWriter respective; they must have same tag, e.g.: "LOSS"
for data in loop:
    writer_train.add_scalar('LOSS', loss.data.item(), idx)
    writer_test.add_scalar('LOSS', loss_test.data.item(), idx)

For working code, please visit github: Examples with tensorboardX
(See more_plots_one_chat.py)
Tutorial: TensorboardX


Instruction: How to connect the input to the output directly using single fully connected layer in PyTorch?

Input:
I am new to deep learning and cnn and trying to get familiar with that field using CIFAR10 tutorial code from PyTorch website. So, in that code I was playing with removing/adding layers to better understand the effect of them and I tried to connect the input(which is the initial data with the batch of 4 images) to the output directly with using only single fully connected layer. I know that does not make much sense, but I do it only for the sake of experiment. So, when I tried to do it, I faced with some errors, which are as follows:

First, here is the code snippet:

########################################################################
# 2. Define a Convolution Neural Network
# ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
# Copy the neural network from the Neural Networks section before and modify it to
# take 3-channel images (instead of 1-channel images as it was defined).

import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        #self.conv1 = nn.Conv2d(3, 6, 5)
        #self.pool = nn.MaxPool2d(2, 2)
        #self.conv2 = nn.Conv2d(6, 16, 5)
        #self.fc1 = nn.Linear(16 * 5 * 5, 120)
        #self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(768 * 4 * 4, 10)

    def forward(self, x):
        #x = self.pool(F.relu(self.conv1(x)))
        #x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 768 * 4 * 4)
        #x = F.relu(self.fc1(x))
        #x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

net = Net()

#######################################################################
# 3. Define a Loss function and optimizer
# ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
# Let's use a Classification Cross-Entropy loss and SGD with momentum.

import torch.optim as optim

criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)

########################################################################
# 4. Train the network
# ^^^^^^^^^^^^^^^^^^^^
#
# This is when things start to get interesting.
# We simply have to loop over our data iterator, and feed the inputs to the
# network and optimize.

for epoch in range(4):  # loop over the dataset multiple times

    running_loss = 0.0
    for i, data in enumerate(trainloader, 0):
        # get the inputs
        inputs, labels = data

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = net(inputs)
        print(len(outputs))
        print(len(labels))
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()
        if i % 2000 == 1999:    # print every 2000 mini-batches
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 2000))
            running_loss = 0.0

print('Finished Training')

So, when I run the code, I get the following error:

Traceback (most recent call last):
      File "C:\Users\Andrey\Desktop\Machine_learning_Danila\Homework 3\cifar10_tutorial1.py", line 180, in <module>
        loss = criterion(outputs, labels)
      File "C:\Program Files\Python36\lib\site-packages\torch\nn\modules\module.py", line 477, in __call__
        result = self.forward(*input, **kwargs)
      File "C:\Program Files\Python36\lib\site-packages\torch\nn\modules\loss.py", line 862, in forward
        ignore_index=self.ignore_index, reduction=self.reduction)
      File "C:\Program Files\Python36\lib\site-packages\torch\nn\functional.py", line 1550, in cross_entropy
        return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
      File "C:\Program Files\Python36\lib\site-packages\torch\nn\functional.py", line 1405, in nll_loss
        .format(input.size(0), target.size(0)))
    ValueError: Expected input batch_size (1) to match target batch_size (4).

I was trying to check the length of x and it turns out, that it is 4 initially but it becomes 1 after the line

x = x.view(-1, 768 * 4 * 4)

I think my numbers are correct, but it seems like I am having only 1 tensor instead of 4 as I supposed to have, and I feel like that is what causes that error.
I am wondering, why is that and what is the best way to fix that?
Also, what would be the best optimal number for output dimension output in  nn.Linear(Fully connected Layer) in this case?

Output:
There are two obvious errors in your modified code (from the official ones from PyTorch webpage). First,

torch.nn.Linear(in_features, out_features)

is the correct syntax. But, you're passing 768 * 4 * 4 as in_features. This is 4 times the actual number of neurons (pixels) in one CIFAR10 image (32*32*3 = 3072).

The second bug is related to the first one. When you prepare your inputs tensor,

# forward + backward + optimize;
# `inputs` should be a tensor of shape [batch_size, input_shape]
        outputs = net(inputs)

you should pass it as a tensor of shape [batch_size, input_size], which according to your requirement is [4, 3072] since you want to use a batch size of 4. This is where you should provide the batch dimension; Not in nn.Linear which is what you're currently doing and that is causing the error.

Finally, you should also fix the line in forward method. Change the below line

x = x.view(-1, 768 * 4 * 4)

to

x = x.view(-1, 32*32*3)

Fixing these bugs should fix your errors.

Having said that I'm unsure whether this would actually work, in a conceptual sense. Because this is a simple linear transformation (i.e. an affine transformation without any non-linearity). The data points (which correspond to images in CIFAR10) would most probably be not linearly separable, in this 3072 dimensional space (manifold). Hence, the accuracy would be drastically poor. Thus it is advisable to add at least a hidden layer with non-linearity such as ReLU.


Instruction: Pytorch - TypeError: 'torch.Size' object cannot be interpreted as an integer

Input:
Hi I am training a PyTorch model and occurred this error:

----> 5     for i, data in enumerate(trainloader, 0):

TypeError: 'torch.Size' object cannot be interpreted as an integer

Not sure what this error means.

You can find my code here :

model.train()
for epoch in range(10):
    running_loss = 0

    for i, data in enumerate(trainloader, 0):

        inputs, labels = data

        optimizer.zero_grad()

        outputs = model(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        if i % 2000 == 0:
          print (loss.item())
        running_loss += loss.item()
        if i % 1000 == 0:
            print ('[%d, %5d] loss: %.3f' % (epoch, i, running_loss/ 1000))
            running_loss = 0

torch.save(model, 'FeatureNet.pkl')

Update

This is the codeblock for DataLoader. I am using a customized dataloader and datasets, which x are pictures with size (1025, 16) and y are one-hot encoded vectors for classification.

x_train.shape = (1100, 1025, 16)

y_train.shape = (1100, 10)

clean_dir = '/home/tk/Documents/clean/'
mix_dir = '/home/tk/Documents/mix/'
clean_label_dir = '/home/tk/Documents/clean_labels/'
mix_label_dir = '/home/tk/Documents/mix_labels/'

class MSourceDataSet(Dataset):

    def __init__(self, clean_dir, mix_dir, clean_label_dir, mix_label_dir):

        with open(clean_dir + 'clean0.json') as f:
            clean0 = torch.Tensor(json.load(f))

        with open(mix_dir + 'mix0.json') as f:
            mix0 = torch.Tensor(json.load(f))

        with open(clean_label_dir + 'clean_label0.json') as f:
            clean_label0 = torch.Tensor(json.load(f))

        with open(mix_label_dir + 'mix_label0.json') as f:
            mix_label0 = torch.Tensor(json.load(f))

        self.spec = torch.cat([clean0, mix0], 0)
        self.label = torch.cat([clean_label0, mix_label0], 0)

    def __len__(self):
        return self.spec.shape

    def __getitem__(self, index):

        spec = self.spec[index]
        label = self.label[index]
        return spec, label

getitem

a, b = trainset.__getitem__(1000)
print (a.shape)
print (b.shape)

a.shape = torch.Size([1025, 16]);
b.shape = torch.Size([10])

Error message

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-9-3bd71e5c00e1> in <module>()
      3     running_loss = 0
      4
----> 5     for i, data in enumerate(trainloader, 0):
      6
      7         inputs, labels = data

~/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py in __next__(self)
    311     def __next__(self):
    312         if self.num_workers == 0:  # same-process loading
--> 313             indices = next(self.sample_iter)  # may raise StopIteration
    314             batch = self.collate_fn([self.dataset[i] for i in indices])
    315             if self.pin_memory:

~/anaconda3/lib/python3.7/site-packages/torch/utils/data/sampler.py in __iter__(self)
    136     def __iter__(self):
    137         batch = []
--> 138         for idx in self.sampler:
    139             batch.append(idx)
    140             if len(batch) == self.batch_size:

~/anaconda3/lib/python3.7/site-packages/torch/utils/data/sampler.py in __iter__(self)
     32
     33     def __iter__(self):
---> 34         return iter(range(len(self.data_source)))
     35
     36     def __len__(self):

TypeError: 'torch.Size' object cannot be interpreted as an integer

Output:
Your problem is the __len__ function. You cannot use the shape as return value.

Here is an example for illustration:

import torch
class Foo:
    def __init__(self, data):
        self.data = data
    def __len__(self):
        return self.data.shape

myFoo = Foo(data=torch.rand(10, 20))
print(len(myFoo))

Will raise exactly the same error:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-285-e97aace2f622> in <module>
      7
      8 myFoo = Foo(data=torch.rand(10, 20))
----> 9 print(len(myFoo))

TypeError: 'torch.Size' object cannot be interpreted as an integer

Since shape represents a torch.Size tuple:

print(myFoo.data.shape)

Output:

torch.Size([10, 20])

So you have to decide which dimension you want to hand over to __len__, for example the first dimension:

import torch
class Foo:
    def __init__(self, data):
        self.data = data
    def __len__(self):
        return self.data.shape[0] # choosing first dimension for len

myFoo = Foo(data=torch.rand(10, 20))
print(len(myFoo))
# prints 10

Works fine and returns 10. Of course you can also choose any other dimension of your input, but you have to choose one.

So in your code of your MSourceDataSet you have to change your __len__ function to for example:

def __len__(self):
    return self.spec.shape[0] # as said of course you can also choose other dimensions

This should solve your problem.


Instruction: Train SqueezeNet model using MNIST dataset Pytorch

Input:
I want to train SqueezeNet 1.1 model using MNIST dataset instead of ImageNet dataset.
Can i have the same model as torchvision.models.squeezenet?
Thanks!

Output:
TorchVision provides only ImageNet data pretrained model for the SqueezeNet architecture. However, you can train your own model using MNIST dataset by taking only the model (but not the pre-trained one) from torchvision.models.

In [10]: import torchvision as tv

# get the model architecture only; ignore `pretrained` flag
In [11]: squeezenet11 = tv.models.squeezenet1_1()

In [12]: squeezenet11.training
Out[12]: True

Now, you can use this architecture to train a model on MNIST data, which should not take too long.

One modification to keep in mind is to update the number of classes which is 10 for MNIST. Specifically, the 1000 should be changed to 10, and the kernel and stride accordingly.

  (classifier): Sequential(
    (0): Dropout(p=0.5)
    (1): Conv2d(512, 1000, kernel_size=(1, 1), stride=(1, 1))
    (2): ReLU(inplace)
    (3): AvgPool2d(kernel_size=13, stride=1, padding=0)
  )

Here's the relevant explanation: finetuning_torchvision_models-squeezenet