Instruction: Pytorch setting part of the training value into zero

Input:
When I am training network, I want to assign a specific part of the output into zero. For example, temp = nn.conv2d(3,6) in 3-channel / 16 batch size
would have tensor value like this.

temp = tensor(16,6,36,36)

I want half of the channel value into zero.

temp[:,:3,:,:] = 0

But I got this error.

RuntimeError: one of the variables needed for gradient computation
has been modified by an inplace operation: [torch.cuda.FloatTensor
[1, 128, 15, 15]], which is output 0 of ReluBackward1, is at version
2; expected version 1 instead. Hint: the backtrace further above
shows the operation that failed to compute its gradient. The variable
in question was changed in there or anywhere later. Good luck!

How can I approach this problem?

I've tried with this code

def weight_init(self,x,label):
    if label.data[0]:
        x[:,:64,:,:] =0
        return x
    else:
        x[:,64:,:,:] =0
        return x

Output:
You are doing operation on tensor, and this operation will be tracked for the future gradient computation. you just need to temporally switch off the changes tracking with torch.no_grad():

Code:

def weight_init(self,x,label):
    with torch.no_grad():
        if label.data[0]:
            x[:,:64,:,:] =0
            return x
        else:
            x[:,64:,:,:] =0
            return x


Instruction: Multi-GPU training of AllenNLP coreference resolution

Input:
I'm trying to replicate (or come close) to the results obtained by the End-to-end Neural Coreference Resolution paper on the CoNLL-2012 shared task. I intend to do some enhancements on top of this, so I decided to use AllenNLP's CoreferenceResolver. This is how I'm initialising & training the model:

import torch
from allennlp.common import Params
from allennlp.data import Vocabulary
from allennlp.data.dataset_readers import ConllCorefReader
from allennlp.data.dataset_readers.dataset_utils import Ontonotes
from allennlp.data.iterators import BasicIterator, MultiprocessIterator
from allennlp.data.token_indexers import SingleIdTokenIndexer, TokenCharactersIndexer
from allennlp.models import CoreferenceResolver
from allennlp.modules import Embedding, FeedForward
from allennlp.modules.seq2seq_encoders import PytorchSeq2SeqWrapper
from allennlp.modules.seq2vec_encoders import CnnEncoder
from allennlp.modules.text_field_embedders import BasicTextFieldEmbedder
from allennlp.modules.token_embedders import TokenCharactersEncoder
from allennlp.training import Trainer
from allennlp.training.learning_rate_schedulers import LearningRateScheduler
from torch.nn import LSTM, ReLU
from torch.optim import Adam

def read_data(directory_path):
    data = []
    for file_path in Ontonotes().dataset_path_iterator(directory_path):
        data += dataset_reader.read(file_path)
    return data

INPUT_FILE_PATH_TEMPLATE = "data/CoNLL-2012/v4/data/%s"
dataset_reader = ConllCorefReader(10, {"tokens": SingleIdTokenIndexer(),
                                       "token_characters": TokenCharactersIndexer()})
training_data = read_data(INPUT_FILE_PATH_TEMPLATE % "train")
validation_data = read_data(INPUT_FILE_PATH_TEMPLATE % "development")

vocabulary = Vocabulary.from_instances(training_data + validation_data)
model = CoreferenceResolver(vocab=vocabulary,
                            text_field_embedder=BasicTextFieldEmbedder({"tokens": Embedding.from_params(vocabulary, Params({"embedding_dim": embeddings_dimension, "pretrained_file": "glove.840B.300d.txt"})),
                                                                        "token_characters": TokenCharactersEncoder(embedding=Embedding(num_embeddings=vocabulary.get_vocab_size("token_characters"), embedding_dim=8, vocab_namespace="token_characters"),
                                                                                encoder=CnnEncoder(embedding_dim=8, num_filters=50, ngram_filter_sizes=(3, 4, 5), output_dim=100))}),
                            context_layer=PytorchSeq2SeqWrapper(LSTM(input_size=400, hidden_size=200, num_layers=1, dropout=0.2, bidirectional=True, batch_first=True)),
                            mention_feedforward=FeedForward(input_dim=1220, num_layers=2, hidden_dims=[150, 150], activations=[ReLU(), ReLU()], dropout=[0.2, 0.2]),
                            antecedent_feedforward=FeedForward(input_dim=3680, num_layers=2, hidden_dims=[150, 150], activations=[ReLU(), ReLU()], dropout=[0.2, 0.2]),
                            feature_size=20,
                            max_span_width=10,
                            spans_per_word=0.4,
                            max_antecedents=250,
                            lexical_dropout=0.5)

if torch.cuda.is_available():
    cuda_device = 0
    model = model.cuda(cuda_device)
else:
    cuda_device = -1

iterator = BasicIterator(batch_size=1)
iterator.index_with(vocabulary)
optimiser = Adam(model.parameters(), weight_decay=0.1)
Trainer(model=model,
        train_dataset=training_data,
        validation_dataset=validation_data,
        optimizer=optimiser,
        learning_rate_scheduler=LearningRateScheduler.from_params(optimiser, Params({"type": "step", "step_size": 100})),
        iterator=iterator,
        num_epochs=150,
        patience=1,
        cuda_device=cuda_device).train()

After reading the data I've trained the model but ran out of GPU memory: RuntimeError: CUDA out of memory. Tried to allocate 4.43 GiB (GPU 0; 11.17 GiB total capacity; 3.96 GiB already allocated; 3.40 GiB free; 3.47 GiB cached). Therefore, I attempted to make use of multiple GPUs to train this model. I'm making use of Tesla K80s (which have 12GiB memory).

I've tried making use of AllenNLP's MultiprocessIterator, by itialising the iterator as MultiprocessIterator(BasicIterator(batch_size=1), num_workers=torch.cuda.device_count()). However, only 1 GPU is being used (by monitoring the memory usage through the nvidia-smi command) & got the error below. I also tried fiddling with its parameters (increasing num_workers or decreasing output_queue_size) & the ulimit (as mentioned by this PyTorch issue) to no avail.

Process Process-3:
Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/home/user/.local/lib/python3.6/site-packages/allennlp/data/iterators/multiprocess_iterator.py", line 32, in _create_tensor_dicts
    output_queue.put(tensor_dict)
  File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/home/user/.local/lib/python3.6/site-packages/allennlp/data/iterators/multiprocess_iterator.py", line 32, in _create_tensor_dicts
    output_queue.put(tensor_dict)
  File "<string>", line 2, in put
  File "<string>", line 2, in put
  File "/usr/lib/python3.6/multiprocessing/managers.py", line 772, in _callmethod
    raise convert_to_error(kind, result)
  File "/usr/lib/python3.6/multiprocessing/managers.py", line 772, in _callmethod
    raise convert_to_error(kind, result)
multiprocessing.managers.RemoteError:
---------------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/managers.py", line 228, in serve_client
    request = recv()
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 251, in recv
    return _ForkingPickler.loads(buf.getbuffer())
  File "/home/user/.local/lib/python3.6/site-packages/torch/multiprocessing/reductions.py", line 276, in rebuild_storage_fd
    fd = df.detach()
  File "/usr/lib/python3.6/multiprocessing/resource_sharer.py", line 58, in detach
    return reduction.recv_handle(conn)
  File "/usr/lib/python3.6/multiprocessing/reduction.py", line 182, in recv_handle
    return recvfds(s, 1)[0]
  File "/usr/lib/python3.6/multiprocessing/reduction.py", line 161, in recvfds
    len(ancdata))
RuntimeError: received 0 items of ancdata
---------------------------------------------------------------------------

I also tried achieving this through PyTorch's DataParallel, by wrapping the model's context_layer, mention_feedforward, antecedent_feedforward with a custom DataParallelWrapper (to provide compatibility with the AllenNLP-assumed class functions). Still, only 1 GPU is used & it eventually runs out of memory as before.

class DataParallelWrapper(DataParallel):
    def __init__(self, module):
        super().__init__(module)

    def get_output_dim(self):
        return self.module.get_output_dim()

    def get_input_dim(self):
        return self.module.get_input_dim()

    def forward(self, *inputs):
        return self.module.forward(inputs)

Output:
After some digging through the code I found out that AllenNLP does this under the hood directly through its Trainer. The cuda_device can either be a single int (in the case of single-processing) or a list of ints (in the case of multi-processing):

  cuda_device : Union[int, List[int]], optional (default = -1)
  An integer or list of integers specifying the CUDA device(s) to use. If -1, the CPU is used.

So all GPU devices needed should be passed on instead:

if torch.cuda.is_available():
    cuda_device = list(range(torch.cuda.device_count()))
    model = model.cuda(cuda_device[0])
else:
    cuda_device = -1

Note that the model still has to be manually moved to the GPU (via model.cuda(...)), as it would otherwise try to use multiple CPUs instead.


Instruction: How to install nvidia apex on Google Colab

Input:
what I did is follow the instruction on the official github site

!git clone https://github.com/NVIDIA/apex
!cd apex
!pip install -v --no-cache-dir ./

it gives me the error:

ERROR: Directory './' is not installable. Neither 'setup.py' nor 'pyproject.toml' found.
Exception information:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/pip/_internal/cli/base_command.py", line 178, in main
    status = self.run(options, args)
  File "/usr/local/lib/python3.6/dist-packages/pip/_internal/commands/install.py", line 326, in run
    self.name, wheel_cache
  File "/usr/local/lib/python3.6/dist-packages/pip/_internal/cli/base_command.py", line 268, in populate_requirement_set
    wheel_cache=wheel_cache
  File "/usr/local/lib/python3.6/dist-packages/pip/_internal/req/constructors.py", line 248, in install_req_from_line
    "nor 'pyproject.toml' found." % name
pip._internal.exceptions.InstallationError: Directory './' is not installable. Neither 'setup.py' nor 'pyproject.toml' found.

Output:
(wanted to just add a comment but I don't have enough reputation...)

it works for me but the cd is actually not required. Also, I needed the two global options as suggested here: https://github.com/NVIDIA/apex/issues/86

%%writefile setup.sh

git clone https://github.com/NVIDIA/apex
pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./apex

then

!sh setup.sh


Instruction: Derivative in both arguments of torch.nn.BCELoss()

Input:
When using a torch.nn.BCELoss() on two arguments that are both results of some earlier computation, I get some curious error, which this question is about:

RuntimeError: the derivative for 'target' is not implemented

The MCVE is as follows:

import torch
import torch.nn.functional as F

net1 = torch.nn.Linear(1,1)
net2 = torch.nn.Linear(1,1)
loss_fcn = torch.nn.BCELoss()

x = torch.zeros((1,1))

y = F.sigmoid(net1(x)) #make sure y is in range (0,1)
z = F.sigmoid(net2(y)) #make sure z is in range (0,1)

loss = loss_fcn(z, y) #works if we replace y with y.detach()

loss.backward()

It turns out if we call .detach() on y the error disappears. But this results in a different computation, now in the .backward()-pass, the gradients with respect to the second argument of the BCELoss will not be computed.

Can anyone explain what I'm doing wrong in this case? As far as I know all pytorch modules in torch.nn should support computing gradients. And this error message seems to tell me that the derivative is not implemented for y, which is somehow strange, as you can compute the gradient of y, but not of y.detach() which seems to be contradictory.

Output: It seems I misunderstood the error message. It is not y that doesn't allow the computation for gradients, it is BCELoss() that doesn't have the ability to compute gradients with respect to the second argument. A similar problem was discussed here.


Instruction: I can't load my model because I can't put a PosixPath

Input:
I'm setting up a script and I need to use some functions from fast-ai package. The fact is that I'm on Windows and when I define my paths, the function from fast-ai named load_learner can't load the model.

I've tried to change the function into the package as:

state = pickle.load(open(str(path) + '/' + str(fname), 'rb'))

instead of:

state = pickle.load(open(path/fname, 'rb'))

but I obtain this error:

 File "lib\site-packages\fastai\basic_train.py", line 462, in load_learner
    state = pickle.load(open(path/fname, 'rb'))
  File "\lib\pathlib.py", line 1006, in __new__
    % (cls.__name__,))
NotImplementedError: cannot instantiate 'PosixPath' on your system

My paths are defined as:

folder_path = './models/model1'
fname = 'model.pkl'

and I call the function as:
model = load_learner(folder_path, fname)

How can I use Windows paths in this function?

UPDATE 1

The answer posted was correct only on Linux. I still have the issue on Windows. I didn't find a way to pass through the PosixPath on Windows. The only solution that I found is to change the internal packages from my modules but it is not a secure way to solve this kind of issue.

Thanks in advance.

Output:
Just redirect PosixPath to WindowsPath.
import pathlib
temp = pathlib.PosixPath
pathlib.PosixPath = pathlib.WindowsPath

I am also loading fastai models and this trick works.


Instruction: How to install torchtext 0.4.0 on conda

Input: The torchtext 0.4.0 library exists (can be downloaded thru pip), but conda install torchtext=0.4.0 will not work. How can I download torchtext to a anaconda environment?

Output:
Create conda env with python:

$ conda create -n torchtext python=3.7
Activate it:

$ conda activate torchtext

torchtext 0.4.0 installation currently is not available via conda. So we will use pip, anyway pip installs torchtext inside your activated conda env.

Let's find torchtex available versions:

(torchtext)$ pip search torchtext
torchtext (0.3.1) - Text utilities and datasets for PyTorch

As you can see there is no 0.4.0.
4. But we can install it from official github repo:

(torchtext)$ pip install git+https://github.com/pytorch/text.git

Verify the installation:

(torchtext)$ conda list

...

torch                1.1.0                    pypi_0    pypi

torchtext            0.4.0                    pypi_0    pypi

tqdm                 4.32.2                   pypi_0    pypi

...

(torchtext)$ python -c "import torchtext; print('torchtext is fine!')"

torchtext is fine!


Instruction: PyTorch Module with attrs cannot get parameter list

Input:
The attr's package somehow ruins pytorch's parameter() method for a module. I am wondering if anyone has any work-arounds or solutions, so that the two packages can seamlessly integrate?

If not, any advice on which github to post the issue to? My instinct would be to post this onto attr's github, but the stack trace is almost entirely relevant to pytorch's codebase.

Python 3.7.3
attrs== 19.1.0
torch==1.1.0.post2
torchvision==0.3.0

import attr
import torch

class RegularModule(torch.nn.Module):
    pass

@attr.s
class AttrsModule(torch.nn.Module):
    pass

module = RegularModule()
print(list(module.parameters()))

module = AttrsModule()
print(list(module.parameters()))

The actual output is:

$python attrs_pytorch.py
[]
Traceback (most recent call last):
  File "attrs_pytorch.py", line 18, in <module>
    print(list(module.parameters()))
  File "/usr/local/anaconda3/envs/bgg/lib/python3.7/site-packages/torch/nn/modules/module.py", line 814, in parameters
    for name, param in self.named_parameters(recurse=recurse):
  File "/usr/local/anaconda3/envs/bgg/lib/python3.7/site-packages/torch/nn/modules/module.py", line 840, in named_parameters
    for elem in gen:
  File "/usr/local/anaconda3/envs/bgg/lib/python3.7/site-packages/torch/nn/modules/module.py", line 784, in _named_members
    for module_prefix, module in modules:
  File "/usr/local/anaconda3/envs/bgg/lib/python3.7/site-packages/torch/nn/modules/module.py", line 975, in named_modules
    if self not in memo:
TypeError: unhashable type: 'AttrsModule'

The expected output is:

$python attrs_pytorch.py
[]
[]

Output:
You may get it to work with one workaround and using dataclasses (which you should, as it's in standard Python library since 3.7 which you are apparently using). Though I think simple __init__ is more readable. One could do something similar using attrs library (disabling hashing), I just prefer the solution using standard libraries if possible.

The reason (if you manage to handle hashing related errors) is that you are calling torch.nn.Module.__init__() which generates _parameters attribute and other framework-specific data.

First solving hashing with dataclasses:

@dataclasses.dataclass(eq=False)
class AttrsModule(torch.nn.Module):
    pass

This solves hashing issues as, as stated by the documentation, section about hash and eq:

  By default, dataclass() will not implicitly add a hash() method
  unless it is safe to do so.

which is needed by PyTorch so the model can be used in C++ backed (correct me if I'm wrong), furthermore:

  If eq is false, hash() will be left untouched meaning the
  hash() method of the superclass will be used (if the superclass is object, this means it will fall back to id-based hashing).

So you are fine using torch.nn.Module __hash__ function (refer to documentation of dataclasses if any further errors arise).

This leaves you with the error:

AttributeError: 'AttrsModule' object has no attribute '_parameters'

Because torch.nn.Module constructor is not called. Quick and dirty fix:

@dataclasses.dataclass(eq=False)
class AttrsModule(torch.nn.Module):
    def __post_init__(self):
        super().__init__()

__post_init__ is a function called after __init__ (who would of guessed), where you can initialize torch-specific parameters.

Still, I would advise against using those two modules together. For example, you are destroying PyTorch's __repr__ using your code, so repr=False should be passed to the dataclasses.dataclass constructor, which gives this final code (obvious collisions between libraries eliminated I hope):

import dataclasses

import torch

class RegularModule(torch.nn.Module):
    pass

@dataclasses.dataclass(eq=False, repr=False)
class AttrsModule(torch.nn.Module):
    def __post_init__(self):
        super().__init__()

module = RegularModule()
print(list(module.parameters()))

module = AttrsModule()
print(list(module.parameters()))

For more on attrs please see hynek answer and his blog post.


Instruction: Normalization (Feature scaling) of Point Cloud Dataset

Input:
I have point cloud data set where single data is represented by N * 3 where N is number of points. Similarly I have "M" number of points clouds in Dataset. The range of these point clouds varies largely. Some have very large values (e.g., in term of 10^6 for all N points) while some has very small values (e.g., in term of 10^1 for all N points). I want to normalize each point cloud. How should i do that?

Q1. Should I normalize(min-max) each point cloud (single point cloud N*3) individually along x, y, z dimension by choosing min and max from this point cloud only. In this scenario, for all "M" point clouds we have different min-max. The same is done for output  point cloud. Please view the image for more understanding https://i.stack.imgur.com/tKauw.jpg

Q2. Or Should I normalize (min-max) all point clouds along x, y, z dimension by choosing min and max (from M * N *3 in x, y, z columns) from this entire data set only. In this scenario, for all "m" point clouds we have same min-max. Please view the image for more understanding  https://i.stack.imgur.com/0HAhn.jpg

Output:
You should use option 1.

The point of normalisation is to standardise the inputs to your network - in the case of min-max normalisation this will map your 3 features (I assume xyz) to the interval [0,1].

Option 2 is undesirable as it standardises the normalisation instead. As the centroids of your point clouds are highly variable, this will increase the difficulty of input discrimination for your model.

You could also consider standardising using variable standard deviation.


Instruction: Combine index and value of tenor to from a new tensor

Input:
I have a  tensor like a = torch.tensor([1,2,0,1,2]). I want to calculate a tensor b which has indices and values of tensor a such that:

b = tensor([ [0,1], [1,2], [2,0], [3,1], [4,2] ]).

Edit: a[i] is >=  0.

Output:
One way of doing this is:

b = torch.IntTensor(list(zip(range(0, list(a.size())[0], 1), a.numpy())))

Output:

tensor([[0, 1],
        [1, 2],
        [2, 0],
        [3, 1],
        [4, 2]], dtype=torch.int32)

Alternatively, you can also use torch.cat() as below:

a = torch.tensor([1,2,0,1,2])
indices = torch.arange(0, list(a.size())[0])
res = torch.cat([indices.view(-1, 1), a.view(-1, 1)], 1)

Output:

tensor([[0, 1],
        [1, 2],
        [2, 0],
        [3, 1],
        [4, 2]])


Instruction: Why does setting backward(retain_graph=True) use up lot GPU memory?

Input:
I need to backpropagate through my neural network multiple times, so I set backward(retain_graph=True).

However, this is causing

  RuntimeError: CUDA out of memory

I don't understand why this is.

Are the number of variables or weights doubling? Shouldn't the amount of memory used remain the same regardless of how many times backward() is called?

Output:
The source of the issue :

You are right that no matter how many times we call the backward function, the memory should not increase theorically.

Yet your issue is not because of the backpropagation, but the retain_graph variable that you have set to true when calling the backward function.

When you run your network by passing a set of input data, you call the forward function, which will create a "computation graph".
A computation graph is containing all the operations that your network has performed.

Then when you call the backward function, the computation graph saved will "basically" be runned backward to know which weight should be adjusted in which directions (what is called the gradients).
So PyTorch is saving in memory the computation graph in order to call the backward function.

After the backward function has been called and the gradients have been calculated, we free the graph from the memory, as explained in the doc https://pytorch.org/docs/stable/autograd.html :

  retain_graph (bool, optional) – If False, the graph used to compute the grad will be freed. Note that in nearly all cases setting this option to True is not needed and often can be worked around in a much more efficient way. Defaults to the value of create_graph.

Then usually during training we apply the gradients to the network in order to minimise the loss, then we re-run the network, and so we create a new computation graph. Yet we have only one graph in memory at the same time.

The issue :

If you set retain_graph to true when you call the backward function, you will keep in memory the computation graphs of ALL the previous runs of your network.

And since on every run of your network, you create a new computation graph, if you store them all in memory, you can and will eventually run out of memory.

On the first iteration and run of your network, you will have only one graph in memory. Yet on the 10th run of the network, you have 10 graphs in memory. And on the 10000th run you have 10000 in memory. It is not sustainable, and it is understandable why it is not recommended in the docs.

So even if it may seems that the issue is the backpropagation, it is actually the storing of the computation graphs, and since we usually call the the forward and backward function once per iteration or network run, making a confusion is understandable.

Solution :

What you need to do, is find a way to make your network and architecture work without using retain_graph. Using it will make it almost impossible to train your network, since each iteration increase the usage of your memory and decrease the speed of training, and in your case, even cause you to run out of memory.

You did not mention why you need to backpropagate multiple times, yet it is rarely needed, and i do not know of a case where it cannot be "worked around". For example, if you need to access variables or weights of previous runs you could save them inside variables and later access them, instead of trying doing a new backpropagation.

You likely need to backpropagate multiple times for another reason, yet believe as i have been in this situation, there is likely a way to accomplish what you are trying to do without storing the previous computation graphs.

If you want to share why you need to backpropagate multiple times, maybe others and i could help you more.

More about the backward process :

If you want to learn more about the backward process it is called the "Jacobian-vector product". It is a bit complex and is handled by PyTorch. I do not yet fully understand it, yet this ressource seems good as a starting point, as it seems less intimidating than the PyTorch documentation (in term of algebra) : https://mc.ai/how-pytorch-backward-function-works/


Instruction: Why torch.sum() before doing .backward()?

Input:
I can see what this code below from this video is trying to do. But the sum from y=torch.sum(x**2) confuses me. With sum operation, y becomes a tensor with one single value. As I understand .backward() as calculating derivatives, why would we want to use sum and reduce y to one value?

import pytorch
import matplotlib.pyplot as plt
x = torch.linspace(-10.0,10.0,10, requires_grad=True)
Y = x**2
y = torch.sum(x**2)
y.backward()

plt.plot(x.detach().numpy(), Y.detach().numpy(), label="Y")
plt.plot(x.detach().numpy(), x.grad.detach().numpy(), label="derivatives")
plt.legend()

Output:
You can only compute partial derivatives for a scalar function. What backwards() gives you is d loss/d parameter and you expect a single gradient value per parameter/variable.
Had your loss function been a vector function, i.e., mapping from multiple inputs to multiple outputs, you would have ended up with multiple gradients per parameter/variable.

Please see this answer for more information.


Instruction: ValueError: optimizer got an empty parameter list

Input:
I create the following simple linear class:

class Decoder(nn.Module):
    def __init__(self, K, h=()):
        super().__init__()
        h = (K,)+h+(K,)
        self.layers = [nn.Linear(h1,h2) for h1,h2 in zip(h, h[1:])]

    def forward(self, x):
        for layer in self.layers[:-1]:
            x = F.relu(layer(x))
        return self.layers[-1](x)

However, when I try to put the parameters in a optimizer class I get the error ValueError: optimizer got an empty parameter list.

decoder = Decoder(4)
LR = 1e-3
opt = optim.Adam(decoder.parameters(), lr=LR)

Is there something I'm doing obviously wrong with the class definition?

Output:
Since you store your layers in a regular pythonic list inside your Decoder, Pytorch has no way of telling these members of the self.list are actually sub modules. Convert this list into pytorch's nn.ModuleList and your problem will be solved
class Decoder(nn.Module):
    def __init__(self, K, h=()):
        super().__init__()
        h = (K,)+h+(K,)
        self.layers = nn.ModuleList(nn.Linear(h1,h2) for h1,h2 in zip(h, h[1:]))


Instruction: pytorch - loss.backward() and optimizer.step() in eval mode with batch norm layers?

Input:
I have a ResNet-8 network I am using for a project of Domain Adaptation over images, basically I have trained the network over a dataset and now I want to evaluate it over another dataset simulating a real time environment where I try to predict one image at a time, but here comes the fun part:

The way I want to do the evaluation on the target dataset is by doing, for each image, a forward pass in train mode so that the batch norm layers statistics are updated (with torch.no_grad(), since I don't want to update the network parameters then, but only "adapt" the batch norm layers), and then do another forward pass in eval mode to get the actual prediction so that the batch norm layers will use mean and variance based on the whole set of images seen so far (and not only those of that batch, a single image in this case):

optimizer.zero_grad()
model.train()
with torch.no_grad():
  output_train = model(inputs)
model.eval()
output_eval = model(inputs)
loss = criterion(output_eval, targets)

The idea is that I do domain adaptation just by updating the batch norm layers to the new target distribution.

Then after doing this let's say I get an accuracy of 60%.
Now if I add this two other lines I am able to achieve something like 80% accuracy:

loss.backward()
optimizer.step()

Therefore my question is what happens if I do backward() and step() while in eval mode? Because I know about the different behaviour of batch norm and dropout layers between train and eval mode and I know about torch.no_grad() and how gradient are calculated and then parameters updated by the optimizer, but I wasn't able to find any information about my specific problem.

I think that since the model is then set in eval mode, those two line should be useless, but something clearly happens, does this have something to do with the affine parameters of the batch norm layers?

UPDATE: Ok I misunderstood something: eval mode does not block parameters to be updated, it only changes the behaviour of some layers (batch norm and dropout) during the forward pass, am I right? Therefore with those two lines I am actually training the network, hence the better accuracy. Anyway does this change something if batch norm affine is set to true? Are those parameters considered as "normal" parameters to be updated during optimizer.step() or is it different?

Output:
  eval mode does not block parameters to be updated, it only changes the behaviour of some layers (batch norm and dropout) during the forward pass, am I right?

True.

  Therefore with those two lines I am actually training the network, hence the better accuracy. Anyway does this change something if batch norm affine is set to true? Are those parameters considered as "normal" parameters to be updated during optimizer.step() or is it different?

BN parameters are updated during optimizer step. Look:

    if self.affine:
        self.weight = Parameter(torch.Tensor(num_features))
        self.bias = Parameter(torch.Tensor(num_features))
    else:
        self.register_parameter('weight', None)
        self.register_parameter('bias', None)


Instruction: Convolutional Autoencoder in Pytorch for Dummies

Input:
I am here to ask some more general questions about Pytorch and Convolutional Autoencoders.

If I only use Convolutional Layers (FCN), do I even have to care about the input shape? And then how do I choose the number of featuremaps best?
Does a ConvTranspose2d Layer automatically unpool?
Can you spot any errors or unconventional code in my example?

By the way, I want to make a symmetrical Convolutional Autoencoder to colorize black and white images with different image sizes.

    self.encoder = nn.Sequential (
        # conv 1
        nn.Conv2d(in_channels=3, out_channels=512, kernel_size=3, stride=1, padding=1),
        nn.ReLU,
        nn.MaxPool2d(kernel_size=2, stride=2),  # 1/2
        nn.BatchNorm2d(512),

        # conv 2
        nn.Conv2d(in_channels=512, out_channels=256, kernel_size=3, stride=1, padding=1),
        nn.ReLU,
        nn.MaxPool2d(kernel_size=2, stride=2),  # 1/4
        nn.BatchNorm2d(256),

        # conv 3
        nn.Conv2d(in_channels=256, out_channels=128, kernel_size=3, stride=1, padding=1),
        nn.ReLU,
        nn.MaxPool2d(kernel_size=2, stride=2),  # 1/8
        nn.BatchNorm2d(128),

        # conv 4
        nn.Conv2d(in_channels=128, out_channels=64, kernel_size=3, stride=1, padding=1),
        nn.ReLU,
        nn.MaxPool2d(kernel_size=2, stride=2),  #1/16
        nn.BatchNorm2d(64)
    )

    self.encoder = nn.Sequential (
        # conv 5
        nn.ConvTranspose2d(in_channels=64, out_channels=128, kernel_size=3, stride=1, padding=1),
        nn.ReLU,
        nn.BatchNorm2d(128),

        # conv 6
        nn.ConvTranspose2d(in_channels=128, out_channels=256, kernel_size=3, stride=1, padding=1),
        nn.ReLU,
        nn.BatchNorm2d(256),

        # conv 7
        nn.ConvTranspose2d(in_channels=256, out_channels=512, kernel_size=3, stride=1, padding=1),
        nn.ReLU,
        nn.BatchNorm2d(512),

        # conv 8
        nn.ConvTranspose2d(in_channels=512, out_channels=512, kernel_size=3, stride=1, padding=1),
        nn.Softmax()
    )

    def forward(self, x):
        h = x
        h = self.encoder(h)
        h = self.decoder(h)
        return h

Output:
No, you don't need to care about input width and height with a fully convolutional model. But should probably ensure that each downsampling operation in the encoder is matched by a corresponding upsampling operation in the decoder.
I'm not sure what you mean by unpooling. If you mean upsampling (increasing spatial dimensions), then this is what the stride parameter is for. In PyTorch, a transpose convolution with stride=2 will upsample twice. Note, however, that instead of a transpose convolution, many practitioners prefer to use bilinear upsampling followed by a regular convolution. This is one reason why.

If, on the other hand, you mean actual unpooling, then you should look at the documentation of torch.MaxUnpool2d. You need to collect maximal value indices from the MaxPool2d operation and feed them into MaxUnpool2d.

The general consensus seems to be that you should increase the number of feature maps as you downsample. Your code appears to do the reverse. Consecutive powers of 2 seem like a good place to start. It's hard to suggest a better rule of thumb. You probably need to experiment a little.

In other notes, I'm not sure why you apply softmax to the encoder output.


Instruction: pytorch: "multi-target not supported" error message

Input:
So I want to classify some (3, 50, 50) pictures. First I loaded the dataset from the file without a dataloader or batches, it worked. Now, after adding both things I get that error:
RuntimeError: multi-target not supported at /pytorch/aten/src/THCUNN/generic/ClassNLLCriterion.cu:15

I found a lot of answers in the internet, mostly to use target.squeeze(1) but it doesn´t work for me.
My target-batch looks like following:
tensor([[1, 0],
        [1, 0],
        [1, 0],
        [1, 0],
        [1, 0],
        [1, 0],
        [1, 0],
        [1, 0]], device='cuda:0')

Shouldn't that be okay?
Here the full code (notice that Im only creating the structure of the model on which Im going to apply the full and correct dataset afterwards, because I dont have the full data yet, only 32 pictures and no labels, thats why I added torch.tensor([1, 0]) as a placeholder for all labels):
import torch
import torch.utils.data
import torch.nn as nn
import torch.nn.functional as F
import torch.optim
from torch.autograd import Variable

import numpy as np
from PIL import Image

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()

        # model structur:
        self.conv1 = nn.Conv2d(3, 10, kernel_size=(5,5),  stride=(1,1))
        self.conv2 = nn.Conv2d(10, 20, kernel_size=(5,5),  stride=(1,1))            # with mapool: output = 20 * (9,9) feature-maps -> flatten
        self.fc1 = nn.Linear(20*9*9, 250)
        self.fc2 = nn.Linear(250, 100)
        self.fc3 = nn.Linear(100, 2)

    def forward(self, x):
        # conv layers
        x = F.relu(self.conv1(x))   # shape: 1, 10, 46, 46
        x = F.max_pool2d(x, 2, 2)   # shape: 1, 10, 23, 23
        x = F.relu(self.conv2(x))   # shape: 1, 20, 19, 19
        x = F.max_pool2d(x, 2, 2)   # shape: 1, 20, 9, 9

        # flatten to dense layer:
        x = x.view(-1, 20*9*9)

        # dense layers
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        output = F.log_softmax(self.fc3(x), dim=1)
        return output

class Run:
    def __init__(self, epochs, learning_rate, dropout, momentum):
        # load model
        self.model = Model().cuda()

        # hyperparameters:
        self.epochs = epochs
        self.learning_rate = learning_rate
        self.dropout = dropout

    def preporcessing(self):
        dataset_folder = "/media/theodor/hdd/Programming/BWKI/dataset/bilder/"

        dataset = []
        for i in range(0, 35):
            sample_image = Image.open(dataset_folder + str(i) + ".png")
            data = torch.from_numpy(np.array(sample_image)).type("torch.Tensor").reshape(3, 50, 50)
            target = torch.tensor([[1, 0]])
            sample = (data, target)
            dataset.append(sample)

        train_loader = torch.utils.data.DataLoader(dataset, batch_size=8)

        return train_loader

    def train(self):
        train_set = self.preporcessing()

        criterion = nn.CrossEntropyLoss()
        optimizer = torch.optim.SGD(self.model.parameters(), lr=self.learning_rate)
        for epoch in range(self.epochs):
            epoch_loss = 0
            for i, data in enumerate(train_set, 0):

                sample, target = data
                # set data as cuda varibale
                sample = Variable(sample.float().cuda())
                target = Variable(target.cuda())
                # initialize optimizer
                optimizer.zero_grad()
                # predict
                output = self.model(sample)
                # backpropagation
                print(output, target.squeeze(1))
                loss = criterion(output, target.squeeze(1))    # ERROR MESSAGE: RuntimeError: multi-target not supported at /pytorch/aten/src/THCUNN/generic/ClassNLLCriterion.cu:15
                loss.backward()
                optimizer.step()

                epoch_loss += loss.item()

            print("loss after epoch [", epoch, "|", self.epochs, "] :", epoch_loss)

run = Run(10, 0.001, 0.5, 0.9)
run.train()

So I expected it to start training (of course not learning anything because the labels are wrong).

Output:
For nn.CrossEntropyLoss the target has to be a single number from the interval [0, #classes] instead of a one-hot encoded target vector. Your target is [1, 0], thus PyTorch thinks you want to have multiple labels per input which is not supported.

Replace your one-hot-encoded targets:

[1, 0] --> 0

[0, 1] --> 1


Instruction: Sum all diagonals in feature maps in parallel in PyTorch

Input:
Let's say I have a tensor shaped (1, 64, 128, 128) and I want to create a tensor of shape (1, 64, 255) holding the sums of all diagonals for every (128, 128) matrix (there are 1 main, 127 below, 127 above diagonals so in total 255). What I am currently doing is the following:

x = torch.rand(1, 64, 128, 128)

diag_sums = torch.zeros(1, 64, 255)
j = 0
for k in range(-127, 128):
    diag_sums[j, :, k + 127] = torch.diagonal(x, offset=k, dim1=-2, dim2=-1).sum(dim=2)

This is obviously very slow, since it is using Python loops and is not done in parallel with respect to k.

I don't think this can be done using torch.diagonal since the function explicitly uses a single int for the offset parameter. If I could pass a list there, this would work, but I guess it would be complicated to implement (requiring changes in PyTorch itself).

I think it could be possible to implement this using torch.einsum, but I cannot think of a way to do it.

So this is my question: how do I get the tensor described above?

Output:
Have you considered using torch.nn.functional.conv2d?
You can sum the diagonals with a diagonal filter sliding across the tensor with appropriate zero padding.

import torch
import torch.nn.functional as nnf

# construct a diagonal filter using `eye` function, shape it appropriately
f = torch.eye(x.shape[2])[None, None,...].repeat(x.shape[1], 1, 1, 1)
# compute the diagonal sum with appropriate zero padding
conv_diag_sums = nnf.conv2d(x, f, padding=(x.shape[2]-1,0), groups=x.shape[1])[..., 0]

Note the the result has a slightly different order than the one you computed in the loop:

diag_sums = torch.zeros(1, 64, 255)
for k in range(-127, 128):
    diag_sums[j, :, 127-k] = torch.diagonal(x, offset=k, dim1=-2, dim2=-1).sum(dim=2)

# compare
(conv_diag_sums == diag_sums).all()

results with True - they are the same.


Instruction: Cannot run Flask on Docker (ModuleNotFoundError)

Input:
I have been trying to run my Python API (using Flask) with Docker for a while, but keep running into this issue:

ModuleNotFoundError: No module named 'flask'

Running this on Mac OS X (10.14.5) with Docker version 19.03.1, build 74b1e89.

My Dockerfile looks like this:

FROM pytorch/pytorch:1.1.0-cuda10.0-cudnn7.5-runtime

RUN apt-get update && apt-get install -y python3
RUN apt-get install -y python3-pip
RUN apt-get install -y build-essential
RUN python3 --version
RUN pip3 --version

COPY . /app
WORKDIR /app

RUN pip3 install -r requirements.txt
RUN pip3 install flask # just to be sure Flask gets installed
RUN which flask
RUN ls -alh

ENTRYPOINT ["python3"]
CMD ["api.py"]

Note that I'm extending the PyTorch docker image, because my API will be running an ML model using PyTorch. I tried starting from other images such as ufoym/deepo but with the same Flask error.

I stripped my api.py file to look like this (running it locally is no problem):

from flask import Flask, request, jsonify
#import classifier # commented out for now because I just want the API to work first

app = Flask(__name__)

@app.route('/', methods=['GET', 'POST'])
def process():
    # TODO
    return 'test'

if __name__ == '__main__':
    app.run(debug=True, host='0.0.0.0')

When I build my Docker image, I get the following output:

$ docker build -t flower-image-classifier:latest .
Sending build context to Docker daemon  269.9MB
Step 1/14 : FROM pytorch/pytorch:1.1.0-cuda10.0-cudnn7.5-runtime
 ---> 299bfb9e54db
Step 2/14 : RUN apt-get update && apt-get install -y python3
 ---> Running in f1d7db593022
Get:1 http://archive.ubuntu.com/ubuntu xenial InRelease [247 kB]
Get:2 http://security.ubuntu.com/ubuntu xenial-security InRelease [109 kB]
Get:3 http://archive.ubuntu.com/ubuntu xenial-updates InRelease [109 kB]
Get:4 http://archive.ubuntu.com/ubuntu xenial-backports InRelease [107 kB]
Get:5 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages [908 kB]
Get:6 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages [1558 kB]
Get:7 http://archive.ubuntu.com/ubuntu xenial/restricted amd64 Packages [14.1 kB]
Get:8 http://archive.ubuntu.com/ubuntu xenial/universe amd64 Packages [9827 kB]
Get:9 http://security.ubuntu.com/ubuntu xenial-security/restricted amd64 Packages [12.7 kB]
Get:10 http://security.ubuntu.com/ubuntu xenial-security/universe amd64 Packages [573 kB]
Get:11 http://security.ubuntu.com/ubuntu xenial-security/multiverse amd64 Packages [6117 B]
Get:12 http://archive.ubuntu.com/ubuntu xenial/multiverse amd64 Packages [176 kB]
Get:13 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages [1292 kB]
Get:14 http://archive.ubuntu.com/ubuntu xenial-updates/restricted amd64 Packages [13.1 kB]
Get:15 http://archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages [980 kB]
Get:16 http://archive.ubuntu.com/ubuntu xenial-updates/multiverse amd64 Packages [19.1 kB]
Get:17 http://archive.ubuntu.com/ubuntu xenial-backports/main amd64 Packages [7942 B]
Get:18 http://archive.ubuntu.com/ubuntu xenial-backports/universe amd64 Packages [8532 B]
Fetched 16.0 MB in 9s (1744 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
  dh-python file libmagic1 libmpdec2 libpython3-stdlib libpython3.5-minimal
  libpython3.5-stdlib mime-support python3-minimal python3.5 python3.5-minimal
Suggested packages:
  python3-doc python3-tk python3-venv python3.5-venv python3.5-doc
  binfmt-support
The following NEW packages will be installed:
  dh-python file libmagic1 libmpdec2 libpython3-stdlib libpython3.5-minimal
  libpython3.5-stdlib mime-support python3 python3-minimal python3.5
  python3.5-minimal
0 upgraded, 12 newly installed, 0 to remove and 29 not upgraded.
Need to get 4885 kB of archives.
After this operation, 28.4 MB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 libpython3.5-minimal amd64 3.5.2-2ubuntu0~16.04.5 [524 kB]
Get:2 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 python3.5-minimal amd64 3.5.2-2ubuntu0~16.04.5 [1598 kB]
Get:3 http://archive.ubuntu.com/ubuntu xenial/main amd64 python3-minimal amd64 3.5.1-3 [23.3 kB]
Get:4 http://archive.ubuntu.com/ubuntu xenial/main amd64 mime-support all 3.59ubuntu1 [31.0 kB]
Get:5 http://archive.ubuntu.com/ubuntu xenial/main amd64 libmpdec2 amd64 2.4.2-1 [82.6 kB]
Get:6 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 libpython3.5-stdlib amd64 3.5.2-2ubuntu0~16.04.5 [2134 kB]
Get:7 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 python3.5 amd64 3.5.2-2ubuntu0~16.04.5 [165 kB]
Get:8 http://archive.ubuntu.com/ubuntu xenial/main amd64 libpython3-stdlib amd64 3.5.1-3 [6818 B]
Get:9 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 dh-python all 2.20151103ubuntu1.1 [74.1 kB]
Get:10 http://archive.ubuntu.com/ubuntu xenial/main amd64 python3 amd64 3.5.1-3 [8710 B]
Get:11 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 libmagic1 amd64 1:5.25-2ubuntu1.2 [216 kB]
Get:12 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 file amd64 1:5.25-2ubuntu1.2 [21.2 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 4885 kB in 7s (663 kB/s)
Selecting previously unselected package libpython3.5-minimal:amd64.
(Reading database ... 11013 files and directories currently installed.)
Preparing to unpack .../libpython3.5-minimal_3.5.2-2ubuntu0~16.04.5_amd64.deb ...
Unpacking libpython3.5-minimal:amd64 (3.5.2-2ubuntu0~16.04.5) ...
Selecting previously unselected package python3.5-minimal.
Preparing to unpack .../python3.5-minimal_3.5.2-2ubuntu0~16.04.5_amd64.deb ...
Unpacking python3.5-minimal (3.5.2-2ubuntu0~16.04.5) ...
Selecting previously unselected package python3-minimal.
Preparing to unpack .../python3-minimal_3.5.1-3_amd64.deb ...
Unpacking python3-minimal (3.5.1-3) ...
Selecting previously unselected package mime-support.
Preparing to unpack .../mime-support_3.59ubuntu1_all.deb ...
Unpacking mime-support (3.59ubuntu1) ...
Selecting previously unselected package libmpdec2:amd64.
Preparing to unpack .../libmpdec2_2.4.2-1_amd64.deb ...
Unpacking libmpdec2:amd64 (2.4.2-1) ...
Selecting previously unselected package libpython3.5-stdlib:amd64.
Preparing to unpack .../libpython3.5-stdlib_3.5.2-2ubuntu0~16.04.5_amd64.deb ...
Unpacking libpython3.5-stdlib:amd64 (3.5.2-2ubuntu0~16.04.5) ...
Selecting previously unselected package python3.5.
Preparing to unpack .../python3.5_3.5.2-2ubuntu0~16.04.5_amd64.deb ...
Unpacking python3.5 (3.5.2-2ubuntu0~16.04.5) ...
Selecting previously unselected package libpython3-stdlib:amd64.
Preparing to unpack .../libpython3-stdlib_3.5.1-3_amd64.deb ...
Unpacking libpython3-stdlib:amd64 (3.5.1-3) ...
Selecting previously unselected package dh-python.
Preparing to unpack .../dh-python_2.20151103ubuntu1.1_all.deb ...
Unpacking dh-python (2.20151103ubuntu1.1) ...
Processing triggers for libc-bin (2.23-0ubuntu11) ...
Setting up libpython3.5-minimal:amd64 (3.5.2-2ubuntu0~16.04.5) ...
Setting up python3.5-minimal (3.5.2-2ubuntu0~16.04.5) ...
Setting up python3-minimal (3.5.1-3) ...
Selecting previously unselected package python3.
(Reading database ... 11957 files and directories currently installed.)
Preparing to unpack .../python3_3.5.1-3_amd64.deb ...
Unpacking python3 (3.5.1-3) ...
Selecting previously unselected package libmagic1:amd64.
Preparing to unpack .../libmagic1_1%3a5.25-2ubuntu1.2_amd64.deb ...
Unpacking libmagic1:amd64 (1:5.25-2ubuntu1.2) ...
Selecting previously unselected package file.
Preparing to unpack .../file_1%3a5.25-2ubuntu1.2_amd64.deb ...
Unpacking file (1:5.25-2ubuntu1.2) ...
Processing triggers for libc-bin (2.23-0ubuntu11) ...
Setting up mime-support (3.59ubuntu1) ...
Setting up libmpdec2:amd64 (2.4.2-1) ...
Setting up libpython3.5-stdlib:amd64 (3.5.2-2ubuntu0~16.04.5) ...
Setting up python3.5 (3.5.2-2ubuntu0~16.04.5) ...
Setting up libpython3-stdlib:amd64 (3.5.1-3) ...
Setting up libmagic1:amd64 (1:5.25-2ubuntu1.2) ...
Setting up file (1:5.25-2ubuntu1.2) ...
Setting up python3 (3.5.1-3) ...
running python rtupdate hooks for python3.5...
running python post-rtupdate hooks for python3.5...
Setting up dh-python (2.20151103ubuntu1.1) ...
Processing triggers for libc-bin (2.23-0ubuntu11) ...
Removing intermediate container f1d7db593022
 ---> 562a488914b2
Step 3/14 : RUN apt-get install -y python3-pip
 ---> Running in 3884de8e3194
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
  libexpat1 libexpat1-dev libpython3-dev libpython3.5 libpython3.5-dev
  python-pip-whl python3-dev python3-pkg-resources python3-setuptools
  python3-wheel python3.5-dev
Suggested packages:
  python-setuptools-doc
The following NEW packages will be installed:
  libexpat1-dev libpython3-dev libpython3.5 libpython3.5-dev python-pip-whl
  python3-dev python3-pip python3-pkg-resources python3-setuptools
  python3-wheel python3.5-dev
The following packages will be upgraded:
  libexpat1
1 upgraded, 11 newly installed, 0 to remove and 28 not upgraded.
Need to get 40.7 MB of archives.
After this operation, 62.4 MB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 libexpat1 amd64 2.1.0-7ubuntu0.16.04.4 [71.4 kB]
Get:2 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 libexpat1-dev amd64 2.1.0-7ubuntu0.16.04.4 [115 kB]
Get:3 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 libpython3.5 amd64 3.5.2-2ubuntu0~16.04.5 [1360 kB]
Get:4 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 libpython3.5-dev amd64 3.5.2-2ubuntu0~16.04.5 [37.3 MB]
Get:5 http://archive.ubuntu.com/ubuntu xenial/main amd64 libpython3-dev amd64 3.5.1-3 [6926 B]
Get:6 http://archive.ubuntu.com/ubuntu xenial-updates/universe amd64 python-pip-whl all 8.1.1-2ubuntu0.4 [1110 kB]
Get:7 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 python3.5-dev amd64 3.5.2-2ubuntu0~16.04.5 [413 kB]
Get:8 http://archive.ubuntu.com/ubuntu xenial/main amd64 python3-dev amd64 3.5.1-3 [1186 B]
Get:9 http://archive.ubuntu.com/ubuntu xenial-updates/universe amd64 python3-pip all 8.1.1-2ubuntu0.4 [109 kB]
Get:10 http://archive.ubuntu.com/ubuntu xenial/main amd64 python3-pkg-resources all 20.7.0-1 [79.0 kB]
Get:11 http://archive.ubuntu.com/ubuntu xenial/main amd64 python3-setuptools all 20.7.0-1 [88.0 kB]
Get:12 http://archive.ubuntu.com/ubuntu xenial/universe amd64 python3-wheel all 0.29.0-1 [48.1 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 40.7 MB in 1min 5s (624 kB/s)
(Reading database ... 12000 files and directories currently installed.)
Preparing to unpack .../libexpat1_2.1.0-7ubuntu0.16.04.4_amd64.deb ...
Unpacking libexpat1:amd64 (2.1.0-7ubuntu0.16.04.4) over (2.1.0-7ubuntu0.16.04.3) ...
Selecting previously unselected package libexpat1-dev:amd64.
Preparing to unpack .../libexpat1-dev_2.1.0-7ubuntu0.16.04.4_amd64.deb ...
Unpacking libexpat1-dev:amd64 (2.1.0-7ubuntu0.16.04.4) ...
Selecting previously unselected package libpython3.5:amd64.
Preparing to unpack .../libpython3.5_3.5.2-2ubuntu0~16.04.5_amd64.deb ...
Unpacking libpython3.5:amd64 (3.5.2-2ubuntu0~16.04.5) ...
Selecting previously unselected package libpython3.5-dev:amd64.
Preparing to unpack .../libpython3.5-dev_3.5.2-2ubuntu0~16.04.5_amd64.deb ...
Unpacking libpython3.5-dev:amd64 (3.5.2-2ubuntu0~16.04.5) ...
Selecting previously unselected package libpython3-dev:amd64.
Preparing to unpack .../libpython3-dev_3.5.1-3_amd64.deb ...
Unpacking libpython3-dev:amd64 (3.5.1-3) ...
Selecting previously unselected package python-pip-whl.
Preparing to unpack .../python-pip-whl_8.1.1-2ubuntu0.4_all.deb ...
Unpacking python-pip-whl (8.1.1-2ubuntu0.4) ...
Selecting previously unselected package python3.5-dev.
Preparing to unpack .../python3.5-dev_3.5.2-2ubuntu0~16.04.5_amd64.deb ...
Unpacking python3.5-dev (3.5.2-2ubuntu0~16.04.5) ...
Selecting previously unselected package python3-dev.
Preparing to unpack .../python3-dev_3.5.1-3_amd64.deb ...
Unpacking python3-dev (3.5.1-3) ...
Selecting previously unselected package python3-pip.
Preparing to unpack .../python3-pip_8.1.1-2ubuntu0.4_all.deb ...
Unpacking python3-pip (8.1.1-2ubuntu0.4) ...
Selecting previously unselected package python3-pkg-resources.
Preparing to unpack .../python3-pkg-resources_20.7.0-1_all.deb ...
Unpacking python3-pkg-resources (20.7.0-1) ...
Selecting previously unselected package python3-setuptools.
Preparing to unpack .../python3-setuptools_20.7.0-1_all.deb ...
Unpacking python3-setuptools (20.7.0-1) ...
Selecting previously unselected package python3-wheel.
Preparing to unpack .../python3-wheel_0.29.0-1_all.deb ...
Unpacking python3-wheel (0.29.0-1) ...
Processing triggers for libc-bin (2.23-0ubuntu11) ...
Setting up libexpat1:amd64 (2.1.0-7ubuntu0.16.04.4) ...
Setting up libexpat1-dev:amd64 (2.1.0-7ubuntu0.16.04.4) ...
Setting up libpython3.5:amd64 (3.5.2-2ubuntu0~16.04.5) ...
Setting up libpython3.5-dev:amd64 (3.5.2-2ubuntu0~16.04.5) ...
Setting up libpython3-dev:amd64 (3.5.1-3) ...
Setting up python-pip-whl (8.1.1-2ubuntu0.4) ...
Setting up python3.5-dev (3.5.2-2ubuntu0~16.04.5) ...
Setting up python3-dev (3.5.1-3) ...
Setting up python3-pip (8.1.1-2ubuntu0.4) ...
Setting up python3-pkg-resources (20.7.0-1) ...
Setting up python3-setuptools (20.7.0-1) ...
Setting up python3-wheel (0.29.0-1) ...
Processing triggers for libc-bin (2.23-0ubuntu11) ...
Removing intermediate container 3884de8e3194
 ---> 388a6aa0bb5f
Step 4/14 : RUN apt-get install -y build-essential
 ---> Running in 2e8c8dc95d9e
Reading package lists...
Building dependency tree...
Reading state information...
build-essential is already the newest version (12.1ubuntu2).
0 upgraded, 0 newly installed, 0 to remove and 28 not upgraded.
Removing intermediate container 2e8c8dc95d9e
 ---> 3c08d3467777
Step 5/14 : RUN python3 --version
 ---> Running in 027dcf1dc8b6
Python 3.6.8 :: Anaconda, Inc.
Removing intermediate container 027dcf1dc8b6
 ---> 0a8389f2fe13
Step 6/14 : RUN pip3 --version
 ---> Running in 1df781383970
pip 8.1.1 from /usr/lib/python3/dist-packages (python 3.5)
Removing intermediate container 1df781383970
 ---> ad30a1cd5323
Step 7/14 : COPY . /app
 ---> db6769b7ba80
Step 8/14 : WORKDIR /app
 ---> Running in c234276bf1f6
Removing intermediate container c234276bf1f6
 ---> face402e69b6
Step 9/14 : RUN pip3 install -r requirements.txt
 ---> Running in d89df8c75e5a
Collecting Flask==1.1.1 (from -r requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/9b/93/628509b8d5dc749656a9641f4caf13540e2cdec85276964ff8f43bbb1d3b/Flask-1.1.1-py2.py3-none-any.whl (94kB)
Collecting numpy==1.16.4 (from -r requirements.txt (line 2))
  Downloading https://files.pythonhosted.org/packages/bb/ef/d5a21cbc094d3f4d5b5336494dbcc9550b70c766a8345513c7c24ed18418/numpy-1.16.4-cp35-cp35m-manylinux1_x86_64.whl (17.2MB)
Collecting Pillow==6.1.0 (from -r requirements.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/d6/98/0d360dbc087933679398d73187a503533ec0547ba4ffd2115365605559cc/Pillow-6.1.0-cp35-cp35m-manylinux1_x86_64.whl (2.1MB)
Collecting Werkzeug>=0.15 (from Flask==1.1.1->-r requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/d1/ab/d3bed6b92042622d24decc7aadc8877badf18aeca1571045840ad4956d3f/Werkzeug-0.15.5-py2.py3-none-any.whl (328kB)
Collecting itsdangerous>=0.24 (from Flask==1.1.1->-r requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/76/ae/44b03b253d6fade317f32c24d100b3b35c2239807046a4c953c7b89fa49e/itsdangerous-1.1.0-py2.py3-none-any.whl
Collecting click>=5.1 (from Flask==1.1.1->-r requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/fa/37/45185cb5abbc30d7257104c434fe0b07e5a195a6847506c074527aa599ec/Click-7.0-py2.py3-none-any.whl (81kB)
Collecting Jinja2>=2.10.1 (from Flask==1.1.1->-r requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/1d/e7/fd8b501e7a6dfe492a433deb7b9d833d39ca74916fa8bc63dd1a4947a671/Jinja2-2.10.1-py2.py3-none-any.whl (124kB)
Collecting MarkupSafe>=0.23 (from Jinja2>=2.10.1->Flask==1.1.1->-r requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/6e/57/d40124076756c19ff2269678de7ae25a14ebbb3f6314eb5ce9477f191350/MarkupSafe-1.1.1-cp35-cp35m-manylinux1_x86_64.whl
Installing collected packages: Werkzeug, itsdangerous, click, MarkupSafe, Jinja2, Flask, numpy, Pillow
Successfully installed Flask-1.1.1 Jinja2-2.10.1 MarkupSafe-1.1.1 Pillow-6.1.0 Werkzeug-0.15.5 click-7.0 itsdangerous-1.1.0 numpy-1.16.4
You are using pip version 8.1.1, however version 19.2.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Removing intermediate container d89df8c75e5a
 ---> 95766b1e174e
Step 10/14 : RUN pip3 install flask
 ---> Running in 97e806168780
Requirement already satisfied (use --upgrade to upgrade): flask in /usr/local/lib/python3.5/dist-packages
Requirement already satisfied (use --upgrade to upgrade): itsdangerous>=0.24 in /usr/local/lib/python3.5/dist-packages (from flask)
Requirement already satisfied (use --upgrade to upgrade): Jinja2>=2.10.1 in /usr/local/lib/python3.5/dist-packages (from flask)
Requirement already satisfied (use --upgrade to upgrade): Werkzeug>=0.15 in /usr/local/lib/python3.5/dist-packages (from flask)
Requirement already satisfied (use --upgrade to upgrade): click>=5.1 in /usr/local/lib/python3.5/dist-packages (from flask)
Requirement already satisfied (use --upgrade to upgrade): MarkupSafe>=0.23 in /usr/local/lib/python3.5/dist-packages (from Jinja2>=2.10.1->flask)
You are using pip version 8.1.1, however version 19.2.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Removing intermediate container 97e806168780
 ---> dd8d93e7b8bc
Step 11/14 : RUN which flask
 ---> Running in 0ac4678d5ecd
/usr/local/bin/flask
Removing intermediate container 0ac4678d5ecd
 ---> e9325cc606ec
Step 12/14 : RUN ls -alh
 ---> Running in e249ecafd7fa
total 1.8M
drwxr-xr-x 6 root root  4.0K Aug  4 14:29 .
drwxr-xr-x 1 root root  4.0K Aug  4 14:30 ..
-rw-r--r-- 1 root root   11K Aug  2 11:15 .DS_Store
drwxr-xr-x 5 root root  4.0K Jul 28 11:45 .git
-rw-r--r-- 1 root root    18 Aug  2 13:52 .gitignore
-rw-r--r-- 1 root root   878 Aug  4 14:27 Dockerfile
-rw-r--r-- 1 root root 1004K Jul 24 12:49 Image Classifier Project.html
-rw-r--r-- 1 root root  690K Jul 24 12:50 Image Classifier Project.ipynb
-rw-r--r-- 1 root root  1.1K Feb  6  2018 LICENSE
-rw-r--r-- 1 root root   686 Aug  2 14:04 README.md
drwxr-xr-x 2 root root  4.0K Aug  4 13:05 __pycache__
-rw-r--r-- 1 root root  1018 Aug  4 14:12 api.py
drwxr-xr-x 2 root root  4.0K Jul 20 12:32 assets
-rw-r--r-- 1 root root  2.2K Feb  6  2018 cat_to_name.json
drwxr-xr-x 2 root root  4.0K Jul 24 12:06 checkpoints
-rw-r--r-- 1 root root   245 Aug  2 21:48 environment.yml
-rw-r--r-- 1 root root   921 Aug  4 09:11 environment0.yml
-rw-r--r-- 1 root root  8.6K Aug  4 13:02 flower_classifier.py
-rw-r--r-- 1 root root  1.1K Aug  2 19:17 predict.py
-rw-r--r-- 1 root root    41 Aug  4 12:54 requirements.txt
-rw-r--r-- 1 root root  1.6K Aug  2 18:58 train.py
Removing intermediate container e249ecafd7fa
 ---> dfe9737330ac
Step 13/14 : ENTRYPOINT ["python3"]
 ---> Running in 57782c4a0d03
Removing intermediate container 57782c4a0d03
 ---> 7e6b0d1d5656
Step 14/14 : CMD ["api.py"]
 ---> Running in 8400de10dc2d
Removing intermediate container 8400de10dc2d
 ---> 33733bebf60d
Successfully built 33733bebf60d
Successfully tagged flower-image-classifier:latest

When I run the container, I get the following error:

$ docker run -p 5000:5000 flower-image-classifier
Traceback (most recent call last):
  File "api.py", line 1, in <module>
    from flask import Flask, request, jsonify
ModuleNotFoundError: No module named 'flask'

My question to the Docker/Python experts here: how is it possible that flask is not found, when pip has clearly installed it according to the logs?

Note that I also tried starting from the continuumio/miniconda3 image at first, just like in this issue: Docker Flask ModuleNotFoundError: No module named 'flask'

I created this environment.yml:

name: fic
channels:
  - pytorch
  - defaults
dependencies:
  - flask=1.1.1
  - numpy=1.16.4
  - numpy-base=1.16.4
  - pillow=6.1.0
  - python==3.6.9
  - pytorch=1.1.0
  - torchvision=0.3.0
  - pip:
    - pip==19.2.1
prefix: //anaconda3/envs/fic

Then I modified my Dockerfile a bit to extend from that image, and to run this to install the dependencies with Conda:

RUN echo "source activate $(head -1 ./environment.yml | cut -d' ' -f2)" > ~/.bashrc
ENV PATH /opt/conda/envs/$(head -1 ./environment.yml | cut -d' ' -f2)/bin:$PATH

Unfortunately, this downloads & extracts everything every time I build the Docker image, which takes a while and the downloads often randomly fail: CondaError: Downloaded bytes did not match Content-Length.

And when it finally built successfully, I still got the same ModuleNotFoundError for Flask.

What is wrong here? Thanks for reading

Output:
you need just to specify the correct Python version, so you need just to change your entrypoint to:

ENTRYPOINT ["python3.5"]


Instruction: What is the point of the comma operator when a tuple only has one entry in python?

Input:
This section of code is taken from one of the Pytorch tutorials, I have just removed the non-essential parts so it doesn't error out and added some print statements. The question I have is why the two print statements I provided have slightly different results? Is this a tuple with nothing in the second half of it? I am confused by the comma without anything after it before the assignment operator.

import torch

class MyReLU(torch.autograd.Function):
    @staticmethod
    def forward(ctx, input):
        ctx.save_for_backward(input)
        return input.clamp(min=0)

    @staticmethod
    def backward(ctx, grad_output):
        input, = ctx.saved_tensors
        print("ctx ", ctx.saved_tensors)
        print("inputs ", input)
        grad_input = grad_output.clone()
        grad_input[input < 0] = 0
        return grad_input

relu = MyReLU.apply
relu = MyReLU.apply
y_pred = relu(x.mm(w1)).mm(w2)
loss = (y_pred - y).pow(2).sum()
loss.backward()

Output

ctx  (tensor([[-34.2381,  18.6334,   8.8368,  ...,  13.7337, -31.5657, -11.8838],
        [-25.5597,  -6.2847,   9.9412,  ..., -75.0621,   5.0451, -32.9348],
        [-56.6591, -40.0830,   2.4311,  ...,  -2.8988, -18.9742, -74.0132],
        ...,
        [ -6.4023, -30.3526, -73.9649,  ...,   1.8587, -23.9617, -11.6951],
        [ -3.6425,  34.5828,  27.7200,  ..., -34.3878, -19.7250,  11.1960],
        [ 16.0137, -24.0628,  14.4008,  ...,  -5.4443,   9.9499, -18.1259]],
       grad_fn=<MmBackward>),)
inputs  tensor([[-34.2381,  18.6334,   8.8368,  ...,  13.7337, -31.5657, -11.8838],
        [-25.5597,  -6.2847,   9.9412,  ..., -75.0621,   5.0451, -32.9348],
        [-56.6591, -40.0830,   2.4311,  ...,  -2.8988, -18.9742, -74.0132],
        ...,
        [ -6.4023, -30.3526, -73.9649,  ...,   1.8587, -23.9617, -11.6951],
        [ -3.6425,  34.5828,  27.7200,  ..., -34.3878, -19.7250,  11.1960],
        [ 16.0137, -24.0628,  14.4008,  ...,  -5.4443,   9.9499, -18.1259]],
       grad_fn=<MmBackward>)

Output:
It's just an edge-case of unpacking a single-element list or tuple.

a, = [1]
print(type(a), a)
# <class 'int'> 1

Without the comma, a would have been assigned the entire list:

a = [1]
print(type(a), a)
# <class 'list'> [1]

And the same goes for a tuple:

a, = (1,)  # have to use , with literal single-tuples, because (1) is just 1
print(type(a), a)
# <class 'int'> 1

a = (1,)  # have to use , with literal single-tuples, because (1) is just 1
print(type(a), a)
# <class 'tuple'> (1,)


Instruction: What considerations should be taken into account with directly setting Pytorch model variables and optimizer parameters?

Input:
From the Pytorch forums
https://discuss.pytorch.org/t/layer-weight-vs-weight-data/24271/2
it is mentioned that setting variable weights directly could result in

"Using .data on the other side would work, but is generally not recommended, as changing it after the model was used would yield weird results and autograd cannot throw an error."

I am wonder what would cause weird results. Also, I am also thinking of setting optimizer parameters directly, specifically the momentum/sum of gradients for optimizers which have those parameters. Is there any considerations needed for that case as well?

Output:
It is perfectly legal to update the PyTorch layer weights.
Check this out how we can alter the weights no problem:

lin = nn.Linear(10, 2)
torch.nn.init.xavier_uniform_(lin.weight)

The upper code actually calls with torch.no_grad():

def _no_grad_uniform_(tensor, a, b):
    with torch.no_grad():
        return tensor.uniform_(a, b)

See how the torch.no_grad() will help us in the next example.

lin = nn.Linear(10, 2)
with torch.no_grad():
    lin.weight[0][0] = 1.

x = torch.randn(1, 10)
output = lin(x)
output.mean().backward()

And if we don't use it:

lin = nn.Linear(10, 2)
lin.weight[0][0] = 1.
x = torch.randn(1, 10)
output = lin(x)
output.mean().backward()

We end in:

  RuntimeError: leaf variable has been moved into the graph interior

So you can do it but inside with torch.no_grad():.
This is because every action we do on PyTorch tensor will get caught, if requires grad is set to True.

In case we do lin.weight[0][0] = 1. we will catch grad_fn=<CopySlices>.
The thing is we don't need that to be caught as this is part of our layer setup and not our calculation.


Instruction: Can ReduceLrOnPlateau scheduler in pytorch use test set metric for decreasing learning rate?

Input:
Hi I am currently learning the use of scheduler in deep learning in pytroch. I came across the following code :

import torch
import torch.nn as nn
import torchvision.transforms as transforms
import torchvision.datasets as dsets

# Set seed
torch.manual_seed(0)

# Where to add a new import
from torch.optim.lr_scheduler import ReduceLROnPlateau

'''
STEP 1: LOADING DATASET
'''

train_dataset = dsets.MNIST(root='./data',
                            train=True,
                            transform=transforms.ToTensor(),
                            download=True)

test_dataset = dsets.MNIST(root='./data',
                           train=False,
                           transform=transforms.ToTensor())

'''
STEP 2: MAKING DATASET ITERABLE
'''

batch_size = 100
n_iters = 6000
num_epochs = n_iters / (len(train_dataset) / batch_size)
num_epochs = int(num_epochs)

train_loader = torch.utils.data.DataLoader(dataset=train_dataset,
                                           batch_size=batch_size,
                                           shuffle=True)

test_loader = torch.utils.data.DataLoader(dataset=test_dataset,
                                          batch_size=batch_size,
                                          shuffle=False)

'''
STEP 3: CREATE MODEL CLASS
'''
class FeedforwardNeuralNetModel(nn.Module):
    def __init__(self, input_dim, hidden_dim, output_dim):
        super(FeedforwardNeuralNetModel, self).__init__()
        # Linear function
        self.fc1 = nn.Linear(input_dim, hidden_dim)
        # Non-linearity
        self.relu = nn.ReLU()
        # Linear function (readout)
        self.fc2 = nn.Linear(hidden_dim, output_dim)

    def forward(self, x):
        # Linear function
        out = self.fc1(x)
        # Non-linearity
        out = self.relu(out)
        # Linear function (readout)
        out = self.fc2(out)
        return out
'''
STEP 4: INSTANTIATE MODEL CLASS
'''
input_dim = 28*28
hidden_dim = 100
output_dim = 10

model = FeedforwardNeuralNetModel(input_dim, hidden_dim, output_dim)

'''
STEP 5: INSTANTIATE LOSS CLASS
'''
criterion = nn.CrossEntropyLoss()

'''
STEP 6: INSTANTIATE OPTIMIZER CLASS
'''
learning_rate = 0.1

optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate, momentum=0.9, nesterov=True)

'''
STEP 7: INSTANTIATE STEP LEARNING SCHEDULER CLASS
'''
# lr = lr * factor
# mode='max': look for the maximum validation accuracy to track
# patience: number of epochs - 1 where loss plateaus before decreasing LR
        # patience = 0, after 1 bad epoch, reduce LR
# factor = decaying factor
scheduler = ReduceLROnPlateau(optimizer, mode='max', factor=0.1, patience=0, verbose=True)

'''
STEP 7: TRAIN THE MODEL
'''
iter = 0
for epoch in range(num_epochs):
    for i, (images, labels) in enumerate(train_loader):
        # Load images as Variable
        images = images.view(-1, 28*28).requires_grad_()

        # Clear gradients w.r.t. parameters
        optimizer.zero_grad()

        # Forward pass to get output/logits
        outputs = model(images)

        # Calculate Loss: softmax --> cross entropy loss
        loss = criterion(outputs, labels)

        # Getting gradients w.r.t. parameters
        loss.backward()

        # Updating parameters
        optimizer.step()

        iter += 1

        if iter % 500 == 0:
            # Calculate Accuracy
            correct = 0
            total = 0
            # Iterate through test dataset
            for images, labels in test_loader:
                # Load images to a Torch Variable
                images = images.view(-1, 28*28)

                # Forward pass only to get logits/output
                outputs = model(images)

                # Get predictions from the maximum value
                _, predicted = torch.max(outputs.data, 1)

                # Total number of labels
                total += labels.size(0)

                # Total correct predictions
                # Without .item(), it is a uint8 tensor which will not work when you pass this number to the scheduler
                correct += (predicted == labels).sum().item()

            accuracy = 100 * correct / total

            # Print Loss
            # print('Iteration: {}. Loss: {}. Accuracy: {}'.format(iter, loss.data[0], accuracy))

    # Decay Learning Rate, pass validation accuracy for tracking at every epoch
    print('Epoch {} completed'.format(epoch))
    print('Loss: {}. Accuracy: {}'.format(loss.item(), accuracy))
    print('-'*20)
    scheduler.step(accuracy)

I am using the above strategy. The only thing that I am not able to understand is that how are they using test data to step up the accuracy and decreasing learning rate on the basis of that via scheduler? It is the last line of the code. Can we during training show the test accuracy to the scheduler and ask that it to reduce the learning rate? I found the similar thing on github resnet main.py
too. Can someone please clarify ?

Output:
I think there might be some confusion regarding the term test here.

Difference between test and validation data

What the code actually refers to by test is the validation set not the actual test set. The difference is that the validation set is used during training to see how well the model generalizes. Normally people just cut off a part of the training data and use that for validation. To me it seems like your code is using the same data for training and validation but that's just my assumption because I don't know what ./data looks like.

To work in a strictly scientific way, your model should never see actual test data during training, only training and validation. This way we can assess the models actual ability to generalize on unseen data after training.

Reducing learning rate based on validation accuracy

The reason why you use validation data (called test data in your case) to reduce the learning rate is probably because if you did this using the actual training data and training accuracy the model is more likely to overfit. Why? When you are on a plateau of the training accuracy it does not necessarily imply that it's a plateau of the validation accuracy and the other way round. Meaning you could be stepping in a promising direction regarding the validation accuracy (and thus in a direction of parameters that generalize well) and suddenly you reduce or increase the learning rate because there was a plateau (or non) in the training accuracy.


Instruction: How to transform BERT's network output to readable text?

Input:
I am trying to understand, how to use BERT for QnA and found a tutorial on how to start on PyTorch (here). Now I would like to use these snippets to get started, but i do not understand how to project the output back on the example text.

text = "[CLS] Who was Jim Henson ? [SEP] Jim Henson was a puppeteer [SEP]"

(...)

# Predict the start and end positions logits
with torch.no_grad():
    start_logits, end_logits = questionAnswering_model(tokens_tensor, segments_tensors)

# Or get the total loss
start_positions, end_positions = torch.tensor([12]), torch.tensor([14])

multiple_choice_loss = questionAnswering_model(
                         tokens_tensor,
                         segments_tensors,
                         start_positions=start_positions,
                         end_positions=end_positions)

  start_logits (shape: [1, 16]):tensor([[ 0.0196,  0.1578,  0.0848,  0.1333, -0.4113, -0.0241, -0.1060, -0.3649,
            0.0955, -0.4644, -0.1548,  0.0967, -0.0659,  0.1055, -0.1488, -0.3649]])

  end_logits (shape: [1, 16]):tensor([[ 0.1828, -0.2691, -0.0594, -0.1618,  0.0441, -0.2574, -0.2883,  0.2526,
           -0.0551, -0.0051, -0.1572, -0.1670, -0.1219, -0.1831, -0.4463,  0.2526]])

If my assumption is correct start_logits and end_logits need to be projected back on text, but how do i compute this?

Additionally do you have any resources/guides/tutorials you could recommend to continue further into QnA (except google-research/bert github and the paper for bert)?

Thank you in advance.

Output:
I think you are trying to use Bert for Q&A where the answer is a span of the original text. The original paper uses this for the SQuAD dataset where this is the case. start_logits and end_logits have the logits for a token being the start/end of the answer so you can take argmax and it would be the index of the token in the text.

There is this NAACL tutorial on tranfer learning from the author of the repo you linked https://colab.research.google.com/drive/1iDHCYIrWswIKp-n-pOg69xLoZO09MEgf#scrollTo=qQ7-pH1Jp5EG It's using classification as the target task but you might still find it useful.


Instruction: Pytorch is not found & cannot be installed in pycharm

Input:
I need pytorch module to run my project. But When I tried to install it via command prompt via two different ways, it shows an error - .

C:\Users\Toothless>pip install torchvision --user
Collecting torchvision
Using cached https://files.pythonhosted.org/packages/b7/ff /091b4503d5f228bd1120db784e2c071617211b965a8a78018e75750c7199/torchvision-0.3.0-cp37-cp37m-win_amd64.whl
Requirement already satisfied: six in c:\users\toothless\appdata\local\programs\python\python37\lib\site-packages (from torchvision) (1.12.0)
Collecting pillow>=4.1.1 (from torchvision)
Downloading https://files.pythonhosted.org/packages/ae/96/6f83deebfcd20a5d4ad35e4e989814a16559d8715741457e670aae1a5a09/Pillow-6.1.0-cp37-cp37m-win_amd64.whl (2.0MB)
 |████████████████████████████████| 2.0MB 27kB/s
Requirement already satisfied: numpy in c:\users\toothless\appdata\local\programs\python\python37\lib\site-packages (from torchvision) (1.17.0)
 Collecting torch>=1.1.0 (from torchvision)
 ERROR: Could not find a version that satisfies the requirement torch>=1.1.0 (from torchvision) (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)
 ERROR: No matching distribution found for torch>=1.1.0 (from torchvision)

 C:\Users\Toothless>pip3 install torchvision
 Collecting torchvision
 Using cached https://files.pythonhosted.org/packages/b7/ff/091b4503d5f228bd1120db784e2c071617211b965a8a78018e75750c7199/torchvision-0.3.0-cp37-cp37m-win_amd64.whl
 Requirement already satisfied: six in c:\users\toothless\appdata\local\programs\python\python37\lib\site-packages (from torchvision) (1.12.0)
 Collecting torch>=1.1.0 (from torchvision)
 ERROR: Could not find a version that satisfies the requirement torch>=1.1.0 (from torchvision) (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)
 ERROR: No matching distribution found for torch>=1.1.0 (from torchvision)

Again I've tried to installed through pycharm .

I've been following this question but conda`` is not listed in my envairoment variable.

Edited: When I tried pip install torch torchvision --user this command it gives an error - ERROR: torchvision 0.3.0 has requirement torch>=1.1.0, but you'll have torch 0.1.2.post2 which is incompatible.

Edited: I've also tried pip install torchvision but it shows with error -

C:\Users\Toothless>pip install torchvision
Collecting torchvision
Using cached https://files.pythonhosted.org/packages/b7/ff/091b4503d5f228bd1120db784e2c071617211b965a8a78018e75750c7199/torchvision-0.3.0-cp37-cp37m-win_amd64.whl
Collecting torch>=1.1.0 (from torchvision)
ERROR: Could not find a version that satisfies the requirement torch>=1.1.0 (from torchvision) (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)
ERROR: No matching distribution found for torch>=1.1.0 (from torchvision)

What should i do now to make a workable pytorch?

Output:
Finally I've installed my desired pytorch & torchvision. I've installed it via pip.
The commandline I've used to install pytorch:

pip3 install https://download.pytorch.org/whl/cpu/torch-1.1.0-cp37-cp37m-win_amd64.whl

And for torchVision:

pip3 install https://download.pytorch.org/whl/cpu/torchvision-0.3.0-cp37-cp37m-win_amd64.whl

I've found this code from PyTorch website.


Instruction: Getting the value at the intersection of a row and column in a pytorch tensor matrix

Input:
I am new to pytorch and am looking to get a value at an index from a matrix. There is a matrix called psfm_s that has been initialized with
psfm_s=Var(torch.randn(12,20),requires_grad=True) For example, I would like to to get the number in the first column (out of 12 columns)  and the number in the first row (out of 20 rows).

I have tried doing something like index=torch.tensor([0,0])
num_at_index=psfm_s[index] to get the desired number but that just gets me a tensor with a bunch of numbers in it, I'm not really sure what happens with this method.

I just want the one number at the desired index, how can I go about doing this if it's even possible? Thanks for the help!

Output:
To reproduce the described code in its completeness (for future reference, please provide a [mcve] in your question), and taking the already correct solution from @jodag in the comments, consider this code snippet:

from torch.autograd import Variable
import torch

psfm_s = Variable(torch.randn(12,20), requires_grad=True)
single_value = psfm_s[0,0].item()
print(single_value) # prints a single random number from your tensor

For some background information, consider the official docs:

  Returns the value of this tensor as a standard Python number. This
  only works for tensors with one element. For other cases, see tolist().

  This operation is not differentiable.

Consequently, getting a complete row (or column), would look like this:

from torch.autograd import Variable
import torch

psfm_s = Variable(torch.randn(12,20), requires_grad=True)
single_row_tensor = psfm_s[0,:]
single_row_list = single_row_tensor.tolist()
single_row_numpy_1 = single_row_tensor.data.numpy()
single_row_numpy_2 = single_row_tensor.detach().numpy()

# the following doesn't work, as it is a torch.Variable with gradient history:
single_row_fail = single_row_tensor.numpy()

In the case you want to get a NumPy array, you have to be careful not to directly cast it to .numpy(), as this causes issues with the gradient history of the Variable. You can either use .data.numpy(), or .detach().numpy().
There seems to be some discussion as to which one is preferred, but both should work for your case.


Instruction: Why do I have error with trained model from ubuntu to windows?

Input:
I have trained the model on a super computer(ubuntu). After the training I used the model with Windows 10 and got this error:

SourceChangeWarning: source code of class 'torch.nn.modules.linear.Linear' has changed. you can retrieve the original source code by accessing the object's source attribute or set `torch.nn.Module.dump_patches = True` and use the patch tool to revert the changes.
  warnings.warn(msg, SourceChangeWarning)

I can't load the model I trained.

pytorch version in ubuntu: 1.1.0a0+9a7bcac

pytorch version in windows: 0.4.1

What is going wrong and how can I fix it?

Output:
EDIT after comment discussion:
It seems like yor Windows version is outdated and there is thus a version conflict occurring. I would strongly suggest to update the version on Windows to post 1.0 release, which should fix the problem.

According to this link, you can likely ignore the warning (not an error), as long as your model seems to still work as intended. The usual culprit for such changes is that you have inconsistent versions of PyTorch on your two systems, and therefore might encounter this warning.

Generally, the versions are supposed to be fully backward compatible, but of course there is no guarantee for this. It has nothing to do with the fact that you are on Linux and/or Windows, unless the source code detects changes in the line break character (which is the main difference from what I remember), although I think this is very unlikely to be the case.


Instruction: Why do we need padding in seq2seq network

Input:
To handle sequences of different, I would like to know.

Why do we need padding the sequence the word to the same length?
If the answer is "Yes, you need padding.". Can I set the padding in other indexes? For example, if I have an index word like this:

{0:"<s>,1:"<e>",2:"AAA",3:"BBB",.......,500:"zzz"}

Where <s> is starting word of the sentence and  is the ending word of the sentence.

Can I set the padding flag to the last index?

{0:"<s>,1:"<e>",2:"AAA",3:"BBB",.......,500:"zzz",501:"<pad>"}

Output:
  Why do we need padding the sequence the word to the same length?

Because basically all layers with parameters perform some way of matrix multiplication (actually: tensor multiplication) at some point in their logic. Now, try it yourself. Multiply matrices where not all rows or columns have the same length. E.g. what is this supposed to be?

| 1 2 3 |     | 1 |
| 4 5   |  *  | 2 |  =  ???
              | 3 |

It is simply not possible to do this, unless you put some value in the gap. Some people may even argue that this thing on the left hand side is not even a matrix.

  Can I set the padding in other indexes? Can I set the padding flag to the last index?

Sure. You can take whatever value you want for padding. Ideally, you should use a value that has otherwise no other meaning in the context of you problem and thus cannot be confused with any "real" value.


Instruction: Loading pytorch model in C++, problem with libtorch.dll

Input:
I'm trying to load neural network model trained with pytorch to C++ program.
There is a tutorial how to do it, but can't get it working.

The console appears, and then I get,

  "The Code execution cannot proceed, because the object xxx.dll was not found".

Sometimes it is c10.dll, other time torch.dll, or caffe2.dll.

I checked my C/C++ and Linker properties like 50 times.
I checked it with using debug and release version of libtorch.
I ran it on Debug x86, Debug x64, Release x86, Release x64.
And I added those files manually to Debug folder (where .exe file is)
None of it worked.

This is my code

#include "pch.h"
#include <torch/script.h>
#include <memory>
#include <iostream>

int main()
{
    std::cout << "Hello World!\n";
}

This is how I set directory for header files:

 $(SolutionDir)libtorch\include

...and linker directory for additional libraries:

$(SolutionDir)libtorch\lib

...and all .lib files I added:

torch.lib;onnxifi_loader.lib;onnxifi_dummy.lib;onnx_proto.lib;onnx.lib;libprotoc.lib;libprotobuf-lite.lib;libprotobuf.lib;foxi_loader.lib;foxi_dummy.lib;cpuinfo.lib;clog.lib;caffe2_module_test_dynamic.lib;caffe2_detectron_ops.lib;caffe2.lib;c10.lib;

It's not that some function or classes from this library are not working. I cannot compile a simple "Hello World" program.

I even downloaded some random .dll file to check if it is this particular library problem, and other .dll worked with no problems.

I set up the project manually, and followed the instruction from the link I sent (with creating project with CMAKE), and still I have this error.

I'm working on it for few hours and I'm pretty annoyed about this.I ran out of ideas. I really don't know what else I could miss in this situation.

I'm running VS 2017 Community, version 15.9.14 on Windows 10.

Output:
Ok, ive actually came up with solution by my own.

For some reasons which i completly do not understand i had to put .dll files in my project folder.
Setting up the path for additional libraries in linker properties seems to not work for those libraries, at least on my PC.

This is very confusing because other, random library which i downloaded for test, I can place wherever i want, all I need to do is just set up correct path in Linker properties.

But not THIS particular library (libtorch).

Anyway, problem is solved, hope that someday someone finds this usefull :)


Instruction: pytorch predictions stability

Input:
This is my predict function. is there anything wrong with this? Predictions are not stable, everytime I run on same data, I get different predictions.

def predict(model, device, inputs, batch_size=1024):
    model = model.to(device)
    dataset = torch.utils.data.TensorDataset(*inputs)
    loader = torch.utils.data.DataLoader(
                    dataset,
                    batch_size=batch_size,
                    pin_memory=False
                )

    predictions = []

    for i, batch in enumerate(loader):
        with torch.no_grad():
            pred = model(*(item.to(device) for item in batch))
            pred = pred.detach().cpu().numpy()
        predictions.append(pred)
    return np.concatenate(predictions)

Output:
As Usman Ali suggested, you need to set your model to eval mode by calling

model.eval()

before your prediction function.

What eval mode does:

  Sets the module in evaluation mode.

  This has any effect only on certain modules. See documentations of particular modules for details of their behaviors in training/evaluation mode, if they are affected, e.g. Dropout, BatchNorm, etc.

When you finish your prediction and wish t continue training, don't forget to reset your model to training mode by calling

model.train()

There are several layers in models that may introduce randomness into the forward pass of the net. One such example is the dropout layers. A dropout layer "drops" p percent of its neurons at random to increase model's generalization.
Additionally, BatchNorm (and possibly other adaptive normalization layers) keeps track of the statistics of the data and therefore has a different "behavior" in train mode or in eval mode.


Instruction: how to add BatchNormalization with SWA:stochastic weights average?

Input:
I am a beginner in Deepleaning and Pytorch.

I don't understand how to use BatchNormalization in using SWA.

pytorch.org says in https://pytorch.org/blog/stochastic-weight-averaging-in-pytorch/:

  Note that the SWA averages of the weights are never used to make
  predictions during training, and so the batch normalization layers do
  not have the activation statistics computed after you reset the
  weights of your model with opt.swap_swa_sgd()

This means it's suitable for adding BatchNormalization layer after using SWA?

# it means, in my idea

#for example

opt = torchcontrib.optim.SWA(base_opt)
for i in range(100):
    opt.zero_grad()
    loss_fn(model(input), target).backward()
    opt.step()
    if i > 10 and i % 5 == 0:
        opt.update_swa()
opt.swap_swa_sgd()

#save model once
torch.save(model,"swa_model.pt")

#model_load
saved_model=torch.load("swa_model.pt")

#it means adding BatchNormalization layer??
model2=saved_model
model2.add_module("Batch1",nn.BatchNorm1d(10))

# decay learning_rate more
learning_rate=0.005
optimizer = torch.optim.SGD(model2.parameters(), lr=learning_rate)

# train model again
for epoch in range(num_epochs):
    loss = train(train_loader)
    val_loss, val_acc = valid(test_loader)

I appreciate your replying.

following your advise,

I try to make example model adding  optimizer.bn_update()

# add  optimizer.bn_update() to model

criterion = nn.CrossEntropyLoss()
learning_rate=0.01

base_opt = torch.optim.SGD(model.parameters(), lr=0.1)
optimizer = SWA(base_opt, swa_start=10, swa_freq=5, swa_lr=0.05)

def train(train_loader):
    #mode:train
    model.train()
    running_loss = 0
    for batch_idx, (images, labels) in enumerate(train_loader):
        optimizer.zero_grad()
        outputs = model(images)
        #loss
        loss = criterion(outputs, labels)
        running_loss += loss.item()
        loss.backward()
        optimizer.step()

    optimizer.swap_swa_sgd()
    train_loss = running_loss / len(train_loader)

    return train_loss

def valid(test_loader):

    model.eval()
    running_loss = 0
    correct = 0
    total = 0
    #torch.no_grad
    with torch.no_grad():
        for batch_idx, (images, labels) in enumerate(test_loader):
            outputs = model(images)

            loss = criterion(outputs, labels)
            running_loss += loss.item()

            _, predicted = torch.max(outputs, 1)
            correct += (predicted == labels).sum().item()
            total += labels.size(0)

    val_loss = running_loss / len(test_loader)
    val_acc = float(correct) / total

    return val_loss, val_acc

num_epochs=30

loss_list = []
val_loss_list = []
val_acc_list = []
for epoch in range(num_epochs):
    loss = train(train_loader)
    val_loss, val_acc = valid(test_loader)
    optimizer.bn_update(train_loader, model)
    print('epoch %d, loss: %.4f val_loss: %.4f val_acc: %.4f'
          % (epoch, loss, val_loss, val_acc))

    # logging
    loss_list.append(loss)
    val_loss_list.append(val_loss)
    val_acc_list.append(val_acc)

# optimizer.bn_updata()
optimizer.bn_update(train_loader, model)

# go on evaluating model,,,

Output:
What the documentation is telling you is that since SWA computes averages of weights but those weights aren't used for prediction during training the batch normalization layers won't see those weights. This means they haven't computed the respective statistics for them (as they were never able to) which is important because the weights are used during actual prediction (i.e. not during training).

This means, they assume you have batch normalization layers in your model and want to train it using SWA. This is (more or less) not straight-forward due to the reasons above.

One approach is given as follows:

  To compute the activation statistics you can just make a forward pass on your training data using the SWA model once the training is finished.

Or you can use their helper class:

  In the SWA class we provide a helper function opt.bn_update(train_loader, model). It updates the activation statistics for every batch normalization layer in the model by making a forward pass on the train_loader data loader. You only need to call this function once in the end of training.

In case you are using Pytorch's DataLoader class you can simply supply the model (after training) and the training loader to the bn_update function which updates all batch normalization statistics for you. You only need to call this function once in the end of training.

Steps to proceed:

Train your model that includes batch normalization layers using SWA
After your model has finished training, call opt.bn_update(train_loader, model) using your training data and providing your trained model


Instruction: Pytorch: saving model or state_dict gives different on-disk space occupation

Input:
I was playing around with the function torch.save and I noticed something curious, let's say i load a model from torchvision repository:

model = torchvision.models.mobilenet_v2()

if i save the model in this way:

torch.save(model,'model.pth')

I get a 14MB file, while if i do:

torch.save(model.state_dict(),'state_dict.pth')

The file size blow to ~500MB. Since i didn't find any reference on this behaviour I was wondering what does cause the increment in size. Is it something related to compression? does saving the whole state_dict stores extra stuff like uninitialized gradients?

P.S. the same happens for other model like vgg16

Output:
If ask for what is in model:

vars(vgg16)

Out:

{'_backend': <torch.nn.backends.thnn.THNNFunctionBackend at 0x232c78759b0>,
 '_parameters': OrderedDict(),
 '_buffers': OrderedDict(),
 '_backward_hooks': OrderedDict(),
 '_forward_hooks': OrderedDict(),
 '_forward_pre_hooks': OrderedDict(),
 '_state_dict_hooks': OrderedDict(),
 '_load_state_dict_pre_hooks': OrderedDict(),
 '_modules': OrderedDict([('features', Sequential(
                 (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
                 (1): ReLU(inplace)
                 (2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
                 (3): ReLU(inplace)
                 (4): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
                 (5): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
                 (6): ReLU(inplace)
                 (7): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
                 (8): ReLU(inplace)
                 (9): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
                 (10): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
                 (11): ReLU(inplace)
                 (12): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
                 (13): ReLU(inplace)
                 (14): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
                 (15): ReLU(inplace)
                 (16): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
                 (17): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
                 (18): ReLU(inplace)
                 (19): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
                 (20): ReLU(inplace)
                 (21): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
                 (22): ReLU(inplace)
                 (23): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
                 (24): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
                 (25): ReLU(inplace)
                 (26): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
                 (27): ReLU(inplace)
                 (28): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
                 (29): ReLU(inplace)
                 (30): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
               )),
              ('avgpool', AdaptiveAvgPool2d(output_size=(7, 7))),
              ('classifier', Sequential(
                 (0): Linear(in_features=25088, out_features=4096, bias=True)
                 (1): ReLU(inplace)
                 (2): Dropout(p=0.5)
                 (3): Linear(in_features=4096, out_features=4096, bias=True)
                 (4): ReLU(inplace)
                 (5): Dropout(p=0.5)
                 (6): Linear(in_features=4096, out_features=1000, bias=True)
               ))]),
 'training': True}

You will get it is more than just state dict.

vgg16.state_dict()

State dict is inside _modules (vgg16._modules['features'].state_dict())

This is why when you save the model you save not just the state dict, but also all aforementioned stuff such as parameters, buffers, hooks...

But if you don't use parameters, buffers, hooks for inference time for your model you may avoid saving these.

The sizes when saving:

torch.save(model,'model.pth')
torch.save(model.state_dict(),'state_dict.pth')

should be: model.pth > state_dict.pth because state dict is included into the model.


Instruction: Reduction parameter of the negative log likelihood

Input:
What is the intuitive explanation of the reduction parameter in negative log likelihood loss function in PyTorch? The parameter can take values such as 'mean' or 'sum'. Is it summing over the elements of the batch?

torch.nn.functional.nll_loss(outputs.mean(0), target, reduction="sum")

Output:
From the documentation:

Specifies the reduction to apply to the output: 'none' | 'mean' | 'sum'. 'none': no reduction will be applied, 'mean': the sum of the output will be divided by the number of elements in the output, 'sum': the output will be summed. Note: size_average and reduce are in the process of being deprecated, and in the meantime, specifying either of those two args will override reduction. Default: 'mean'

If you use none, the output will be the same as batch size,

If you use mean, it will be the mean (sum divided by batch)

If you use sum, it will be the sum of all elements.

You can also verify this with following code:

import torch
logit = torch.rand(100,10)
target = torch.randint(10, size=(100,))
m = torch.nn.functional.nll_loss(logit, target)
s = torch.nn.functional.nll_loss(logit, target, reduction="sum")
l = torch.nn.functional.nll_loss(logit, target, reduction="none")
print(torch.abs(m-s/100))
print(torch.abs(l.mean()-m))

The output should be 0 or very close to 0.