Instruction: Loss for binary sparsity

Input:
I have binary images (as the one below) at the output of my net. I need the '1's to be further from each other (not connected), so that they would form a sparse binary image (without white blobs). Something like salt-and-pepper noise. I am looking for a way to define a loss (in pytorch) that would punish based on the density of the '1's.

Thanks.

I

Output:
It depends on how you're generating said image. Since neural networks have to be trained by backpropagation, I'm rather sure your binary image is not the direct output of your neural network (ie not the thing you're applying loss to), because gradient can't blow through binary (discrete) variables. I suspect you do something like pixel-wise binary cross entropy or similar and then threshold.

I assume your code works like that: you densely regress real-valued numbers and then apply thresholding, likely using sigmoid to map from [-inf, inf] to [0, 1]. If it is so, you can do the following. Build a convolution kernel which is 0 in the center and 1 elsewhere, of size related to how big you want your "sparsity gaps" to be.

kernel = [
    [1, 1, 1, 1, 1]
    [1, 1, 1, 1, 1]
    [1, 1, 0, 1, 1]
    [1, 1, 1, 1, 1]
    [1, 1, 1, 1, 1]
]

Then you apply sigmoid to your real-valued output to squash it to [0, 1]:

squashed = torch.sigmoid(nn_output)

then you convolve squashed with kernel, which gives you the relaxed number of non-zero neighbors.

neighborhood = nn.functional.conv2d(squashed, kernel, padding=2)

and your loss will be the product of each pixel's value in squashed with the corresponding value in neighborhood:

sparsity_loss = (squashed * neighborhood).mean()

If you think of this loss applied to your binary image, for a given pixel p it will be 1 if and only if both p and at least one of its neighbors have values 1 and 0 otherwise. Since we apply it to non-binary numbers in [0, 1] range, it will be the differentiable approximation of that.

Please note that I left out some of the details from the code above (like correctly reshaping kernel to work with nn.functional.conv2d).


Instruction: RuntimeError: leaf variable has been moved into the graph interior

Input:
I try to use pytorch for autogradient. When I'm testing is, I met the error. My code is as below:

w11 = torch.rand((100,2), requires_grad=True)
w12 = torch.rand((100,2), requires_grad=True)
w12[:,1] = w12[:,1] + 1
w13 = torch.rand((100,2), requires_grad=True)
w13[:,1] = w13[:,1] + 2
out1=(w11-w12)**2
out2=out1.mean()
out2.backward(retain_graph=True)

Output:
Use with torch.no_grad() when you want to replace something in tensors with requires_grad=True,

w11 = torch.rand((100,2), requires_grad=True)
w12 = torch.rand((100,2), requires_grad=True)
w13 = torch.rand((100,2), requires_grad=True)
with torch.no_grad():
  w12[:,1] = w12[:,1] + 1
  w13[:,1] = w13[:,1] + 2
out1=(w11-w12)**2
out2=out1.mean()
out2.backward(retain_graph=True)

All will go well.


Instruction: What is a pytorch snapshot and a Dlib model? (Running Hopenet)

Input:
I am trying to make Hopenet run on my computer, using the code on github

I have installed all required libraries. This is my forked code, with test_hopenet.py updated to python 3.

I installed all required libraries, and using pip install "pillow<7" because of some old requirements.

python code/test_on_video_dlib.py --snapshot PATH_OF_SNAPSHOT --face_model PATH_OF_DLIB_MODEL --video PATH_OF_VIDEO --output_string STRING_TO_APPEND_TO_OUTPUT --n_frames N_OF_FRAMES_TO_PROCESS --fps FPS_OF_SOURCE_VIDEO

It looks like I am missing some basic understanding which is supposed to be obvious:

What is a SNAPSHOT? Where do I get one?
What is a Dlib Model? Where do I get the right one?

Again - I just want to make this code run, but I can't understand the instructions.

Output:
For the dlib model you can use:
https://github.com/davisking/dlib-models/mmod_human_face_detector.dat.bz2
For the snapshot path you can use hopenet_robust_alpha1.pkl downloadable from the link "300W-LP, alpha 1, robust to image quality" under Pretrained Models section of Nathanial Ruiz's README.md


Instruction: BERT tokenizer & model download

Input:
I`m beginner.. I'm working with Bert. However, due to the security of the company network, the following code does not receive the bert model directly.

tokenizer = BertTokenizer.from_pretrained('bert-base-multilingual-cased', do_lower_case=False)
model = BertForSequenceClassification.from_pretrained("bert-base-multilingual-cased", num_labels=2)

So I think I have to download these files and enter the location manually.
But I'm new to this, and I'm wondering if it's simple to download a format like .py from github and put it in a location.

I'm currently using the bert model implemented by hugging face's pytorch, and the address of the source file I found is:

https://github.com/huggingface/transformers

Please let me know if the method I thought is correct, and if so, what file to get.

Thanks in advance for the comment.

Output:
As described here, what you need to do are download pre_train and configs, then putting them in the same folder. Every model has a pair of links, you might want to take a look at lib code.

For instance

import torch
from transformers import *
model = BertModel.from_pretrained('/Users/yourname/workplace/berts/')

with /Users/yourname/workplace/berts/ refer to your folder

Below are what I found

at src/transformers/configuration_bert.py there are a list of models' configs

BERT_PRETRAINED_CONFIG_ARCHIVE_MAP = {
    "bert-base-uncased": "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-config.json",
    "bert-large-uncased": "https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased-config.json",
    "bert-base-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased-config.json",
    "bert-large-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-cased-config.json",
    "bert-base-multilingual-uncased": "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-uncased-config.json",
    "bert-base-multilingual-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-cased-config.json",
    "bert-base-chinese": "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-chinese-config.json",
    "bert-base-german-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-german-cased-config.json",
    "bert-large-uncased-whole-word-masking": "https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased-whole-word-masking-config.json",
    "bert-large-cased-whole-word-masking": "https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-cased-whole-word-masking-config.json",
    "bert-large-uncased-whole-word-masking-finetuned-squad": "https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased-whole-word-masking-finetuned-squad-config.json",
    "bert-large-cased-whole-word-masking-finetuned-squad": "https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-cased-whole-word-masking-finetuned-squad-config.json",
    "bert-base-cased-finetuned-mrpc": "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased-finetuned-mrpc-config.json",
    "bert-base-german-dbmdz-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-german-dbmdz-cased-config.json",
    "bert-base-german-dbmdz-uncased": "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-german-dbmdz-uncased-config.json",
    "bert-base-japanese": "https://s3.amazonaws.com/models.huggingface.co/bert/cl-tohoku/bert-base-japanese-config.json",
    "bert-base-japanese-whole-word-masking": "https://s3.amazonaws.com/models.huggingface.co/bert/cl-tohoku/bert-base-japanese-whole-word-masking-config.json",
    "bert-base-japanese-char": "https://s3.amazonaws.com/models.huggingface.co/bert/cl-tohoku/bert-base-japanese-char-config.json",
    "bert-base-japanese-char-whole-word-masking": "https://s3.amazonaws.com/models.huggingface.co/bert/cl-tohoku/bert-base-japanese-char-whole-word-masking-config.json",
    "bert-base-finnish-cased-v1": "https://s3.amazonaws.com/models.huggingface.co/bert/TurkuNLP/bert-base-finnish-cased-v1/config.json",
    "bert-base-finnish-uncased-v1": "https://s3.amazonaws.com/models.huggingface.co/bert/TurkuNLP/bert-base-finnish-uncased-v1/config.json",
}

and at src/transformers/modeling_bert.py there are links to pre_trains

BERT_PRETRAINED_MODEL_ARCHIVE_MAP = {
    "bert-base-uncased": "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-pytorch_model.bin",
    "bert-large-uncased": "https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased-pytorch_model.bin",
    "bert-base-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased-pytorch_model.bin",
    "bert-large-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-cased-pytorch_model.bin",
    "bert-base-multilingual-uncased": "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-uncased-pytorch_model.bin",
    "bert-base-multilingual-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-cased-pytorch_model.bin",
    "bert-base-chinese": "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-chinese-pytorch_model.bin",
    "bert-base-german-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-german-cased-pytorch_model.bin",
    "bert-large-uncased-whole-word-masking": "https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased-whole-word-masking-pytorch_model.bin",
    "bert-large-cased-whole-word-masking": "https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-cased-whole-word-masking-pytorch_model.bin",
    "bert-large-uncased-whole-word-masking-finetuned-squad": "https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased-whole-word-masking-finetuned-squad-pytorch_model.bin",
    "bert-large-cased-whole-word-masking-finetuned-squad": "https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-cased-whole-word-masking-finetuned-squad-pytorch_model.bin",
    "bert-base-cased-finetuned-mrpc": "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased-finetuned-mrpc-pytorch_model.bin",
    "bert-base-german-dbmdz-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-german-dbmdz-cased-pytorch_model.bin",
    "bert-base-german-dbmdz-uncased": "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-german-dbmdz-uncased-pytorch_model.bin",
    "bert-base-japanese": "https://s3.amazonaws.com/models.huggingface.co/bert/cl-tohoku/bert-base-japanese-pytorch_model.bin",
    "bert-base-japanese-whole-word-masking": "https://s3.amazonaws.com/models.huggingface.co/bert/cl-tohoku/bert-base-japanese-whole-word-masking-pytorch_model.bin",
    "bert-base-japanese-char": "https://s3.amazonaws.com/models.huggingface.co/bert/cl-tohoku/bert-base-japanese-char-pytorch_model.bin",
    "bert-base-japanese-char-whole-word-masking": "https://s3.amazonaws.com/models.huggingface.co/bert/cl-tohoku/bert-base-japanese-char-whole-word-masking-pytorch_model.bin",
    "bert-base-finnish-cased-v1": "https://s3.amazonaws.com/models.huggingface.co/bert/TurkuNLP/bert-base-finnish-cased-v1/pytorch_model.bin",
    "bert-base-finnish-uncased-v1": "https://s3.amazonaws.com/models.huggingface.co/bert/TurkuNLP/bert-base-finnish-uncased-v1/pytorch_model.bin",
}


Instruction: what does dim=-1 or -2 mean in torch.sum()?

Input:
let me take a 2D matrix as example:

mat = torch.arange(9).view(3, -1)

tensor([[0, 1, 2],
        [3, 4, 5],
        [6, 7, 8]])

torch.sum(mat, dim=-2)

tensor([ 9, 12, 15])

I find the result of torch.sum(mat, dim=-2) is equal to torch.sum(mat, dim=0) and dim=-1 equal to dim=1. My question is how to understand the negative dimension here. What if the input matrix has 3 or more dimensions?

Output: The minus essentially means you go backwards through the dimensions. Let A be a n-dimensional matrix. Then dim=n-1=-1, dim=n-2=-2, ..., dim=1=-(n-1), dim=0=-n. See the numpy doc for more information, as pytorch is heavily based on numpy.


Instruction: What is a dimensional range of [-1,0] in Pytorch?

Input:
So I'm struggling to understand some terminology about collections in Pytorch. I keep running into the same kinds of errors about the range of my tensors being incorrect, and when I try to Google for a solution often the explanations are further confusing.

Here is an example:

m = torch.nn.LogSoftmax(dim=1)
input = torch.tensor([0.3300, 0.3937, -0.3113, -0.2880])
output = m(input)

I don't see anything wrong with the above code, and I've defined my LogSoftmax to accept a 1 dimensional input. So according to my experience with other programming languages the collection [0.3300, 0.3937, -0.3113, -0.2880] is a single dimension.

The above triggers the following error for m(input):

IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

What does that mean?

I passed in a one dimensional tensor, but then it tells me that it was expecting a range of [-1, 0], but got 1.

A range of what?
Why is the error comparing a dimension of 1 to [-1, 0]?
What do the two numbers [-1, 0] mean?

I searched for an explanation for this error, and I find things like this link which make no sense to me as a programmer:

https://github.com/pytorch/pytorch/issues/5554#issuecomment-370456868

So I was able to fix the above code by adding another dimension to my tensor data.

m = torch.nn.LogSoftmax(dim=1)
input = torch.tensor([[-0.3300, 0.3937, -0.3113, -0.2880]])
output = m(input)

So that works, but I don't understand how [-1,0] explains a nested collection.

Further experiments showed that the following also works:

m = torch.nn.LogSoftmax(dim=1)
input = torch.tensor([[0.0, 0.1], [1.0, 0.1], [2.0, 0.1]])
output = m(input)

So dim=1 means a collection of collections, but I don't understand how that means [-1, 0].

When I try using LogSoftmax(dim=2)

m = torch.nn.LogSoftmax(dim=2)
input = torch.tensor([[0.0, 0.1], [1.0, 0.1], [2.0, 0.1]])
output = m(input)

The above gives me the following error:

  IndexError: Dimension out of range (expected to be in range of [-2, 1], but got 2)

Confusion again that dim=2 equals [-2, 1], because where did the 1 value come from?

I can fix the error above by nesting collections another level, but at this point I don't understand what values LogSoftmax is expecting.

m = torch.nn.LogSoftmax(dim=2)
input = torch.tensor([[[0.0, 0.1]], [[1.0, 0.1]], [[2.0, 0.1]]])
output = m(input)

I am super confused by this terminology [-1, 0] and [-2, 1]?

If the first value is the nested depth, then why is it negative and what could the second number mean?

There is no error code associated with this error. So it's been difficult to find documentation on the subject. It appears to be an extremely common error people get confused by and nothing that I can find in the Pytorch documentation that talks specifically about it.

Output:
When specifying a tensor's dimension as an argument for a function (e.g. m = torch.nn.LogSoftmax(dim=1)) you can either use positive dimension indexing starting with 0 for the first dimension, 1 for the second etc.
Alternatively, you can use negative dimension indexing to start from the last dimension to the first: -1 indicate the last dimension, -2 the second from last etc.

Example:
If you have a 4D tensor of dimensions b-by-c-by-h-by-w then

The "batch" dimension (the first) can be accessed as either dim=0 or dim=-4.
The "channel" dimension (the second) can be accessed as either dim=1 or dim=-3.
The "height"/"vertical" dimension (the third) can be accessed as either dim=2 or dim=-2.
The "width"/"horizontal" dimension (the fourth) can be accessed as either dim=3 or dim=-1.

Therefore, if you have a 4D tensor dim argument can take values in the range [-4, 3].

In your case you have a 1D tensor and therefore dim argument can be wither 0 or -1 (which in this deprecate case amounts to the same dimension).


Instruction: Pytorch AttributeError: module 'torch' has no attribute 'as_tensor'

Input:
$ python main.py --hetero
Created directory results/ACMRaw_2020-01-13_01-20-26
Traceback (most recent call last):
  File "main.py", line 101, in <module>
    main(args)
  File "main.py", line 30, in main
    val_mask, test_mask = load_data(args['dataset'])
  File "/home/cnudi1/wook/dgl/examples/pytorch/han/utils.py", line 225, in load_data
    return load_acm_raw(remove_self_loop)
  File "/home/cnudi1/wook/dgl/examples/pytorch/han/utils.py", line 189, in load_acm_raw
    pa = dgl.bipartite(p_vs_a, 'paper', 'pa', 'author')
  File "/home/cnudi1/.conda/envs/lcr_env/lib/python3.6/site-packages/dgl-0.4-py3.6-linux-ppc64le.egg/dgl/convert.py", line 260, in bipartite
    return create_from_scipy(data, utype, etype, vtype)
  File "/home/cnudi1/.conda/envs/lcr_env/lib/python3.6/site-packages/dgl-0.4-py3.6-linux-ppc64le.egg/dgl/convert.py", line 823, in create_from_scipy
    indptr = utils.toindex(spmat.indptr)
  File "/home/cnudi1/.conda/envs/lcr_env/lib/python3.6/site-packages/dgl-0.4-py3.6-linux-ppc64le.egg/dgl/utils.py", line 242, in toindex
    return data if isinstance(data, Index) else Index(data)
  File "/home/cnudi1/.conda/envs/lcr_env/lib/python3.6/site-packages/dgl-0.4-py3.6-linux-ppc64le.egg/dgl/utils.py", line 15, in __init__
    self._initialize_data(data)
  File "/home/cnudi1/.conda/envs/lcr_env/lib/python3.6/site-packages/dgl-0.4-py3.6-linux-ppc64le.egg/dgl/utils.py", line 22, in _initialize_data
    self._dispatch(data)
  File "/home/cnudi1/.conda/envs/lcr_env/lib/python3.6/site-packages/dgl-0.4-py3.6-linux-ppc64le.egg/dgl/utils.py", line 75, in _dispatch
    self._user_tensor_data[F.cpu()] = F.zerocopy_from_numpy(self._pydata)
  File "/home/cnudi1/.conda/envs/lcr_env/lib/python3.6/site-packages/dgl-0.4-py3.6-linux-ppc64le.egg/dgl/backend/pytorch/tensor.py", line 276, in zerocopy_from_numpy
    return th.as_tensor(np_array)
AttributeError: module 'torch' has no attribute 'as_tensor'

I got an error when I try to run the code (https://github.com/dmlc/dgl/blob/master/examples/pytorch/han/main.py)

from the DGL
(https://github.com/dmlc/dgl)

It requires CUDA and Pytorch so I managed to install it.

But I got an error and couldn't find the solution with Google/Stackoverflow search

My environment is

Linux minsky 3.10.0-957.5.1.el7.ppc64le
CentOS
Python 3.6.9
Conda 4.5.11
CUDA 10.1
NVCC 10.1
Pytorch 0.4.0
Torchvision 0.2.1

Pytorch works fine in Python

>>> import torch
>>> print (torch.__version__)
0.4.0
>>> import torchvision
>>> print (torchvision.__version__)
0.2.1

Please could you help me out?

* DGL is installed from the source code

** Pytorch is installed with conda from channel:engility(How to install pytorch on Power 8 or PPC64 machine?)

conda install -c engility pytorch

because other ways(default conda, pip, install from the source code) never works for ppc64le

Output:
tl;dr Upgrade to PyTorch 0.4.1

Notice that DGL requires PyTorch 0.4.1 and you are using PyTorch 0.4.0. If you take a closer look, you'll see that as_tensor was proposed in 30 Apr 2018 and merged in 1 May 2018. You'll also see that PyTorch 0.4.0 was released before that on 24 Apr 2018, whereas PyTorch 0.4.1 was release after on 26 Jul 2018. In fact, if you take a look at the changelog of the 0.4.1 version, you'll notice a new operator being announced: torch.as_tensor :)


Instruction: Is there any essential difference between * and + for Pytorch autograd?

Input:
I was trying to understand the autograd mechanism in more depth. To test my understanding, I tried to write the following code which I expected to produce an error (i.e., Trying to backward through the graph a second time).

b = torch.Tensor([0.5])
for i in range(5):
    b.data.zero_().add_(0.5)
    b = b + a
    c = b*a
    c.backward()

Apparently, it should report an error when c.backward() is called for the second time in the for loop, because the history of b has been freed， however, nothing happens.

But when I tried to change b + a to b * a as follows,

b = torch.Tensor([0.5])
for i in range(5):
    b.data.zero_().add_(0.5)
    b = b * a
    c = b*a
    c.backward()

It did report the error I was expecting.
This looks pretty weird to me. I don't understand why there is no error evoked for the former case, and why it makes a difference to change from + to *.

Output: The difference is adding a constant doesn't change the gradient, but mull by const does. It seems, autograd is aware of it, and optimizes out 'b = b + a'.


Instruction: How can I cast a tensor to the complex type in Pytorch?

Input: I want to do some quantum mechanics calculations with Pytorch, where the quantities are sometimes complex. I would like to know how can I cast an existing real tensor to the complex type.

Output:
PyTorch does have complex number support.

Try this:

import torch

a = torch.tensor([1.0, 2.0], dtype=torch.double)
b = a.type(torch.complex64)


Instruction: What are C classes for a NLLLoss loss function in Pytorch?

Input:
I'm asking about C classes for a NLLLoss loss function.

The documentation states:

  The negative log likelihood loss. It is useful to train a classification problem with C classes.

Basically everything after that point depends upon you knowing what a C class is, and I thought I knew what a C class was but the documentation doesn't make much sense to me. Especially when it describes the expected inputs of (N, C) where C = number of classes. That's where I'm confused, because I thought a C class refers to the output only. My understanding was that the C class was a one hot vector of classifications. I've often found in tutorials that the NLLLoss was often paired with a LogSoftmax to solve classification problems.

I was expecting to use NLLLoss in the following example:

# Some random training data
input = torch.randn(5, requires_grad=True)
print(input)  # tensor([-1.3533, -1.3074, -1.7906,  0.3113,  0.7982], requires_grad=True)
# Build my NN (here it's just a LogSoftmax)
m = nn.LogSoftmax(dim=0)
# Train my NN with the data
output = m(input)
print(output)  # tensor([-2.8079, -2.7619, -3.2451, -1.1432, -0.6564], grad_fn=<LogSoftmaxBackward>)
loss = nn.NLLLoss()
print(loss(output, torch.tensor([1, 0, 0])))

The above raises the following error on the last line:

  ValueError: Expected 2 or more dimensions (got 1)

We can ignore the error, because clearly I don't understand what I'm doing. Here I'll explain my intentions of the above source code.

input = torch.randn(5, requires_grad=True)

Random 1D array to pair with one hot vector of [1, 0, 0] for training. I'm trying to do a binary bits to one hot vector of decimal numbers.

m = nn.LogSoftmax(dim=0)

The documentation for LogSoftmax says that the output will be the same shape as the input, but I've only seen examples of LogSoftmax(dim=1) and therefore I've been stuck trying to make this work because I can't find a relative example.

print(loss(output, torch.tensor([1, 0, 0])))

So now I have the output of the NN, and I want to know the loss from my classification [1, 0, 0]. It doesn't really matter in this example what any of the data is. I just want a loss for a one hot vector that represents classification.

At this point I get stuck trying to resolve errors from the loss function relating to expected output and input structures. I've tried using view(...) on the output and input to fix the shape, but that just gets me other errors.

So this goes back to my original question and I'll show the example from the documentation to explain my confusion:

m = nn.LogSoftmax(dim=1)
loss = nn.NLLLoss()
input = torch.randn(3, 5, requires_grad=True)
train = torch.tensor([1, 0, 4])
print('input', input)  # input tensor([[...],[...],[...]], requires_grad=True)
output = m(input)
print('train', output, train)  # tensor([[...],[...],[...]],grad_fn=<LogSoftmaxBackward>) tensor([1, 0, 4])
x = loss(output, train)

Again, we have dim=1 on LogSoftmax which confuses me now, because look at the input data. It's a 3x5 tensor and I'm lost.

Here's the documentation on the first input for the NLLLoss function:

  Input: (N, C)(N,C) where C = number of classes

The inputs are grouped by the number of classes?

So each row of the tensor input is associated with each element of the training tensor?

If I change the second dimension of the input tensor, then nothing breaks and I don't understand what is going on.

input = torch.randn(3, 100, requires_grad=True)
# 3 x 100 still works?

So I don't understand what a C class is here, and I thought a C class was a classification (like a label) and meaningful only on the outputs of the NN.

I hope you understand my confusion, because shouldn't the shape of the inputs for the NN be independent from the shape of the one hot vector used for classification?

Both the code examples and documentations say that the shape of the inputs is defined by the number of classifications, and I don't really understand why.

I have tried to study the documentations and tutorials to understand what I'm missing, but after several days of not being able to get past this point I've decided to ask this question. It's been humbling because I thought this was going to be one of the easier things to learn.

Output:
Basically you are missing a concept of batch.

Long story short, every input to loss (and the one passed through the network) requires batch dimension (i.e. how many samples are used).

Breaking it up, step by step:

Your example vs documentation

Each step will be each step compared to make it clearer (documentation on top, your example below)

Inputs

input = torch.randn(3, 5, requires_grad=True)
input = torch.randn(5, requires_grad=True)

In the first case (docs), input with 5 features is created and 3 samples are used. In your case there is only batch dimension (5 samples), you have no features which are required. If you meant to have one sample with 5 features you should do:

input = torch.randn(5, requires_grad=True)

LogSoftmax

LogSoftmax is done across features dimension, you are doing it across batch.

m = nn.LogSoftmax(dim=1) # apply over features
m = nn.LogSoftmax(dim=0) # apply over batch

It makes no sense usually for this operation as samples are independent of each other.

Targets

As this is multiclass classification and each element in vector represents a sample, one can pass as many numbers as one wants (as long as it's smaller than number of features, in case of documentation example it's 5, hence [0-4] is fine ).

train = torch.tensor([1, 0, 4])
train = torch.tensor([1, 0, 0])

I assume, you wanted to pass one-hot vector as target as well. PyTorch doesn't work that way as it's memory inefficient (why store everything as one-hot encoded when you can just pinpoint exactly the class, in your case it would be 0).

Only outputs of neural network are one hot encoded in order to backpropagate error through all output nodes, it's not needed for targets.

Final

You shouldn't use torch.nn.LogSoftmax at all for this task. Just use torch.nn.Linear as last layer and use torch.nn.CrossEntropyLoss with your targets.


Instruction: Using CNN with Dataset that has different depths between volumes

Input:
I am working with Medical Images, where I have 130 Patient Volumes, each volume consists of N number of  DICOM Images/slices.

The problem is that between the volumes the the number of slices N, varies.

Majority, 50% of volumes have 20 Slices, rest varies by 3 or 4 slices, some even more than 10 slices (so much so that interpolation to make number of slices equal between volumes is not possible)

I am able to use Conv3d for volumes where the depth N (number of slices) is same between volumes, but I have to make use of entire data set for the classification task. So how do I incorporate entire dataset and feed it to my network model ?

Output:
If I understand your question, you have 130 3-dimensional images, which you need to feed into a 3D ConvNet. I'll assume your batches, if N was the same for all of your data, would be tensors of shape (batch_size, channels, N, H, W), and your problem is that your N varies between different data samples.

So there's two problems. First, there's the problem of your model needing to handle data with different values of N. Second, there's the more implementation-related problem of batching data of different lengths.

Both problems come up in video classification models. For the first, I don't think there's a way of getting around having to interpolate SOMEWHERE in your model (unless you're willing to pad/cut/sample) -- if you're doing any kind of classification task, you pretty much need a constant-sized layer at your classification head. However, the interpolation doesn't have happen right at the beginning. For example, if for an input tensor of size (batch, 3, 20, 256, 256), your network conv-pools down to (batch, 1024, 4, 1, 1), then you can perform an adaptive pool (e.g. https://pytorch.org/docs/stable/nn.html#torch.nn.AdaptiveAvgPool3d) right before the output to downsample everything larger to that size before prediction.

The other option is padding and/or truncating and/or resampling the images so that all of your data is the same length. For videos, sometimes people pad by looping the frames, or you could pad with zeros. What's valid depends on whether your length axis represents time, or something else.

For the second problem, batching: If you're familiar with pytorch's dataloader/dataset pipeline, you'll need to write a custom collate_fn which takes a list of outputs of your dataset object and stacks them together into a batch tensor. In this function, you can decide whether to pad or truncate or whatever, so that you end up with a tensor of the correct shape. Different batches can then have different values of N. A simple example of implementing this pipeline is here: https://github.com/yunjey/pytorch-tutorial/blob/master/tutorials/03-advanced/image_captioning/data_loader.py

Something else that might help with batching is putting your data into buckets depending on their N dimension. That way, you might be able to avoid lots of unnecessary padding.


Instruction: Installing OpenCV into PyCharm

Input:
Idk if this is a stackoverflow-appropriate post so forgive if the question is misplaced.  I'm trying to install OpenCV into my Pycharm IDE through the conda virtual environment.  I typed conda install -c conda-forge opencv inside the PyCharm terminal and it has been doing this for 11 hours and God knows how many more to go.

Pycharm did this with PyTorch as well.  Am I doing something wrong or is this normal?

Output:
While you can install packages directly in PyCharm by going to file->settings select Project Interpreter and click on the '+' icon on the top right (see image)

I would recommend creating a requirements.txt file in the root of your project, and write down all your required packages. When a package is missing, PyCharm will automatically suggest to install the package for you.

e.g. for installing opencv you can add the following to you requirements.txt

opencv-python

Or even specify the version that your project needs

opencv-python==4.1.2

edit: the advantage of using a requirement.txt is that you can more easily port the project to another machine, and re-install the packages if needed.


Instruction: Finding parameters with backpropagation and gradient descent in PyTorch

Input:
I am experimenting with PyTorch and autodifferentiation and gradient descent

To that end I would like to estimate the parameters that would produce a certain value to an arbitrary linear in the parameters function.

My code is here:

import torch

X = X.astype(float)

X = np.array([[3.], [4.], [5.]])

X = torch.from_numpy(X)

X.requires_grad = True

W = np.random.randn(3,3)

W = np.triu(W, k=0)

W = torch.from_numpy(W)

W.requires_grad = True

out = 10 - (X@torch.transpose(X, 1,0) * W).sum()

out is :

My objective is to make out close to 0 (within an interval of [-.00001 , 0.0001]) by adjusting W using the gradient of W.

How should I proceed from here to achieve this end with pytorch?

Update

@Umang: this is what I get when I run the code you propose:

In fact the algorithm diverges.

Output:
# your code as it is
import torch
import numpy as np

X = np.array([[3.], [4.], [5.]])
X = torch.from_numpy(X)
X.requires_grad = True
W = np.random.randn(3,3)
W = np.triu(W, k=0)
W = torch.from_numpy(W)
W.requires_grad = True

# define parameters for gradient descent
max_iter=100
lr_rate = 1e-3

# we will do gradient descent for max_iter iteration, or convergence till the criteria is met.
i=0
out = compute_out(X,W)
while (i<max_iter) and (torch.abs(out)>0.01):
    loss = (out-0)**2
    W = W - lr_rate*torch.autograd.grad(loss, W)[0]
    i+=1
    print(f"{i}: {out}")
    out = compute_out(X,W)

print(W)

We define a loss function such that its minima is at the desired point and run gradient descent. Here, I have used squared-error but you may use other loss functions too with desired minima.


Instruction: Loading ResNet50 on RTX2070 - Out of Memory

Input:
I'm trying to load ResNext50, and on top of it CenterNet, I'm able to do it with Google Colab or Kaggle's GPU. But,

Would love to know how much GPU Memory (VRAM) does this network need?
When using RTX 2070 with free 5.5GB VRAM left on it (out of 8GB), I'm not able to load it.

Batch size is 1, #of workers is 1, everything is set to minimum values. OS: Ubuntu 18.04 (Using PyTorch)

In TensorFlow, I know that I can restrict the amount of VRAM (which enables me to load and run networks although I don't have enough VRAM), but in PyTorch I didn't find this functionality yet.

Any ideas how to solve this?

Output:
Using third party dependency

You could get size of model in bytes using third party library torchfunc (disclaimer I'm the author).

import torchfunc

# Assuming model is loaded
print(torchfunc.sizeof(model))

No dependencies

This function is pretty simple and shirt, you could just copy it, see source code

Exact size

It's just size of your model, there is more VRAM memory usage during forward and backward and depends on size of your batch. You could try pytorch_modelsize library for estimations of those (not sure whether it will work for your networks though).

Clear cache

You should clear your cache before running network (sometimes restarting workstation helped in my case), as you definitely have enough memory. Killing processes using GPU should help as well.

Once again torchfunc could help, issue the following command:

import torchfunc

torchfunc.cuda.reset()


Instruction: I want to show accuracy under each picture output shown in Image Classification with Transfer Learning in PyTorch

Input:
I'm follow this link : https://stackabuse.com/image-classification-with-transfer-learning-and-pytorch/#settingupapretrainedmodel

but I'm new with coding skill. please tell me how to show accuracy value under image.

Output:
The model used in that example returns a tensor of logits of shape (batch size, classes). Assuming what you mean by "accuracy value" is the predicted probability of the class with the largest probability, what you need to do is first compute your probabilities by taking the SoftMax of the output from the model, which gives the predicted probabilities for each image in your batch. Their visualize_model function would look something like the following, though I haven't tested it.

def visualize_model(model, num_images=6):
    was_training = model.training
    model.eval()
    images_handeled = 0
    fig = plt.figure()

    with torch.no_grad():
        for i, (inputs, labels) in enumerate(dataloaders['val']):
            inputs = inputs.to(device)
            labels = labels.to(device)

            outputs = model(inputs)
            probabilities = nn.functional.softmax(outputs, dim=-1) # compute probabilities
            _, preds = torch.max(outputs, 1)

            for j in range(inputs.size()[0]):
                images_handeled += 1
                ax = plt.subplot(num_images//2, 2, images_handeled)
                ax.axis('off')
                ax.set_title('predicted: {}, probability: {}'.format(class_names[preds[j]], probabilities[preds[j]])) # add predicted class probability
                imshow(inputs.cpu().data[j])

                if images_handeled == num_images:
                    model.train(mode=was_training)
                    return
        model.train(mode=was_training)

Or do you mean overall classification accuracy?


Instruction: Using `DataParallel` when network needs a shared (constant) `Tensor`

Input:
I would like to use DataParallel to distribute my computations across multiple GPUs along the batch dimension. My network requires a Tensor (let's call it A) internally, which is constant and doens't change through the optimization. It seems that DataParallel does not automatically copy this Tensor to all the GPUs in question, and the network will thus complain that the chunk of the input data x that it sees resides on a different GPU than A.

Is there a way DataParallel can handle this situation automatically? Alternatively, is there a way to copy a Tensor to all GPUs? Or should I just keep one Tensor for each GPU and manually figure out which copy to use depending on where the chunk seen by forward resides?

Output:
You should wrap your tensor in torch.nn.Parameter and set requires_grad=False during it's creation.

torch.nn.Parameter does not mean the tensor has to be trainable.

It merely means it is part of the model and should be transferred if needed (e.g. multiple GPU).

If that wasn't the case, there is no way for torch to know which tensor inside __init__ is part of model (you could do some operations on tensors and add to self just to get something done).

I don't see a need for another function to do just that, though the name might be confusing a little bit.


Instruction: Image clustering - allocating memory on GPU

Input:
I've written this code for image classification by pretrained googlenet:

gnet = models.googlenet(pretrained=True).cuda()

transform = transforms.Compose([transforms.Resize(256), transforms.CenterCrop(32), transforms.ToTensor()])
images = {}
resultDist = {}
i = 1

for f in glob.iglob("/data/home/student/HW3/trainData/train2014/*"):
    print(i)
    i = i + 1
    image = Image.open(f)
    # transform, create batch and get gnet weights
    img_t = transform(image).cuda()
    batch_t = torch.unsqueeze(img_t, 0).cuda()
    try:
        gnet.eval()
        out = gnet(batch_t)
        resultDist[f[-10:-4]] = out
        del out
    except:
        print(img_t.shape)
    del img_t
    del batch_t
    image.close()
    torch.cuda.empty_cache()
    i = i + 1

torch.save(resultDist, '/data/home/student/HW3/googlenetOutput1.pkl')

I deleted all the possible tensors from the GPU after using them, but after about 8000 images from my dataset the GPU is full. I found the problem to be in:

resultDist[f[-10:-4]] = out

The dictionary taking alot of space and I can't delete it because I want to save my data to pkl file.

Output:
Since you're not doing backprop wrap your whole loop with a with torch.no_grad(): statement since otherwise a computation graph is created and intermittent results may be stored on the GPU for later application of backprop. This takes a fair amount of space. Also you probably want to save out.cpu() so your results aren't left on the GPU.

...
with torch.no_grad():
    for f in glob.iglob("/data/home/student/HW3/trainData/train2014/*"):
        ...
            resultDist[f[-10:-4]] = out.cpu()
        ...

torch.save(resultDist, '/data/home/student/HW3/googlenetOutput1.pkl')


Instruction: no reference to Module.parameters() after using more than once

Input:
I have a class that inherits from torch.nn.Module,

now when I do this code:

d = net.parameters()
print(len(list(d)))
print(len(list(d)))
print(len(list(d)))

the output is:

10
0
0

So I have reference to the net.parameters() only once, whys that?

Then it apparently disappear..
I got this error while trying to make my own Optimizer, so I pass this net.parameters() as a parameter to my new class, and apparently I couldn't use it because of that odd situation.

Output:
This is working as expected. Module.parameters() returns an iterator, more specifically, a Python generator. One thing about them is that you cannot rewind a generator. So, in the first list(d) call, you are actually "consuming" all the generator. Then, if you try to do that again, it will be empty.

If you're wondering, the .parameters() implementation can be seen here, and it is very simple:

def parameters(self, recurse=True):
    for name, param in self.named_parameters(recurse=recurse):
        yield param

Perhaps it is easier to wrap your mind around it with this toy example:

def g():
    for x in [0, 1, 2, 3, 4]:
        yield x

d = g()
print(list(d))  # prints: [0, 1, 2, 3, 4]
print(list(d))  # prints: []


Instruction: Tensor Entry Selection Logic Divergence in PyTorch & Numpy

Input:
Description

I'm setting up a torch.Tensor for masking purpose. When attempting to select entries by indices, it turns out that behaviors between using numpy.ndarray and torch.Tensor to hold index data are different. I would like to have access to the design in both frameworks and related documents that explain the difference.

Steps to replicate

Environment

Pytorch 1.3 in container from official release: pytorch/pytorch:1.3-cuda10.1-cudnn7-devel

Example

Say I need to set up mask as torch.Tensor object with shape [3,3,3] and set values at entries (0,0,1) & (1,2,0) to 1. The code below explains the difference.

mask = torch.zeros([3,3,3])
indices = torch.tensor([[0, 1],
                        [0, 2],
                        [1, 0]])

mask[indices.numpy()] = 1 # Works
# mask[indices] = 1 # Incorrect result

I noticed that when using mask[indices.numpy()] a new torch.Tensor of shape [2], while mask[indices] returns a new torch.Tensor of shape [3, 2, 3, 3], which suggests difference in tensor slicing logic.

Output:
You get different results because that's how indexing is implemented in Pytorch. If you pass an array as index, then it gets "unpacked". For example:

indices = torch.tensor([[0, 1], [0, 2], [1, 0]])

mask = torch.arange(1,28).reshape(3,3,3)

# tensor([[[ 1,  2,  3],
#          [ 4,  5,  6],
#          [ 7,  8,  9]],

#         [[10, 11, 12],
#          [13, 14, 15],
#          [16, 17, 18]],

#         [[19, 20, 21],
#          [22, 23, 24],
#          [25, 26, 27]]])

The mask[indices.numpy()] is equivalent to mask[[0, 1], [0, 2], [1, 0]], i.e. the elements of the i-th row of indices.numpy() are used to select elements of mask along i-th axis. So it returns tensor([mask[0,0,1], mask[1,2,0]]), i.e. tensor([2, 16]).

On the other hand, when passing a tensor as index (I don't know the exact reason for this differentiation between arrays and tensors for indexing), it is not "unpacked" like an array, and the elements of the i-th row of the indices tensor are used for selecting the elements of mask along the axis-0. That is, mask[indices] is equivalent to mask[[[0, 1], [0, 2], [1, 0]], :, :]

>>> mask[ind]

tensor([[[[ 1,  2,  3],
          [ 4,  5,  6],
          [ 7,  8,  9]],

         [[10, 11, 12],
          [13, 14, 15],
          [16, 17, 18]]],

        [[[ 1,  2,  3],
          [ 4,  5,  6],
          [ 7,  8,  9]],

         [[19, 20, 21],
          [22, 23, 24],
          [25, 26, 27]]],

        [[[10, 11, 12],
          [13, 14, 15],
          [16, 17, 18]],

         [[ 1,  2,  3],
          [ 4,  5,  6],
          [ 7,  8,  9]]]])

which is basically tensor(mask[[0,1], :, :], mask[[0,2],: ,:], mask[[1,0], :, :]) and has shape indices.shape + mask[0,:,:].shape == (3,2,3,3). So whole "sheets" are selected and stacked into new dimensions. Note that this is not a new tensor, but a special view of mask. Therefore if you assign mask[indices] = 1, with this particular indices, then all the elements of mask will become 1.


Instruction: How to specify pytorch / cuda version in pipenv

Input:
I am trying to install a specific version of pytorch that is compatible with a specific cuda driver version with pipenv. The pytorch website shows how to to this with pip:

pip3 install torch==1.3.1+cu92 torchvision==0.4.2+cu92 -f https://download.pytorch.org/whl/torch_stable.html

I tried to convert this into an entry in my Pipfile like this:

[[source]]
name = "pytorch"
url = "https://download.pytorch.org/whl/torch_stable.html"
verify_ssl = false

pytorch = {version="==1.3.1+cu92", index="pytorch"}
torchvision = {version="==0.4.2+cu92", index="pytorch"}

However, this does not work. The dependency with this version can not be resolved. I am not sure if the url that is listed with the -f parameter in the pip3 command is even a valid source for pipenv.

I could install both libraries by just passing the command through to pip like this:

pipenv run pip install torch==1.3.1+cu92 torchvision==0.4.2+cu92 -f https://download.pytorch.org/whl/torch_stable.html

but I am not really satisfied with that solution since the dependencies are not in the Pipfile and I have to manually document the usage of this command.

Output:
The problem with the approach above lies in the structure of https://download.pytorch.org/whl/torch_stable.html. Pipenv can only find torch versions 0.1 to 0.4.1 because all others have the cuda (or cpu) version as a prefix e.g. cu92/torch-0.4.1-cp27-cp27m-linux_x86_64.whl.
But the cuda version is a subdirectory. So if you change the url of the source to the cuda version and only specify the torch version in the dependencies it works.
[[source]]
name = "pytorch"
url = "https://download.pytorch.org/whl/cu92"
verify_ssl = false

[packages]
torch = {index = "pytorch",version = "==1.4.0"}

The only problem I encountered is that numpy is not recognized as a dependency of pytoch 1.4.0. But this seems to be a problem of the specific pytorch wheel. With version 1.3.1 or 1.5.1 and a recent pipenv version it works.
So if after the installation with pipenv install, the command pipenv run python -c "import torch" throws an error, numpy must be added manually.


Instruction: Error while downloading pre-trained GAN model in Pytorch : 'memory' file not found

Input:
I was following the steps given in https://modelzoo.co/model/pytorch-cyclegan-and-pix2pix to download a pre-trained model.
These where the first 3 commands given there:
git clone https://github.com/pytorch/vision
cd vision
**python setup.py install**

However when I ran the third line, I got an error:
fatal error: 'memory' file not found

#include <memory>

error: command 'gcc' failed with exit status 1

If anyone has some idea on how to overcome this error, it would be really helpful.

Output:
See this post. Try the line

apt-get install build-essential


Instruction: Repeating a pytorch tensor without copying memory

Input:
Does pytorch support repeating a tensor without allocating significantly more memory?

Assume we have a tensor

t = torch.ones((1,1000,1000))
t10 = t.repeat(10,1,1)

Repeating t 10 times will require take 10x the memory. Is there a way how I can create a tensor t10 without allocating significantly more memory?

Here is a related question, but without answers.

Output:
You can use torch.expand

t = torch.ones((1, 1000, 1000))
t10 = t.expand(10, 1000, 1000)

Keep in mind that the t10 is just a reference to t. So for example, a change to t10[0,0,0] will result in the same change in t[0,0,0] and every member of t10[:,0,0].

Other than direct access, most operations performed on t10 will cause memory to be copied which will break the reference and cause more memory to be used. For example: changing the device (.cpu(), .to(device=...), .cuda()), changing the datatype (.float(), .long(), .to(dtype=...)), or using .contiguous().


Instruction: How does torch.distributed.barrier() work

Input:
I've read all the documentations I could find about torch.distributed.barrier(), but still having trouble understanding how it's being used in this script and would really appreciate some help.

So the official doc of torch.distributed.barrier says it "Synchronizes all processes.This collective blocks processes until the whole group enters this function, if async_op is False, or if async work handle is called on wait()."

It's used in two places in the script:

First place

    if args.local_rank not in [-1, 0] and not evaluate:
        torch.distributed.barrier()  # Make sure only the first process in distributed training process the dataset, and the others will use the cache

        ... (preprocesses the data and save the preprocessed data)

    if args.local_rank == 0 and not evaluate:
        torch.distributed.barrier()

Second place

    if args.local_rank not in [-1, 0]:
        torch.distributed.barrier()  # Make sure only the first process in distributed training will download model & vocab

        ... (loads the model and the vocabulary)

    if args.local_rank == 0:
        torch.distributed.barrier()  # Make sure only the first process in distributed training will download model & vocab

I'm having trouble relating the comment in the code to the functionality of this function stated in the official doc. How does it make sure only the first process executes the code between the two calls of torch.distributed.barrier() and why it only checks whether the local rank is 0 before the second call?

Thanks in advance!

Output:
First you need to understand the ranks. To be brief: in a multiprocessing context we typically assume that rank 0 is the first process or base process. The other processes are then ranked differently, e.g. 1, 2, 3, totalling four processes in total.
Some operations are not necessary to be done in parallel or you just need one process to do some preprocessing or caching so that the other processes can use that data.
In your example, if the first if statement is entered by the non-base processes (rank 1, 2, 3), they will block (or "wait") because they run into the barrier. They wait there, because barrier() blocks until all processes have reached a barrier, but the base process has not reached a barrier yet.
So at this point the non-base processes (1, 2, 3) are blocked, but the base process (0) continues. The base process will do some operations (preprocess and cache data, in this case) until it reaches the second if-statement. There, the base process will run into a barrier. At this point, all processes have stopped at a barrier, meaning that all current barriers can be lifted and all processes can continue. Because the base process prepared the data, the other processes can now use that data.
Perhaps the most important thing to understand is:

when a process encounters a barrier it will block
the position of the barrier is not important (not all processes have to enter the same if-statement, for instance)
a process is blocked by a barrier until all processes have encountered a barrier, upon which those barriers are lifted for all processes


Instruction: Is there a way to remove 1-2 (or more) specific neuron connections between the layers in the NN using Tensorflow (with Keras) or PyTorch?

Input:
I am in the process of making a GUI for creating different unique Neural Networks. I am down for using TensorFlow 2.0 (with Keras API) or PyTorch as the back-end. But I am lacking information on those topics, and would be super grateful if anyone can answer these:

Language: Python

1) How to remove specific neuron connections between the layers in NN using any of these frameworks?
2) How to set the specific learning rule for some of the neurons in the layer?
3) How to set the specific activating function for some of the neurons in the layer?

Huge thanks, please don't hesitate to answer, any information is useful.

P.S. If anyone wants to contribute to the project, It would be awesome.

Output:
The way these libraries work is that the connections/weights for a layer are represented as a "tensor" (i.e. a multi-dimensional array or matrix). This makes the application of a layer behave as a linear algebra operation (a matrix multiplication). PyTorch/Tensorflow aren't representing the individual neural connections as distinct objects in the code in a way that makes sense to think about them as something to be operated on or deleted individually.

  1) How to remove specific neuron connections between the layers in NN using any of these frameworks?

You could set one of the weights to zero, i.e. layer.weights[x, y]=0, though that doesn't actually "remove" it or prevent it from later being changed to non-zero.

Maybe you could use a sparse tensor instead of dense, which is a coordinate format containing a list of all non-zero indices and values. Sparse is only going to be more efficient if you have a low percent non-zero values.

  2) How to set the specific learning rule for some of the neurons in the layer?

By learning rule do you mean optimizers? You can find other posts about multiple optimizers, which would probably work out to be similar to (3) below. E.g.

https://discuss.pytorch.org/t/two-optimizers-for-one-model/11085

  3) How to set the specific activating function for some of the neurons in the layer?

Operators and activation functions are typically implemented to efficiently operate on a full tensor. You could split a layer into two separate smaller layers, and run them next to each other (at the same level in the network).

E.g. if you had

layer1=torch.nn.Linear(10, 10)

but instead of just torch.relu(layer1(input)) you wanted to apply relu to some outputs and, say, sigmoid to others, you could just:

layer1a = torch.nn.Linear(10, 5)
layer2b = torch.nn.Linear(10, 5)

and then

torch.cat( (torch.relu(layer1a(x)), torch.sigmoid(layer1b(x)) ), 0)

Similarly you can split any tensor into pieces, apply various functions to different ranges/values, and stitch the results back together with torch.cat.


Instruction: How to select correct value of learning rate multiplier?

Input: I want to manually select correct Learning rate in an image classification problem using Pytorch by running the model for few epochs. I have used LR scheduler to decay the learning rate and also have manipulated Learning rate in optimizer parameter group but i am unable to see any change in Loss.

Output:
Adjusting the learning rate and finding "the one" can be very tedious and time consuming.

Luckily for you, you are not the first one to be bothered by this issue and there are several approaches to adjust the learning rate in a more systematic way.
To name just two of these methods:

Hyper-gradient descent - treating the learning rate as a parameter and "learning" it as well.
Cyclical learning rates - searching for an "optimal" learning rate within an interval.


Instruction: How to implement LadderNet (2 U-Nets) in Keras? (With available PyTorch script as reference)

Input:
I am trying to implement the architecture of LadderNet (https://arxiv.org/abs/1810.07810) in Keras, with only the PyTorch version available as reference. The architecture in the paper is comprised of 2 U-Nets:

The codes for the PyTorch implementation of LadderNet's architecture (obtained from https://github.com/juntang-zhuang/LadderNet/blob/master/src/LadderNetv65.py) and Keras' implementation of U-Net (obtained from https://github.com/zhixuhao/unet/blob/master/model.py) are respectively:

drop = 0.25

def conv3x3(in_planes, out_planes, stride=1):
    """3x3 convolution with padding"""
    return nn.Conv2d(in_planes, out_planes, kernel_size=3, stride=stride,
                     padding=1, bias=True)

class BasicBlock(nn.Module):
    expansion = 1

    def __init__(self, inplanes, planes, stride=1, downsample=None):
        super(BasicBlock, self).__init__()
        if inplanes!= planes:
            self.conv0 = conv3x3(inplanes,planes)

        self.inplanes = inplanes
        self.planes = planes

        self.conv1 = conv3x3(planes, planes, stride)
        #self.bn1 = nn.BatchNorm2d(planes)
        self.relu = nn.ReLU(inplace=True)
        #self.conv2 = conv3x3(planes, planes)
        #self.bn2 = nn.BatchNorm2d(planes)
        self.downsample = downsample
        self.stride = stride
        self.drop = nn.Dropout2d(p=drop)

    def forward(self, x):
        if self.inplanes != self.planes:
            x = self.conv0(x)
            x = F.relu(x)

        out = self.conv1(x)
        #out = self.bn1(out)
        out = self.relu(out)

        out = self.drop(out)

        out1 = self.conv1(out)
        #out1 = self.relu(out1)

        out2 = out1 + x

        return F.relu(out2)

class Bottleneck(nn.Module):
    expansion = 4

    def __init__(self, inplanes, planes, stride=1, downsample=None):
        super(Bottleneck, self).__init__()
        self.conv1 = nn.Conv2d(inplanes, planes, kernel_size=1, bias=False)
        self.bn1 = nn.BatchNorm2d(planes)
        self.conv2 = nn.Conv2d(planes, planes, kernel_size=3, stride=stride,
                               padding=1, bias=False)
        self.bn2 = nn.BatchNorm2d(planes)
        self.conv3 = nn.Conv2d(planes, planes * self.expansion, kernel_size=1, bias=False)
        self.bn3 = nn.BatchNorm2d(planes * self.expansion)
        self.relu = nn.ReLU(inplace=True)
        self.downsample = downsample
        self.stride = stride

    def forward(self, x):
        residual = x

        out = self.conv1(x)
        out = self.bn1(out)
        out = self.relu(out)

        out = self.conv2(out)
        out = self.bn2(out)
        out = self.relu(out)

        out = self.conv3(out)
        out = self.bn3(out)

        if self.downsample is not None:
            residual = self.downsample(x)

        out += residual
        out = self.relu(out)

        return out

class Initial_LadderBlock(nn.Module):

    def __init__(self,planes,layers,kernel=3,block=BasicBlock,inplanes = 3):
        super().__init__()
        self.planes = planes
        self.layers = layers
        self.kernel = kernel

        self.padding = int((kernel-1)/2)
        self.inconv = nn.Conv2d(in_channels=inplanes,out_channels=planes,
                                kernel_size=3,stride=1,padding=1,bias=True)

        # create module list for down branch
        self.down_module_list = nn.ModuleList()
        for i in range(0,layers):
            self.down_module_list.append(block(planes*(2**i),planes*(2**i)))

        # use strided conv instead of poooling
        self.down_conv_list = nn.ModuleList()
        for i in range(0,layers):
            self.down_conv_list.append(nn.Conv2d(planes*2**i,planes*2**(i+1),stride=2,kernel_size=kernel,padding=self.padding))

        # create module for bottom block
        self.bottom = block(planes*(2**layers),planes*(2**layers))

        # create module list for up branch
        self.up_conv_list = nn.ModuleList()
        self.up_dense_list = nn.ModuleList()
        for i in range(0, layers):
            self.up_conv_list.append(nn.ConvTranspose2d(in_channels=planes*2**(layers-i), out_channels=planes*2**max(0,layers-i-1), kernel_size=3,
                                                        stride=2,padding=1,output_padding=1,bias=True))
            self.up_dense_list.append(block(planes*2**max(0,layers-i-1),planes*2**max(0,layers-i-1)))

    def forward(self, x):
        out = self.inconv(x)
        out = F.relu(out)

        down_out = []
        # down branch
        for i in range(0,self.layers):
            out = self.down_module_list[i](out)
            down_out.append(out)
            out = self.down_conv_list[i](out)
            out = F.relu(out)

        # bottom branch
        out = self.bottom(out)
        bottom = out

        # up branch
        up_out = []
        up_out.append(bottom)

        for j in range(0,self.layers):
            out = self.up_conv_list[j](out) + down_out[self.layers-j-1]
            #out = F.relu(out)
            out = self.up_dense_list[j](out)
            up_out.append(out)

        return up_out

class LadderBlock(nn.Module):

    def __init__(self,planes,layers,kernel=3,block=BasicBlock,inplanes = 3):
        super().__init__()
        self.planes = planes
        self.layers = layers
        self.kernel = kernel

        self.padding = int((kernel-1)/2)
        self.inconv = block(planes,planes)

        # create module list for down branch
        self.down_module_list = nn.ModuleList()
        for i in range(0,layers):
            self.down_module_list.append(block(planes*(2**i),planes*(2**i)))

        # use strided conv instead of poooling
        self.down_conv_list = nn.ModuleList()
        for i in range(0,layers):
            self.down_conv_list.append(nn.Conv2d(planes*2**i,planes*2**(i+1),stride=2,kernel_size=kernel,padding=self.padding))

        # create module for bottom block
        self.bottom = block(planes*(2**layers),planes*(2**layers))

        # create module list for up branch
        self.up_conv_list = nn.ModuleList()
        self.up_dense_list = nn.ModuleList()
        for i in range(0, layers):
            self.up_conv_list.append(nn.ConvTranspose2d(planes*2**(layers-i), planes*2**max(0,layers-i-1), kernel_size=3,
                                                        stride=2,padding=1,output_padding=1,bias=True))
            self.up_dense_list.append(block(planes*2**max(0,layers-i-1),planes*2**max(0,layers-i-1)))

    def forward(self, x):
        out = self.inconv(x[-1])

        down_out = []
        # down branch
        for i in range(0,self.layers):
            out = out + x[-i-1]
            out = self.down_module_list[i](out)
            down_out.append(out)

            out = self.down_conv_list[i](out)
            out = F.relu(out)

        # bottom branch
        out = self.bottom(out)
        bottom = out

        # up branch
        up_out = []
        up_out.append(bottom)

        for j in range(0,self.layers):
            out = self.up_conv_list[j](out) + down_out[self.layers-j-1]
            #out = F.relu(out)
            out = self.up_dense_list[j](out)
            up_out.append(out)

        return up_out

class Final_LadderBlock(nn.Module):

    def __init__(self,planes,layers,kernel=3,block=BasicBlock,inplanes = 3):
        super().__init__()
        self.block = LadderBlock(planes,layers,kernel=kernel,block=block)

    def forward(self, x):
        out = self.block(x)
        return out[-1]

class LadderNetv6(nn.Module):
    def __init__(self,layers=3,filters=16,num_classes=2,inplanes=3):
        super().__init__()
        self.initial_block = Initial_LadderBlock(planes=filters,layers=layers,inplanes=inplanes)
        #self.middle_block = LadderBlock(planes=filters,layers=layers)
        self.final_block = Final_LadderBlock(planes=filters,layers=layers)
        self.final = nn.Conv2d(in_channels=filters,out_channels=num_classes,kernel_size=1)

    def forward(self,x):
        out = self.initial_block(x)
        #out = self.middle_block(out)
        out = self.final_block(out)
        out = self.final(out)
        #out = F.relu(out)
        out = F.log_softmax(out,dim=1)
        return out

and

def unet(pretrained_weights = None,input_size = (256,256,1)):
    inputs = Input(input_size)
    conv1 = Conv2D(64, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(inputs)
    conv1 = Conv2D(64, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv1)
    pool1 = MaxPooling2D(pool_size=(2, 2))(conv1)
    conv2 = Conv2D(128, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(pool1)
    conv2 = Conv2D(128, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv2)
    pool2 = MaxPooling2D(pool_size=(2, 2))(conv2)
    conv3 = Conv2D(256, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(pool2)
    conv3 = Conv2D(256, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv3)
    pool3 = MaxPooling2D(pool_size=(2, 2))(conv3)
    conv4 = Conv2D(512, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(pool3)
    conv4 = Conv2D(512, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv4)
    drop4 = Dropout(0.5)(conv4)
    pool4 = MaxPooling2D(pool_size=(2, 2))(drop4)

    conv5 = Conv2D(1024, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(pool4)
    conv5 = Conv2D(1024, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv5)
    drop5 = Dropout(0.5)(conv5)

    up6 = Conv2D(512, 2, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(UpSampling2D(size = (2,2))(drop5))
    merge6 = concatenate([drop4,up6], axis = 3)
    conv6 = Conv2D(512, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(merge6)
    conv6 = Conv2D(512, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv6)

    up7 = Conv2D(256, 2, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(UpSampling2D(size = (2,2))(conv6))
    merge7 = concatenate([conv3,up7], axis = 3)
    conv7 = Conv2D(256, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(merge7)
    conv7 = Conv2D(256, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv7)

    up8 = Conv2D(128, 2, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(UpSampling2D(size = (2,2))(conv7))
    merge8 = concatenate([conv2,up8], axis = 3)
    conv8 = Conv2D(128, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(merge8)
    conv8 = Conv2D(128, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv8)

    up9 = Conv2D(64, 2, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(UpSampling2D(size = (2,2))(conv8))
    merge9 = concatenate([conv1,up9], axis = 3)
    conv9 = Conv2D(64, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(merge9)
    conv9 = Conv2D(64, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv9)
    conv9 = Conv2D(2, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv9)
    conv10 = Conv2D(1, 1, activation = 'sigmoid')(conv9)

    model = Model(input = inputs, output = conv10)

    model.compile(optimizer = Adam(lr = 1e-4), loss = 'binary_crossentropy', metrics = ['accuracy'])

    #model.summary()

    if(pretrained_weights):
        model.load_weights(pretrained_weights)

    return model

I'm very new to PyTorch, and I am still familiarizing myself with the transition between Keras and PyTorch, and I'm also hoping that the above can help in this transition of mine.

With regards to the implementation in Keras for LadderNet, if I understood the paper correctly, is it simply just 2 U-Nets superimposed side-by-side (named LaddderNetKeras) as follows:

def LadderNetKeras(pretrained_weights = None,input_size = (256,256,1)):
    inputs = Input(input_size)
    conv1 = Conv2D(64, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(inputs)
    conv1 = Conv2D(64, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv1)
    pool1 = MaxPooling2D(pool_size=(2, 2))(conv1)
    conv2 = Conv2D(128, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(pool1)
    conv2 = Conv2D(128, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv2)
    pool2 = MaxPooling2D(pool_size=(2, 2))(conv2)
    conv3 = Conv2D(256, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(pool2)
    conv3 = Conv2D(256, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv3)
    pool3 = MaxPooling2D(pool_size=(2, 2))(conv3)
    conv4 = Conv2D(512, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(pool3)
    conv4 = Conv2D(512, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv4)
    drop4 = Dropout(0.5)(conv4)
    pool4 = MaxPooling2D(pool_size=(2, 2))(drop4)

    conv5 = Conv2D(1024, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(pool4)
    conv5 = Conv2D(1024, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv5)
    drop5 = Dropout(0.5)(conv5)

    up6 = Conv2D(512, 2, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(UpSampling2D(size = (2,2))(drop5))
    merge6 = concatenate([drop4,up6], axis = 3)
    conv6 = Conv2D(512, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(merge6)
    conv6 = Conv2D(512, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv6)

    up7 = Conv2D(256, 2, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(UpSampling2D(size = (2,2))(conv6))
    merge7 = concatenate([conv3,up7], axis = 3)
    conv7 = Conv2D(256, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(merge7)
    conv7 = Conv2D(256, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv7)

    up8 = Conv2D(128, 2, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(UpSampling2D(size = (2,2))(conv7))
    merge8 = concatenate([conv2,up8], axis = 3)
    conv8 = Conv2D(128, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(merge8)
    conv8 = Conv2D(128, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv8)

    up9 = Conv2D(64, 2, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(UpSampling2D(size = (2,2))(conv8))
    merge9 = concatenate([conv1,up9], axis = 3)
    conv9 = Conv2D(64, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(merge9)
    conv9 = Conv2D(64, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv9)
    conv9 = Conv2D(2, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv9)
    conv10 = Conv2D(1, 1, activation = 'sigmoid')(conv9)

    # SECOND U-NET
    conv1 = Conv2D(64, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv10)
    conv1 = Conv2D(64, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv1)
    pool1 = MaxPooling2D(pool_size=(2, 2))(conv1)
    conv2 = Conv2D(128, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(pool1)
    conv2 = Conv2D(128, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv2)
    pool2 = MaxPooling2D(pool_size=(2, 2))(conv2)
    conv3 = Conv2D(256, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(pool2)
    conv3 = Conv2D(256, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv3)
    pool3 = MaxPooling2D(pool_size=(2, 2))(conv3)
    conv4 = Conv2D(512, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(pool3)
    conv4 = Conv2D(512, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv4)
    drop4 = Dropout(0.5)(conv4)
    pool4 = MaxPooling2D(pool_size=(2, 2))(drop4)

    conv5 = Conv2D(1024, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(pool4)
    conv5 = Conv2D(1024, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv5)
    drop5 = Dropout(0.5)(conv5)

    up6 = Conv2D(512, 2, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(UpSampling2D(size = (2,2))(drop5))
    merge6 = concatenate([drop4,up6], axis = 3)
    conv6 = Conv2D(512, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(merge6)
    conv6 = Conv2D(512, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv6)

    up7 = Conv2D(256, 2, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(UpSampling2D(size = (2,2))(conv6))
    merge7 = concatenate([conv3,up7], axis = 3)
    conv7 = Conv2D(256, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(merge7)
    conv7 = Conv2D(256, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv7)

    up8 = Conv2D(128, 2, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(UpSampling2D(size = (2,2))(conv7))
    merge8 = concatenate([conv2,up8], axis = 3)
    conv8 = Conv2D(128, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(merge8)
    conv8 = Conv2D(128, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv8)

    up9 = Conv2D(64, 2, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(UpSampling2D(size = (2,2))(conv8))
    merge9 = concatenate([conv1,up9], axis = 3)
    conv9 = Conv2D(64, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(merge9)
    conv9 = Conv2D(64, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv9)
    conv9 = Conv2D(2, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv9)
    conv10 = Conv2D(1, 1, activation = 'sigmoid')(conv9)

    model = Model(input = inputs, output = conv10)

    model.compile(optimizer = Adam(lr = 1e-4), loss = 'binary_crossentropy', metrics = ['accuracy'])

    #model.summary()

    if(pretrained_weights):
        model.load_weights(pretrained_weights)

    return model

Thank you and some insights will be deeply appreciated!

Output: There is an implementation of laddernet in Keras available here : https://github.com/divamgupta/ladder_network_keras/blob/master/ladder_net.py. Consider this as a starting point, I have used at a point this repository successfully.


Instruction: Why use relu before maxpooling?

Input:
In this pytorch neural network tutorial tutorial link

I'm confuse why we need to use relu before max pooling.
Isn't the pixel values in the image are already positive?
I don't know why relu max(0, x) is needed.
Can anybody give me some advise on this issue?

 class Net(nn.Module):
  ...(init function)

   def forward(self, x):
     x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2)) # Max pooling over a (2, 2) window

Output: The weights of the neural net can be negative thus you can have a negative activation and by using the relu function, you're only activating the nodes that serve the purpose.


Instruction: how to add tensor size pytorch

Input:
I have a problem while working on PyTorch.

I'm trying to increase tensor size from [1,97,1] to [1,97,2]. How can I add "1" in tensor size?

Output:
Use pad

import torch.nn.functional as F
a = torch.empty((1,97,1))

a = F.pad(input=a, pad=(0,1))     # pad = (padding_left,padding_right)

print(a.shape)
>>> torch.Size([1, 97, 2])

it will pad 0 constant by default at right sight.


Instruction: Pytorch - Apply pooling on specific dimension

Input:
I have a 3 dimension vector. I would like to perform a 1d max pool on the second dimension.

According to the documentation of pytorch the pooling is always performed on the last dimension.

  https://pytorch.org/docs/stable/nn.html#maxpool1d

For example:

>>> x = torch.rand(5, 64, 32)
>>> pool = nn.MaxPool1d(2, 2)
>>> pool(x).shape
torch.Size([5, 64, 16])

My desired output:

torch.Size([5, 32, 32])

How can i do that?

Output:
You can simply permute the dimensions:

x = torch.rand(5, 128, 32)
pool = nn.MaxPool1d(2, 2)
pool(x.permute(0,2,1)).permute(0,2,1)  # shape (5, 128, 32) -> (5, 64, 32)


Instruction: How do I make a mask of diagonal matrix, but starting from the 2nd column?

Input:
So here is what I can get with torch.eye(3,4) now

The matrix I get:

[[1, 0, 0, 0],
 [0, 1, 0, 0],
 [0, 0, 1, 0]]

Is there any (easy)way to transform it, or make such a mask in this format:

The matrix I want:

[[0, 1, 0, 0],
 [0, 0, 1, 0],
 [0, 0, 0, 1]]

Output:
You can do it by using torch.diagonal and specifying the diagonal you want:

>>> torch.diag(torch.tensor([1,1,1]), diagonal=1)[:-1]

tensor([[0, 1, 0, 0],
        [0, 0, 1, 0],
        [0, 0, 0, 1]])

  If :attr:diagonal = 0, it is the main diagonal.
  If :attr:diagonal > 0, it is above the main diagonal.
  If :attr:diagonal < 0, it is below the main diagonal.


Instruction: Why does renewing an optimizer give a bad result?

Input:
I tried to change my optimizer, but first of all, I want to check whether the following two codes give the same results:

optimizer = optim.Adam(params, lr)
for epoch in range(500):
    ....
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
for epoch in range(500):
    ....
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

If I insert the same optimizer between 'for loops',

optimizer = optim.Adam(params, lr)
for epoch in range(500):
    ....
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

optimizer = optim.Adam(params, lr)
for epoch in range(500):
    ....
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

The result become bad. Why does this happens? Doesn't optimizer just receive gradients from loss and operate gradiet descent like steps?

Output:
Different optimizers may have some "memory".
For instance, Adam updates rule tracks the first and second moments of the gradients of each parameter and uses them to calculate the step size for each parameter.
Therefore, if you initialize your optimizer you erase this information and consequently make the optimizer "less informed" resulting with sub optimal choises for step sizes.


Instruction: PyTorch model to C++

Input:
I have trained the detection algorithm and saved my best model. Now I want to convert my model (pretrained) to C++ and use it in my app. I wanted to know what are the possible ways to convert a pyTorch model to c++?

Thanks!

Output:
You can use TorchScript intermediate representation of a PyTorch model, through tracing and scripting, that can be run in C++ environment. For this, you'll probably have to modify the model itself in order for it to be traced or scripted.
You can use ONNX (Open Neural Network Exchange), through which you can export your model and load it in another C++ framework such as Caffe. It comes with its own implications though.
The easiest is to try Embedding Python, through which you can run your python (pytorch) model in C++ environment. Note that the model will still run in python, but only through C++, so there won't be any speed gains that you might be expecting in C++.

Also, with the release of torchvision 0.5, all models in torchvision have native support for TorchScript and ONNX.


Instruction: Does PyTorch loss() and backpropagation understand lambda layers?

Input:
I've been working with a resnet56 model from the code provided here: https://github.com/akamaster/pytorch_resnet_cifar10/blob/master/resnet.py.

I noticed that the implementation is different from many of the other available ResNet examples online, and I was wondering if PyTorch's backpropagation algorithm using loss() can account for the lambda layer and shortcut in the code provided.

If that is the case, can anyone provide insight into how PyTorch is able to interpret the lambda layer for backpropagation (i.e. how does PyTorch know how to differentiate with respect to the layer's operations)?

P.S. I also had to modify the code to fit my own use-case, and it seems like my own implementation with option == 'A' does not produce great results. This may simply be because option == 'B,' which uses convolutional layers instead of padding, is better for my data.

        self.shortcut = nn.Sequential()
        if stride != 1 or in_planes != planes:
            if option == 'A':
                top = (int) ((self.expansion*planes - in_planes) / 2)
                bot = (self.expansion*planes - in_planes) - top
                self.shortcut = LambdaLayer(lambda x:
                                            F.pad(x[:, :, ::stride, ::stride], (0, 0, 0, 0, top, bot), "constant", 0))

Output:
"I was wondering if PyTorch's backpropagation algorithm using loss() can account for the lambda layer and shortcut in the code provided."

PyTorch has no problem with backpropagating through lambda functions. Your LambdaLayer is just defining the forward pass of the Module as the evaluation of the lambda function, so your question boils down to whether PyTorch can backpropagate through lambda functions.

"If that is the case, can anyone provide insight into how PyTorch is able to interpret the lambda layer for backpropagation (i.e. how does PyTorch know how to differentiate with respect to the layer's operations)?"

The lambda function performs the torch.nn.functional.Pad function on x, which we can packpropagate through because it is has a defined backwards() function.

PyTorch handles lambda functions the same way an autodiff tool like PyTorch handles any function: it breaks it up into primitive operations, and uses the differentiation rules for each primitive operation to build up the derivative of the entire computation.


Instruction: What does Pytorch's default padding in Conv2D signify?

Input:
In the Pytorch documentation for the MaxPool2D states:

  padding (int or tuple, optional) – Zero-padding added to both sides of the input. Default: 0

Does it mean that the default values for padding would be 0 or that there will be no padding?

In my code: nn.Conv2d(input_channels, output_channels, kernel_size, stride), I didn't pass any padding parameter and when I printed it, it shows the following

  (0): Conv2d(4, 8, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2))

So, I'm assuming it by default adds padding to match the input size with values 0.

Now, if that's the case, then how can I turn it off and/or add a different padding value?

Output:
  Does it mean that the default values for padding would be 0 or that
  there will be no padding?

It means that there will be no padding at all (because the parameter padding specifies the size of the padding for each dimension and by default it is padding=0, i.e. (0, 0)).

>>> conv = torch.nn.Conv2d(4, 8, kernel_size=(3, 3), stride=(1, 1))
>>> conv.padding
(0, 0)

The convolution layer is agnostic of the input height and width, it only expects the input to have 4 channels (in this particular example). The input height and weight are computed after you pass the input to it, and if the image size is smaller than the kernel size, then it will throw an error -- it will not pad the input automatically to match the size. It would help if you posted full code to see where the padding comes from.


Instruction: Constant loss during LSTM training - PyTorch

Input:
I'm trying to implement an LSTM network for predicting the next word in a sentence. This is my first time building a neural network and I'm confused by all the information I found on the Internet.

I'm trying to use the following architecture:

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

class WordLSTM(nn.Module):
    def __init__(self, vocabulary_size, embedding_dim, hidden_dim):
        super().__init__()

        # Word embeddings
        self.encoder = nn.Embedding(vocabulary_size, embedding_dim)

        # LSTM input dim is embedding_dim, output dim is hidden_dim
        self.lstm = nn.LSTM(embedding_dim, hidden_dim)

        # Linear layer to map hidden states to vocabulary space
        self.decoder = nn.Linear(hidden_dim, vocabulary_size)

    def forward(self, sentence):
        encoded = self.encoder(sentence)
        output, _ = self.lstm(
            encoded.view(len(sentence), 1, -1))
        decoded = self.decoder(output)
        word_scores = F.softmax(decoded, dim=1)

        return word_scores[-1].view(1, -1)

I've created a dictionary with all the sentences from my dataset, and each word is encoded with their respective indices from the dictionary. They're followed by an encoded next word (target vector). Here's a bunch of training examples that I'm trying to use:

[tensor([39]), tensor([13698])],
[tensor([   39, 13698]), tensor([11907])],
[tensor([   39, 13698, 11907]), tensor([70])]

I'm passing one sentence at a time during training, so my batch size is always 1.

NUM_EPOCHS = 100
LEARNING_RATE = 0.0005

rnn = WordLSTM(vocab_size, 64, 32)

optimizer = optim.SGD(rnn.parameters(), lr=LEARNING_RATE)

for epoch in range(NUM_EPOCHS):
    training_example = generate_random_training_example(training_ds)
    optimizer.zero_grad()

    for sentence, next_word in training_example:
        output = rnn(sentence)
        loss = F.cross_entropy(output, next_word)
        loss.backward()

    optimizer.step()

    print(f"Epoch: {epoch}/{NUM_EPOCHS} Loss: {loss:.4f}")

However, when I start the training, the loss does not change with time:

Epoch: 0/100 Loss: 10.3929
Epoch: 1/100 Loss: 10.3929
Epoch: 2/100 Loss: 10.3929
Epoch: 3/100 Loss: 10.3929
Epoch: 4/100 Loss: 10.3929
Epoch: 5/100 Loss: 10.3929
Epoch: 6/100 Loss: 10.3929

I've tried placing optimizer.zero_grad() and optimizer.step() in a different places already, but didn't help either.

What could be the problem in this case? Am I calculating the loss in the wrong way, or do I pass the tensors in the wrong format?

Output:
Delete F.softmax. You do log_softmax(softmax(x)).

CrossEntropyLoss

  This criterion combines nn.LogSoftmax() and nn.NLLLoss() in one single class.

import torch as t

class Net(t.nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.emb = t.nn.Embedding(100, 8)
        self.lstm = t.nn.LSTM(8, 16, batch_first=True)
        self.linear = t.nn.Linear(16, 100)

    def forward(self, x):
        x = self.emb(x)
        x, _ = self.lstm(x)
        x = self.linear(x[:, -1])
        #x = t.nn.Softmax(dim=1)(x)
        return x

t.manual_seed(0)
net = Net()

batch_size = 1
X = t.LongTensor(batch_size, 5).random_(0, 100)
Y = t.LongTensor(batch_size).random_(0, 100)

optimizer = t.optim.Adam(net.parameters())
criterion = t.nn.CrossEntropyLoss()

for epoch in range(10):
    optimizer.zero_grad()
    output = net(X)
    loss = criterion(output, Y)
    loss.backward()
    optimizer.step()

    print(loss.item())

4.401515960693359
4.389760494232178
4.377873420715332
4.365848541259766
4.353675365447998
4.341339588165283
4.328824520111084
4.316114902496338
4.303196430206299
4.2900567054748535

With uncommented t.nn.Softmax:

4.602912902832031
4.6027679443359375
4.602619171142578
4.6024675369262695
4.602311611175537
4.602152347564697
4.601987361907959
4.601818084716797
4.6016435623168945
4.601463794708252

Use softmax during evaluation:

net.eval()
t.nn.Softmax(dim=1)(net(X[0].view(1,-1)))

tensor([[0.0088, 0.0121, 0.0098, 0.0072, 0.0085, 0.0083, 0.0083, 0.0108, 0.0127,
         0.0090, 0.0094, 0.0082, 0.0099, 0.0115, 0.0094, 0.0107, 0.0081, 0.0096,
         0.0087, 0.0131, 0.0129, 0.0127, 0.0118, 0.0107, 0.0087, 0.0073, 0.0114,
         0.0076, 0.0103, 0.0112, 0.0104, 0.0077, 0.0116, 0.0091, 0.0091, 0.0104,
         0.0106, 0.0094, 0.0116, 0.0091, 0.0117, 0.0118, 0.0106, 0.0113, 0.0083,
         0.0091, 0.0076, 0.0089, 0.0076, 0.0120, 0.0107, 0.0139, 0.0097, 0.0124,
         0.0096, 0.0097, 0.0104, 0.0128, 0.0084, 0.0119, 0.0096, 0.0100, 0.0073,
         0.0099, 0.0086, 0.0090, 0.0089, 0.0098, 0.0102, 0.0086, 0.0115, 0.0110,
         0.0078, 0.0097, 0.0115, 0.0102, 0.0103, 0.0107, 0.0095, 0.0083, 0.0090,
         0.0120, 0.0085, 0.0113, 0.0128, 0.0074, 0.0096, 0.0123, 0.0106, 0.0105,
         0.0101, 0.0112, 0.0086, 0.0105, 0.0121, 0.0103, 0.0075, 0.0098, 0.0082,
         0.0093]], grad_fn=)


Instruction: Visualizing models, data, and training with tensorboard in pytorch

Input:
Following this Tutorial
to visualize images in tensorboard using torch.utils.tensorboard got error

from torch.utils.tensorboard import SummaryWriter
writer = SummaryWriter('runs/fashion_mnist_experiment_1')
writer.add_embedding(features,
                metadata=class_labels,
                label_img=images.unsqueeze(1))
writer.close()

Error:

        AttributeError: module 'tensorflow_core._api.v2.io.gfile' has no attribute 'get_filesystem'

Output:
try adding

writer = SummaryWriter(path)
img_grid = torchvision.utils.make_grid(torch.FloatTensor(imgs)) #imgs represent a batch of images with shape BxCxHxW
writer_add_image('images', img_grid)

source : https://pytorch.org/tutorials/intermediate/tensorboard_tutorial.html#writing-to-tensorboard


Instruction: Confused about calculation of convolutional layer shapes

Input:
I am new in this forum and I have started studying the theory of CNN.
It is probably a stupid question but I am confused about the calculation of the CNN outputs shape.
I am following a course on Udacity and in one of the tutorials they provide this CNN architecture.

import torch.nn as nn
import torch.nn.functional as F

# define the CNN architecture
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        # convolutional layer (sees 32x32x3 image tensor)
        self.conv1 = nn.Conv2d(3, 16, 3, padding=1)
        # convolutional layer (sees 16x16x16 tensor)
        self.conv2 = nn.Conv2d(16, 32, 3, padding=1)
        # convolutional layer (sees 8x8x32 tensor)
        self.conv3 = nn.Conv2d(32, 64, 3, padding=1)
        # max pooling layer
        self.pool = nn.MaxPool2d(2, 2)
        # linear layer (64 * 4 * 4 -> 500)
        self.fc1 = nn.Linear(64 * 4 * 4, 500)
        # linear layer (500 -> 10)
        self.fc2 = nn.Linear(500, 10)
        # dropout layer (p=0.25)
        self.dropout = nn.Dropout(0.25)

Could you please help in understanding the way they calculate the outputs of the CNN layers? (The starting shape of the images is 32x32x3)
More specifically how did they end up with this:

# linear layer (64 * 4 * 4 -> 500)
self.fc1 = nn.Linear(64 * 4 * 4, 500)

Thanks a lot

Output:
It misses the definition of the forward pass and one can guess there is a 2x2 pooling after each conv layer. Hence, the pooling implies a subsampling each time (see the comments) and the 32x32 images becomes 16x16 after conv1 (+ 2x2 pooling), 8x8 after conv2 (+ 2x2 pooling) and 4x4 after conv3 (+ 2x2 pooling). Since conv3 has 64 filters, it outputs 64 feature maps of size 4x4. Then, the fc1 maps this tensor to a fully connected layer of size 500. It is exactly what is defined by the line

self.fc1 = nn.Linear(64 * 4 * 4, 500)


Instruction: neighbours of a cell in matrix pytorch

Input:
I am trying to get neighbours of a cell of matrix in pytorch using below part of code.
it works correctly but it is very time consumming.
Have you any suggestion to to get it faster

def neighbour(x):
    result=F.pad(input=x, pad=(1, 1, 1, 1), mode='constant', value=0)
    for m in range(1,x.size(0)+1):
        for n in range(1,x.size(1)+1):
                y=torch.Tensor([result[m][n],result[m-1][n-1],result[m-1][n],result[m-1]
           [n+1],result[m][n-1],result[m][n+1],result[m+1][n-1],result[m+1][n],result[m+1][n+1]])
                x[m-1][n-1]=y.mean()

    return x

Output:
If you are only after the mean of the 9 elements centered at each pixel, then your best option would be to use a 2D convolution with a constant 3x3 filter:

import torch.nn.functional as nnf

def mean_filter(x_bchw):
  """
  Calculating the mean of each 3x3 neighborhood.
  input:
    - x_bchw: input tensor of dimensions batch-channel-height-width
  output:
    - y_bchw: each element in y is the average of the 9 corresponding elements in x_bchw
  """
  # define the filter
  box = torch.ones((3, 3), dtype=x_bchw.dtype, device=x_bchw.device, requires_grad=False)
  box = box / box.sum()
  box = box[None, None, ...].repeat(x_bchw.size(1), 1, 1, 1)
  # use grouped convolution - so each channel is averaged separately.
  y_bchw = nnf.conv2d(x_bchw, box, padding=1, groups=x_bchw.size(1))
  return y_bchw

however, if you want to apply a more elaborate function over each neighborhood, you may want to use nn.Unfold. This operation converts each 3x3 (or whatever rectangular neighborhood you define) to a vector. Once you have all the vectors you may apply your function to them.
See this answer for more details on unfold and fold.


Instruction: Taking the maximum values of each row in a tensor [PyTorch]

Input:
Suppose I have a tensor of the form

[[-5, 0, -1],
 [3, 100, 87],
 [17, -34, 2],
 [45, 1, 25]]

I want to find the maximum value in each row and return a rank 1 tensor as follows:

[0,
 100,
 17,
 45]

How would I do this in PyTorch?

Output:
You can use the torch.max() function. So you can do something like

x = torch.Tensor([[-5, 0, -1],
                  [3, 100, 87],
                  [17, -34, 2],
                  [45, 1, 25]])
out, inds = torch.max(x,dim=1)

and this will return the maximum values across each row (dimension 1). It will return max values with their indices.


Instruction: How to store and load training data comprised 50 millions 25x25 numpy arrays while training a multi-class CNN model?

Input:
I have an image processing problem where there are five classes, each class has approximately 10 millions examples as training data where an image is a z-scored 25x25 numpy array.

Obviously, I can’t load all the training data into memory, so I have to use fit_generator.

I also the one who generates and augments these training data matrices, but I can’t do it in real time within fit_generator because it will be too slow to train the model.

First, how to store 50 millions 25x25 .npy arrays on disk? What would be the best practice?

Second, Should I use a database to store these matrices and to query from it during training? I don’t think SQLite supports multi threads, and SQL datasets support is still experimental in tensorflow.

I would love to know if there is a neat way to store these 50 million matrices, so the retrieval during training will be optimal.

Third, what about using HDF5 format? Should I switch to pytorch instead?

Output:
Here is some code I found on medium (can't find the original post).
This will help to generate training data on-the-fly in a producer-consumer fashion:
import tensorflow as tf
import numpy as np

from time import sleep

class DataGen():
    counter = 0

    def __init__(self):
        self.gen_num = DataGen.counter
        DataGen.counter += 1

    def py_gen(self, gen_name):
        gen_name = gen_name.decode('utf8') + '_' + str(self.gen_num)
        for num in range(10):
            sleep(0.3)
            yield '{} yields {}'.format(gen_name, num)

Dataset = tf.data.Dataset
dummy_ds = Dataset.from_tensor_slices(['Gen1', 'Gen2', 'Gen3'])
dummy_ds = dummy_ds.interleave(lambda x: Dataset.from_generator(DataGen().py_gen, output_types=(tf.string), args=(x,)),
                               cycle_length=5,
                               block_length=2,
                               num_parallel_calls=5)
data_tf = dummy_ds.as_numpy_iterator()
for d in data_tf:
  print(d)

Output:
b'Gen1_0 yields 0'
b'Gen1_0 yields 1'
b'Gen2_0 yields 0'
b'Gen2_0 yields 1'
b'Gen3_0 yields 0'
b'Gen3_0 yields 1'
b'Gen1_0 yields 2'
b'Gen1_0 yields 3'
b'Gen2_0 yields 2'
b'Gen2_0 yields 3'
b'Gen3_0 yields 2'
b'Gen3_0 yields 3'
b'Gen1_0 yields 4'
b'Gen1_0 yields 5'
b'Gen2_0 yields 4'
b'Gen2_0 yields 5'
b'Gen3_0 yields 4'
b'Gen3_0 yields 5'
b'Gen1_0 yields 6'
b'Gen1_0 yields 7'
b'Gen2_0 yields 6'
b'Gen2_0 yields 7'
b'Gen3_0 yields 6'
b'Gen3_0 yields 7'
b'Gen1_0 yields 8'
b'Gen1_0 yields 9'
b'Gen2_0 yields 8'
b'Gen2_0 yields 9'
b'Gen3_0 yields 8'
b'Gen3_0 yields 9'


Instruction: How can I load CSV data for a PyTorch neural network?

Input:
This may be a simple question, and I apologize if it's too simple. But I have some data in a CSV:

Date,Open,High,Low,Close,Adj Close,Volume
1993-01-29,43.968750,43.968750,43.750000,43.937500,26.453930,1003200
1993-02-01,43.968750,44.250000,43.968750,44.250000,26.642057,480500
1993-02-02,44.218750,44.375000,44.125000,44.343750,26.698507,201300
1993-02-03,44.406250,44.843750,44.375000,44.812500,26.980742,529400
1993-02-04,44.968750,45.093750,44.468750,45.000000,27.093624,531500
1993-02-05,44.968750,45.062500,44.718750,44.968750,27.074818,492100
1993-02-08,44.968750,45.125000,44.906250,44.968750,27.074818,596100
1993-02-09,44.812500,44.812500,44.562500,44.656250,26.886669,122100
....

I want to create a "training set", which is basically a random vector of 10 rows of data (I can figure out the normalizing, etc) randomly sampled from anywhere in the file. I think I'll have to use pandas to do the loading maybe?

If what I'm trying to ask is unclear, please add comments and I will adjust the question accordingly. Thank you.

Output:
import pandas as pd

sample = pd.read_csv('myfile.csv').sample(n=10)

you should load the file only 1 time and then sample as you go:

df = pd.read_csv('myfile.csv')
sample1 = df.sample(n=10)
sample2 = df.sample(n=10)


Instruction: Pytorch Sigmoid Function what is e

Input:
I have a question on setting up the sigmoid function in pytroch.

So I define it as this

# Sigmoid function
def sigmoid(x):
    return 1/(1 + torch.exp(-x))

But then looking at the sigmoid function from here http://mathworld.wolfram.com/SigmoidFunction.html

The sigmoid should be defined as

y = 1/(1 + e^-x)

I see the 1/(1+  part but I don't get the e^-x part. Can someone explain why

torch.exp(-x) == e^-x

What is e here? Is that the tensor. But i thought that x was the tensor

Output:
Here e is the exponential constant e and
torch.exp(-x) == e^-x

torch.exp(x) returns e^x


Instruction: How to process input data for audio classification using CNN with PyTorch?

Input:
As an engineer student works towards DSP and ML fields, I am working on an audio classification project with inputs being short clips (4 sec.) of instruments like bass, keyboard, guitar, etc. (NSynth Dataset by the Magenta team at Google).

The idea is to convert all the short clips (.wav files) to spectrograms or melspectrograms then apply a CNN to train the model.

However, my questions is since the entire dataset is large (approximately 23GB), I wonder if I should firstly convert all the audio files to images like PNG then apply CNN. I feel like this can take a lot of time, and it will double the storage space for my input data as now it is audio + image (maybe up to 70GB).

Thus, I wonder if there is any workaround here that can speed the process.

Thanks in advance.

Output:
Preprocessing is totally worth it. You will very likely end up, running multiple experiments before your network will work as you want it to and you don't want to waste time pre-processing the features every time, you want to change a few hyper-parameters.

Rather than using PNG, I would rather save directly PyTorch tensors (torch.save that uses Python's standard pickling protocols) or NumPy arrays (numpy.savez saves serialized arrays into a zip file). If you are concerned with disk space, you can consider numpy.save_compressed.