Instruction: How to do 'same' padding in PyTorch if (n - 1) / 2 is no integer value

Input: I am trying to reconstruct a neural network written in tensorflow. For the convolutional layer, they just use padding='SAME'. This doesn't exist in pytorch. I know, that I can calculate the padding with p = (n - 1) / 2 for stride=1. But what if this doesn't result in an integer value? In my case, n is 4 and I always want to achieve same padding.

Output:
Use math.floor function to round down to the nearest integer or the math.ceil function to round up to the nearest integer:
import math

# for flooring
p = math.floor((n - 1) / 2))

# for ceiling
p = math.ceil((n - 1) / 2))

For example, by default, pytorch uses flooring for MaxPool layers. So, I think flooring is a good starting point.


Instruction: Pytorch - Trick AutoGrad into thinking another output is the final outcome

Input:
The scenario:

I have a simple torch CNN network that predicts if a given image input is a dog or a cat.

After getting the output of the neural network, I need to apply a modifier of X to each prediction. For example, if the neural network return [0.6, 0.4], and I want to apply a modifier of [0.05, -0.03], I need the result to be [0.65, 0.37].

The desired result:

I would like AutoGrad to think the final output is [0.65, 0.37]. That is, AutoGrad shouldn't consider the modifier addition at all. In fact, It needs to be tricked into thinking the last operation result is [0.65, 0.37] instead of [0.6, 0.4], and apply the backpropagation with that in mind.

Here's what I need to know:

How am I suppose to do this? I know Torch records every single operation and compute a dynamic graph accordingly. I don't want this operation to be recorded at all, and I can't use the torch.no_grad() wrapper, because I need to be able to do the backpropagation after.

Edit 1
@trsvchn Here's what happen when I do your method. Is this expected? Will autograd use the "T" value instead of the "data" value when doing the backpropagation?

Output:
Use torch.no_grad context manager to deactivate autograd, for example this operation is not going to be recorded, but you should use in-place operation, otherwise you will add another tensor without grad_fn and break the graph. In your case:
out = model(inputs)  # [0.6, 0.4]

with torch.no_grad():
    out.add_(torch.tensor([0.05, -0.03]))  # inplace add_ op


Instruction: Huggingface transformers unusual memory use

Input:
I have the following code attempting to use XL transformers to vectorize text:
  text = "Some string about 5000 characters long"

  tokenizer = TransfoXLTokenizerFast.from_pretrained('transfo-xl-wt103', cache_dir=my_local_dir, local_files_only=True)
  model = TransfoXLModel.from_pretrained("transfo-xl-wt103", cache_dir=my_local_dir, local_files_only=True)

  encoded_input = tokenizer(text, return_tensors='pt')
  output = model(**encoded_input)

This produces:
    output = model(**encoded_input)
  File "/home/user/w/default/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/user/w/default/lib/python3.7/site-packages/transformers/modeling_transfo_xl.py", line 863, in forward
    output_attentions=output_attentions,
  File "/home/user/w/default/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/user/w/default/lib/python3.7/site-packages/transformers/modeling_transfo_xl.py", line 385, in forward
    dec_inp, r, attn_mask=dec_attn_mask, mems=mems, head_mask=head_mask, output_attentions=output_attentions,
  File "/home/user/w/default/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/user/w//default/lib/python3.7/site-packages/transformers/modeling_transfo_xl.py", line 338, in forward
    attn_score = attn_score.float().masked_fill(attn_mask[:, :, :, None], -1e30).type_as(attn_score)
RuntimeError: [enforce fail at CPUAllocator.cpp:64] . DefaultCPUAllocator: can't allocate memory: you tried to allocate 2007869696 bytes. Error code 12 (Cannot allocate memory)

I'm a little perplexed by this because this is asking for 2007869696, which is only 2GB and this machine has 64G of RAM. So I both don't understand why it is asking for this, and even more, why it is failing to get it.
Where can I change the setting that controls this and allow this process more RAM?  This is such a small invocation of the example code, and I just see very few places that would even accept this argument.

Output:
Are you sure you are using the gpu instead of cpu?
Try to run the python script with CUDA_LAUNCH_BLOCKING=1 python script.py. This will produce the correct python stack trace (as CUDA calls are asynchronous)
Also you can set the CUDA_VISIBLE_DEVICES using export CUDA_VISIBLE_DEVICES=device_number.
There is also an issue still open on the pytorch github, try to check it out.


Instruction: Runtime Error : both arguments to matmul need to be at least 1d but they are 0d and 2d

Input:
This is the code I have written, I have tried modifying here and there and have always gotten the same error. As I am a beginner to PyTorch, I am just trying things out to see if machine learning will work on a linear dataset. So, with random, I initialized a dataset. Then, made a single linear neural network. Then, train the neural network on the linear data. Nevertheless, I have an error stating that matmul expects a 1d data, but then my data is at least 1d but is read as 0d instead.
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import random

class LinearNetwork(nn.Module):
    def __init__(self):
        super(LinearNetwork, self).__init__()
        self.fc1 = nn.Linear(1, 1)

    def forward(self, input):
        x = self.fc1(input)
        return x;

#Generate a Dataset to train on. Just a linear function, and we wanna see if this thing is working
DataSet = []
grad = random.randint(0,100)
const = random.randint(0,100)
for i in range(0, 1000):
    DataSet.append([i, grad*i + const])

DataSet = torch.tensor(DataSet,dtype = torch.double, requires_grad = True)

#Declare a Linear Network
Net = LinearNetwork()
criterion = nn.MSELoss()
optimizer = optim.SGD(Net.parameters(), lr = 0.01)
print(DataSet)
#Train Model
for i, data in enumerate(DataSet, 0):
    Input, Target = data
    optimizer.zero_grad()
    Output = Net(Input)
    loss = criterion(Output, Target)
    loss.backward()
    optimizer.step()

print("Gradient of the Function is : " + str(grad))
print("Constant Value of the Function is : " + str(const))
print("Learned Gradient and Constant : " + list(Net.parameters()))

Output:
There are two errors in your code, first regarding shapes, second regarding dtypes. BTW. Please use snake_case for variables (e.g. my_dataset, net) and CamelCase for classes as it's a common Python convention.
Shape error
This one lies here:
for i, data in enumerate(DataSet, 0):
    input, target = data
    optimizer.zero_grad()
    output = net(input)
    loss = criterion(output, target)
    loss.backward()
    optimizer.step()

When you print input.shape you get torch.Size([]) which is a 0d tensor. Matrix multiplication needs 1d tensor so you should unsqueeze it so it has this dimension. Change above output = net(input) to:
output = net(input.unsqueeze(dim=0))

dtype error
Here:
DataSet = torch.tensor(DataSet,dtype = torch.double, requires_grad = True)

You create tensor of torch.double type while your neural network is (by default) of type torch.float. The latter is usually used as that's enough numerical precision and saves memory. So instead of the above you should do:
DataSet = torch.tensor(DataSet,dtype = torch.float, requires_grad = True)

Shape again
Coming back to shapes, neural networks should operate on at least two dimensions: (batch, features). In your case batch=1 and features=1, so it should be unsqueezed once more (and i advise you to try your code using batches).


Instruction: Pytorch listed by conda but cannot import

Input:
I am well aware similar questions have been asked
at least twice, but none of the answers seams to solve the issue at hand
My configuration
Windows 10.0.18363,
Anaconda 4.8.5,
Cuda 10.1.243
conda env create -n torch -y python 3.7
conda activate torch
conda install conda -y
conda install pytorch torchvision cudatoolkit=10.2 -c pytorch -y

Here's the bug
python -c "import torch"

Traceback (most recent call last):   File "<stdin>", line 1, in <module> ModuleNotFoundError: No module named 'torch'

What I tried
Verifying python and conda
where python

C:\ProgramData\Anaconda3\envs\torch\python.exe
C:\ProgramData\Anaconda3\python.exe
C:\csvn\Python25\python.exe

python -c "import site; print(site.getsitepackages())"

['C:\ProgramData\Anaconda3',
'C:\ProgramData\Anaconda3\lib\site-packages']

conda update -n base -y conda
conda update --all -y
conda init

...
No action taken.

Verifying the torch installation
conda list | findstr torch

_pytorch_select           0.1                       cpu_0
pytorch                   1.6.0           cpu_py37h538a6d7_0
torchvision               0.7.0                py37_cu102    pytorch

Output:
More a suggestion than a solution: you can at least reduce the problem surface by working with a YAML instead of using a series of create/activate/install commands. Create the file:
torch.yaml
name: torch
channels:
  - pytorch
  - defaults
dependencies:
  - python=3.7
  - pytorch
  - torchvision
  - cudatoolkit=10.2

Then simply use
conda env create -f torch.yaml

The result should be equivalent to the environment you indicated, covering both env creation and installation of all packages in a single command.1 Plus, you don't need all those pesky --yes|-y flags.
Any problems still persisting are most likely due to PATH or other environment variable management issues.

[1] I excluded conda from the YAML because that package should only be installed in base. Perhaps you meant anaconda package?


Instruction: How does Pytorch's `autograd` handle non-mathematical functions?

Input: During the course of my training process, I tend to use a lot of calls to torch.cat() and copying tensors into new tensors. How are these operations handled by autograd? Is the gradient value affected by these operations?

Output:
As pointed out in the comments, cat is a mathematical function. For example we could write the following (special case) definition of cat in more traditional mathematical notation as

The Jacobian of this function w.r.t. either of its inputs can be expressed as

Since the Jacobian is well defined you can, of course, apply back-propagation.
In reality you generally wouldn't define these operations with such notation, and a general definition of the cat operation used by pytorch in such a way would be cumbersome.
That said, internally autograd uses backward algorithms that take into account the gradients of such "index style" operations just like any other function.


Instruction: Kernel keeps dying when plotting a graph after importing the torch library

Input:
I'm trying to run the following code:
import matplotlib.pyplot as plt
%matplotlib inline
import torch

x = y = torch.tensor([1,2,3]).numpy()
plt.plot(x,y);

I keep getting the message: The kernel appears to have died. It will restart automatically. and a restart and a red "Dead kernel" tag on the toolbar.
But the weird thing is, if I import matplotlib.pyplot and plot some random graph first, the above code plots just fine. In other words, the following code works fine.
import matplotlib.pyplot as plt
%matplotlib inline

plt.subplots(figsize=(0.01,0.01))
plt.gca().set_visible(False);

import torch
x = torch.tensor([1,2,3]).numpy()

plt.plot(x,x);

What is going on here? I'm using numpy 1.18.5, pytorch 1.6.0, matplotlib 3.2.2 on Python 3.7.7, if it matters. Thank you.

Output:
import os
os.environ["KMP_DUPLICATE_LIB_OK"]="TRUE"

Run this first, then it'll resolve your problem. Although I guess, it is a temporary solution you can refer to this link: https://www.programmersought.com/article/53286415201/


Instruction: How to monitor GPU memory usage when training a DNN?

Input: I give a result example. I want to ask how to get the data like this graph.

Output: You can use pytorch commands such as torch.cuda.memory_stats to get information about current GPU memory usage and then create a temporal graph based on these reports.


Instruction: How is cross entropy loss work in pytorch?

Input:
I am experimenting with some of the pytorch codes. With cross entropy loss I found some interesting results and I have used both binary cross entropy loss and cross entropy loss of pytorch.
import torch
import torch.nn as nn

X = torch.tensor([[1,0],[1,0],[0,1],[0,1]],dtype=torch.float)
softmax = nn.Softmax(dim=1)

bce_loss = nn.BCELoss()
ce_loss= nn.CrossEntropyLoss()

pred = softmax(X)

bce_loss(X,X) # tensor(0.)
bce_loss(pred,X) # tensor(0.3133)
bce_loss(pred,pred) # tensor(0.5822)

ce_loss(X,torch.argmax(X,dim=1)) # tensor(0.3133)

I expected the cross entropy loss for the same input and output to be zero. Here X, pred and torch.argmax(X,dim=1) are same/similar with some transformations. This reasoning only worked for bce_loss(X,X) # tensor(0.) where-else all other resulted in a loss greater than zero. I speculated the output for bce_loss(pred,X), bce_loss(pred,pred) and ce_loss(X,torch.argmax(X,dim=1)) should be zero.
What is the mistake here?

Output:
The reason that you are seeing this is because nn.CrossEntropyLoss accepts logits and targets, a.k.a X should be logits, but is already between 0 and 1. X should be much bigger, because after softmax it will go between 0 and 1.
ce_loss(X * 1000, torch.argmax(X,dim=1)) # tensor(0.)

nn.CrossEntropyLoss works with logits, to make use of the log sum trick.
The way you are currently trying after it gets activated, your predictions become about [0.73, 0.26].
Binary cross entropy example works since it accepts already activated logits. By the way, you probably want to use nn.Sigmoid for activating binary cross entropy logits. For the 2-class example, softmax is also ok.


Instruction: What is a subspace of a dimension in pytorch?

Input:
The documentation of torch.Tensor.view says:

each new view dimension must either be a subspace of an original dimension, or only span across original dimensions ...

https://pytorch.org/docs/stable/tensors.html?highlight=view#torch.Tensor.view
What is a subspace of a dimension?

Output:
The 'subspace of an original dimension' dilemma
In order to use tensor.view() the tensor must satisfy two conditions-
each new view dimension must either be a

subspace of an original dimension
or only span across original dimensions ...

Lets discuss this one by one,
First, regarding subspace of an original dimension you need to understand the concept behind subspace. Not going into mathematical detail but in short - subspace is a subset of infinite number of n dimensional vectors(Rn) and the vectors inside the subspace must follow 2 rules -
i) Sub space will contain the Zero vector(0n)
ii) Must satisfy closure under Multiplication and addition
To visualise this in mind you can consider a 2D plane containing infinite lines. So the subspace of that 2D vector space will be set of lines which will pass through origin. These lines satisfies above two conditions.
Now there is a concept called projection of subspaces. Without digging into too much mathematical detail you can consider it as a regular line projection but for subspaces.
Now back to the point, lets assume if you have a tensor of size (4,5), you can actually consider it as a 20 dimensional vector. Assume you have a 20D space, and the subspaces will pass through the origin, and if you want to make projection of any line l1 from subspace with respect to any 2 axes, tensor.view() will output projection of that line with (2,10).
For the second part, you need to understand the concept of contiguous and non-contiguous memory allocation in pytorch. As its out of scope of the question I am going to explain it in very brief. If you view a n dimensional vector as (n/k, k) and if you run tensor.stride() on the new vector it will show you the stride for memory allocation in x and y direction. Now if you run view() again with different dimensions then this following equation must hold true for successful conversion due to non-contiguous memory allocation.

I tried my best to explain it in brief, let me know if you have more questions.


Instruction: pyTorch gradient becomes none when dividing by scalar

Input:
Consider the following code block:
import torch as torch

n=10
x = torch.ones(n, requires_grad=True)/n
y = torch.rand(n)
z = torch.sum(x*y)
z.backward()
print(x.grad) # results in None
print(y)

As written, x.grad is None. However, if I change the definition of x by removing the scalar multiplication (x = torch.ones(n, requires_grad=True)) then indeed I got a non-None gradient that is equivalent to y.
I've googled a bunch looking for this issue, and I think it reflects something fundamental in what I don't understand about how the computational graph in torch. I'd love some clarification. Thanks!

Output:
When you set x to a tensor divided by some scalar, x is no longer what is called a "leaf" Tensor in PyTorch. A leaf Tensor is a tensor at the beginning of the computation graph (which is a DAG graph with nodes representing objects such as tensors, and edges which represent a mathematical operation). More specifically, it is a tensor which was not created by some computational operation which is tracked by the autograd engine.
In your example - torch.ones(n, requires_grad=True) is a leaf tensor, but you can't access it directly in your code.
The reasoning behind not keeping the grad for non-leaf tensors is that typically, when you train a network, the weights and biases are leaf tensors and they are what we need the gradient for.
If you want to access the gradients of a non-leaf tensor, you should call the retain_grad function, which means in your code you should add:
x.retain_grad()

after the assignment to x.


Instruction: I want to use Conv1D and MaxPool1D in pytorch for a 3-d tensor to its third dimension

Input:
For example, there is a 3-d tensor, I want to run the conv1d calculation on its third dimension,
import torch
import torch.nn as nn
x = torch.rand(4,5,6)
conv1d =nn.Conv1d(in_channels=1,out_channels=2,kernel_size=5,stride=3,padding=0)
y = conv1d(x)

I hope the shape of y is (4,5,2,-1), but I get an error

Given groups=1, weight of size [2, 1, 5], expected input[4, 5, 6] to have 1 channels, but got 5 channels instead

Then I modified the code,
import torch
import torch.nn as nn
x = torch.rand(4,5,6)
conv1d =nn.Conv1d(in_channels=1,out_channels=2,kernel_size=5,stride=3,padding=0)
x = x.unsqueeze(2)
y = conv1d(x)

There is another error:

Expected 3-dimensional input for 3-dimensional weight [2, 1, 5], but got 4-dimensional input of size [4, 5, 1, 6] instead

And if I want to run the maxpoo1d calulation in a tensor whose shape is (4,5,2,-1) ,in its last two dimension, what should I do?
I am searching for a long time on net. But no use. Please help or try to give some ideas how to achieve this. Thank you all for your help.
I made an attempt, but I felt it couldn’t meet the actual needs, I wanted to know if it's good practice to do that and what would be the best way to do that?
import torch
import torch.nn as nn
x = torch.rand(4,5,6)
conv1d =nn.Conv1d(in_channels=1,out_channels=2,kernel_size=2,stride=3,padding=0)
x = x.unsqueeze(2)
for i in range(4):
    y = conv1d(x[i,:,:,:])
    y = y.unsqueeze(0)
    if i==0:
        z = y
    else:
        z = torch.cat((z,y),0)
    print(y)
print(z.size())

Output:
To use Conv1d you need your input to have 3 dimensions:
[batch_size, in_channels, data_dimension]

So, this would work:
x = torch.rand(4, 1, 50)  # [batch_size=4, in_channels=1, data_dimension=50]
conv1d = nn.Conv1d(in_channels=1,out_channels=2,kernel_size=2,stride=3,padding=0)
x = conv1d(x)
print(x.shape) # Will output [4, 2, 16] 4=batch_size, 2=channels, 16=data_dimension

You can use MaxPool1d in the same way:
maxpool1d = nn.MaxPool1d(5)
x = maxpool1d(x)
print(x.shape) # Will output [4, 2, 3]  4=batch_size, 2=channels, 3=data_dimension


Instruction: Why does the evaluation loss increases when training a huggingface transformers NER model?

Input:
Training a huggingface transformers NER model according to the documentation, the evaluation loss increases after a few epochs, but the other scores (accuracy, precision, recall, f1) continuously getting better. The behaviour seems unexpected, is there a simple explanation for this effect? Can this depend on the given data?

model = TokenClassificationModel.from_pretrained('roberta-base', num_labels=len(tag_values))

model.train()
model.zero_grad()

   for epoch in range(epochs):
      for batch in range(batches):
         -- train --
         ...

      train_loss = model.evaluate(train_data)
      validation_loss = model.evaluate(validation_data)

Output:
Accuracy and loss are not necessarily exactly (inversely) correlated.
The loss function is often an approximation of the accuracy function - unlike accuracy, the loss function must be differentiable.
A good explanation can be found here.


Instruction: element-wise operation in pytorch

Input:
I have two Tensors A and B, A.shape is (b,c,100,100), B.shape is (b,c,80,80),
how can I get tensor C with shape (b,c,21,21) subject to
C[:, :, i, j] = torch.mean(A[:, :, i:i+80, j:j+80] - B)?
I wonder whether there's an efficient way to solve this?
Thanks very much.

Output:
You should use an average pool to compute the sliding window mean operation.
It is easy to see that:
mean(A[..., i:i+80, j:j+80] - B) = mean(A[..., i:i+80, j:j+80]) - mean(B)

Using avg_pool2d:
import torch.nn.functional as nnf

C = nnf.avg_pool2d(A, kernel_size=80, stride=1, padding=0) - torch.mean(B, dim=(2,3), keepdim=True)

If you are looking for a more general way of performing sliding window operations in PyTorch, you should look at fold and unfold.


Instruction: The size of tensor a (707) must match the size of tensor b (512) at non-singleton dimension 1

Input:
I am trying to do text classification using pretrained BERT model. I trained the model on my dataset, and in the phase of testing; I know that BERT can only take to 512 tokens, so I wrote if condition to check the length of the test senetence in my dataframe. If it is longer than 512 I split the sentence into sequences each sequence has 512 token. And then do tokenizer encode. The length of the seqience is 512, however, after doing tokenize encode the length becomes 707 and I get this error.
The size of tensor a (707) must match the size of tensor b (512) at non-singleton dimension 1

Here is the code I used to do the preivous steps:
tokenizer = BertTokenizer.from_pretrained('bert-base-cased', do_lower_case=False)
import math

pred=[]
if (len(test_sentence_in_df.split())>512):

  n=math.ceil(len(test_sentence_in_df.split())/512)
  for i in range(n):
    if (i==(n-1)):
      print(i)
      test_sentence=' '.join(test_sentence_in_df.split()[i*512::])
    else:
      print("i in else",str(i))
      test_sentence=' '.join(test_sentence_in_df.split()[i*512:(i+1)*512])

      #print(len(test_sentence.split()))  ##here's the length is 512
    tokenized_sentence = tokenizer.encode(test_sentence)
    input_ids = torch.tensor([tokenized_sentence]).cuda()
    print(len(tokenized_sentence)) #### here's the length is 707
    with torch.no_grad():
      output = model(input_ids)
      label_indices = np.argmax(output[0].to('cpu').numpy(), axis=2)
    pred.append(label_indices)

print(pred)

Output:
This is because, BERT uses word-piece tokenization. So, when some of the words are not in the vocabulary, it splits the words to it's word pieces. For example: if the word playing is not in the vocabulary, it can split down to play, ##ing. This increases the amount of tokens in a given sentence after tokenization.
You can specify certain parameters to get fixed length tokenization:
tokenized_sentence = tokenizer.encode(test_sentence, padding=True, truncation=True,max_length=50, add_special_tokens = True)


Instruction: Correct way to feed data to RNN in PyTorch

Input:
I have a data sequence a which is of shape [seq_len, 2], seq_len is the length of the sequence. There is time correlation among elements of a[:, 0] and a[:, 1], but a[:, 0] and a[:, 1] are independent of each other. For training I prepare data of shape [batch_size, seq_len, 2]. The initialization of BRNN that I use is
birnn_layer = nn.RNN(input_size=2, hidden_size=100, batch_first=True, bidirectional=True)

From the docs,

input_size – The number of expected features in the input x

hidden_size – The number of features in the hidden state h

What does "number of expected features" mean? Since there is correlation along the seq_len axis should my input_size be set as seq_len and the input be permuted? Thanks.

Output:
The question is how, if at all, your data contributes to the overall optimization problem. You said that elements of a[:, 0] are time-correlated and elements of a[:, 1] are time-correlated. Are  a[i, 0] and a[i, 1] time-correlated? Does it makes sense for both sequences to be set together?
If, for example, you are trying to predict whether certain electrical machine is going to malfunction based on sequences of voltage applied to the machine a[:, 0] and humidity in the room a[:, 1] over time plus these signals were collected at the same time it is ok.  But should they were collected in different time, does it makes sense? Or if you would have measured something different than humidity, would it help you predict malfunction?
number of expected features means number of features in a single timestamp so to speak. So, going along with my previous analogy, how many signals (voltage, humidity.. ) I measure simultanously.
Of course this is only an example, you do not have to have classification-over-time problem, it can be anything else. The bottom point is how your RNN and data work together.


Instruction: How to stop importing line arguments with imported file

Input:
I have two programs train.py and predict.py and I am importing a trained model from train to predict.
Both programs accept line arguments and train runs fine, but when I run predict with its line arguments, an error occurs that I haven't typed in the arguments required by train.py.
How I can solve this?

Output:
Your question could use some more context. But here is what suspect might be happenning :
context
When you import a file (module), its content is executed. If your file only contain declarations (such as variable, class and function definitions) all is good, and you can use them from the place you wrote your import statement.
Now if the module you import contain actual code like function calls, it's going to run !
It's likely that your train.py file is expecting functions that fail (by missing some arguments apparently).
The usual solution to avoid this is to wrap all calls in a if __name__ == "__main__": clause. That way, it will only be executed if that file is called directly (as opposed to imported).
tl;dr:
Look into your train.py file for function calls
def a_function():
    pass

class SomeClass:
   pass

a_function() # <--- this is a call

my_var = SomeClass() # <--- this too !

and put them in that clause
def a_function():
    pass

class SomeClass:
   pass

if __name__ == "__main__":
   #only executed if you call this file (python train.py)
   a_function()
   my_var = SomeClass()


Instruction: Python3 Pytorch RuntimeError on GCP - no msg

Input:
My System
I am running a neural network training on using Python 3.6.9 with pytorch 1.6.0
I am using a google cloud platform N1 Server with a Tesla T4, 2 cores CPU, 12GB RAM.
This is on an Ubuntu 18.04 image.
Problem
When my code reaches the training line I get the following RuntimeError with no real explanation that I can see:
-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/home/or/.local/share/virtualenvs/or-M3_AaJfY/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
    fn(i, *args)
  File "/home/or/my_model/train.py", line 88, in train_and_eval
    train(rank, epoch, hps, generator, optimizer_g, train_loader, logger, writer)
  File "/home/or/my_model/train.py", line 117, in train
    scaled_loss.backward()
  File "/home/or/.local/share/virtualenvs/or-M3_AaJfY/lib/python3.6/site-packages/torch/tensor.py", line 185, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/or/.local/share/virtualenvs/or-M3_AaJfY/lib/python3.6/site-packages/torch/autograd/__init__.py", line 127, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError

This happens while the 2 CPU cores are being used at 100% for a long while.
The RAM and GPU, though going up (as expected while training) do not reach close to their
limit.
I checked journalctl to see if this was an operating system issue but there is nothing  there. I also did not find anything relevant in the /var/log/ directory or using dmesg.
I would be happy to provide more log data but I am not aware (after searching) any python logs I can look at, or any other system logs.

Please let me know of ways I can get more information if you have any ideas.
The exact same code works 100% fine on other physical machines I have tested, and a GPU only version of it runs fine on another cloud computing provider
What I am looking for

Ways to get more information about this problem and figure out why it is happening.
Ways to fix this problem

Thanks in advance for your time and any help you may be able to provide.

Output:
Anthony Leo thank you so much for your detailed answer!
Unfortunately this ended up being a problem with one of the modules I installed while setting up my server.
This did not end up being a problem of the server itself or of my code, I just installed a module incorrectly while setting up.
I am sorry for all the time other people spent on this issue.


Instruction: Command Errored Exit Status 1: - Pytorch Object Detection

Input:
I tried to follow this tutorial to learn how to run my own object detection, but I am running into an error that I can't seem to fix. I found a solution on some git hubs issues pages. They suggested running: !pip install git+https://github.com/philferriere/cocoapi.git, but I still get the same error. I am using Google Colab. Any suggestions?
input:
!pip installgit+https://github.com/cocodataset/cocoapi.git

output:
Collecting git+https://github.com/cocodataset/cocoapi.git
  Cloning https://github.com/cocodataset/cocoapi.git to /tmp/pip-req-build-u21a5wjs
  Running command git clone -q https://github.com/cocodataset/cocoapi.git /tmp/pip-req-build-u21a5wjs
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Output:
This is what is written in https://github.com/cocodataset/cocoapi

To install:
For Python, run "make" under coco/PythonAPI

So you cannot do pip install even in local.
Follow these steps in your google colaboratory to install,
!git clone https://github.com/cocodataset/cocoapi.git
# by default you are in /content in google colab use !ls to check
!ls
cd cocoapi
!ls
# you will see these,  common   license.txt  LuaAPI  MatlabAPI  PythonAPI  README.txt  results
cd PythonAPI
!make
!make install

Now your cocoapi github repository is installed and is ready to use.
As a sample code to test you can try these lines, it should show no errors.
from pycocotools.coco import COCO


Instruction: Batch-wise beam search in pytorch

Input:
I'm trying to implement a beam search decoding strategy in a text generation model. This is the function that I am using to decode the output probabilities.
def beam_search_decoder(data, k):
    sequences = [[list(), 0.0]]
    # walk over each step in sequence
    for row in data:
        all_candidates = list()
        for i in range(len(sequences)):
            seq, score = sequences[i]
            for j in range(len(row)):
                candidate = [seq + [j], score - torch.log(row[j])]
                all_candidates.append(candidate)
        # sort candidates by score
        ordered = sorted(all_candidates, key=lambda tup:tup[1])
        sequences = ordered[:k]
    return sequences

Now you can see this function is implemented with batch_size 1 in mind. Adding another loop for batch size would make the algorithm O(n^4). It is slow as it is now. Is there any way to improve the speed of this function. My model output is usually of the size (32, 150, 9907) which follows the format (batch_size, max_len, vocab_size)

Output:
Below is my implementation, which may be a little bit faster than the for loop implementation.
import torch

def beam_search_decoder(post, k):
    """Beam Search Decoder

    Parameters:

        post(Tensor) – the posterior of network.
        k(int) – beam size of decoder.

    Outputs:

        indices(Tensor) – a beam of index sequence.
        log_prob(Tensor) – a beam of log likelihood of sequence.

    Shape:

        post: (batch_size, seq_length, vocab_size).
        indices: (batch_size, beam_size, seq_length).
        log_prob: (batch_size, beam_size).

    Examples:

        >>> post = torch.softmax(torch.randn([32, 20, 1000]), -1)
        >>> indices, log_prob = beam_search_decoder(post, 3)

    """

    batch_size, seq_length, _ = post.shape
    log_post = post.log()
    log_prob, indices = log_post[:, 0, :].topk(k, sorted=True)
    indices = indices.unsqueeze(-1)
    for i in range(1, seq_length):
        log_prob = log_prob.unsqueeze(-1) + log_post[:, i, :].unsqueeze(1).repeat(1, k, 1)
        log_prob, index = log_prob.view(batch_size, -1).topk(k, sorted=True)
        indices = torch.cat([indices, index.unsqueeze(-1)], dim=-1)
    return indices, log_prob


Instruction: RuntimeError: Can only calculate the mean of floating types. Got Byte instead. for mean += images_data.mean(2).sum(0)

Input:
I have the following pieces of code:
# Device configuration
device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
seed = 42
np.random.seed(seed)
torch.manual_seed(seed)

# split the dataset into validation and test sets
len_valid_set = int(0.1*len(dataset))
len_train_set = len(dataset) - len_valid_set

print("The length of Train set is {}".format(len_train_set))
print("The length of Test set is {}".format(len_valid_set))

train_dataset , valid_dataset,  = torch.utils.data.random_split(dataset , [len_train_set, len_valid_set])

# shuffle and batch the datasets
train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=8, shuffle=True, num_workers=4)
test_loader = torch.utils.data.DataLoader(valid_dataset, batch_size=8, shuffle=True, num_workers=4)

print("LOADERS",
    len(dataloader),
    len(train_loader),
    len(test_loader))

The length of Train set is 720
The length of Test set is 80
LOADERS 267 90 10
mean = 0.0
std = 0.0
nb_samples = 0.0
for data in train_loader:
    images, landmarks = data["image"], data["landmarks"]
    batch_samples = images.size(0)

    images_data = images.view(batch_samples, images.size(1), -1)
    mean += images_data.mean(2).sum(0)
    std += images_data.std(2).sum(0)
    nb_samples += batch_samples

mean /= nb_samples
std /= nb_samples

And I get this error:
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-23-9e47ddfeff5e> in <module>
      7
      8     images_data = images.view(batch_samples, images.size(1), -1)
----> 9     mean += images_data.mean(2).sum(0)
     10     std += images_data.std(2).sum(0)
     11     nb_samples += batch_samples

RuntimeError: Can only calculate the mean of floating types. Got Byte instead.

The fixed code is taken from https://stackoverflow.com/a/64349380/2414957 it worked for dataloader but not train_loader
Also, these are the results of
print(type(images_data))
print(images_data)

We have:
<class 'torch.Tensor'>
tensor([[[74, 74, 74,  ..., 63, 63, 63],
         [73, 73, 73,  ..., 61, 61, 61],
         [75, 75, 75,  ..., 61, 61, 61],
         ...,
         [74, 74, 74,  ..., 38, 38, 38],
         [75, 75, 75,  ..., 39, 39, 39],
         [72, 72, 72,  ..., 38, 38, 38]],

        [[75, 75, 75,  ..., 65, 65, 65],
         [75, 75, 75,  ..., 62, 62, 62],
         [75, 75, 75,  ..., 63, 63, 63],
         ...,
         [71, 71, 71,  ..., 39, 39, 39],
         [74, 74, 74,  ..., 38, 38, 38],
         [73, 73, 73,  ..., 37, 37, 37]],

        [[72, 72, 72,  ..., 62, 62, 62],
         [74, 74, 74,  ..., 63, 63, 63],
         [75, 75, 75,  ..., 61, 61, 61],
         ...,
         [74, 74, 74,  ..., 38, 38, 38],
         [74, 74, 74,  ..., 39, 39, 39],
         [73, 73, 73,  ..., 37, 37, 37]],

        ...,

        [[75, 75, 75,  ..., 63, 63, 63],
         [73, 73, 73,  ..., 63, 63, 63],
         [74, 74, 74,  ..., 62, 62, 62],
         ...,
         [74, 74, 74,  ..., 38, 38, 38],
         [73, 73, 73,  ..., 39, 39, 39],
         [73, 73, 73,  ..., 37, 37, 37]],

        [[73, 73, 73,  ..., 62, 62, 62],
         [75, 75, 75,  ..., 62, 62, 62],
         [74, 74, 74,  ..., 63, 63, 63],
         ...,
         [73, 73, 73,  ..., 39, 39, 39],
         [74, 74, 74,  ..., 38, 38, 38],
         [74, 74, 74,  ..., 38, 38, 38]],

        [[74, 74, 74,  ..., 62, 62, 62],
         [74, 74, 74,  ..., 63, 63, 63],
         [74, 74, 74,  ..., 62, 62, 62],
         ...,
         [74, 74, 74,  ..., 38, 38, 38],
         [73, 73, 73,  ..., 38, 38, 38],
         [72, 72, 72,  ..., 36, 36, 36]]], dtype=torch.uint8)

When I tried
images_data = images_data.float()
mean += images_data.mean(2).sum(0)

I didn't get a tensor for 3 values for mean and 3 values for std like I expected but got a very large tensor (each torch.Size([600]))

Output:
As the error says, your images_data is a ByteTensor, i.e. has dtype uint8. Torch refuses to compute the mean of integers. You can convert the data to float with:
(images_data * 1.0).mean(2)

Or
torch.Tensor.float(images_data).mean(2)


Instruction: Run pytorch in pyodide?

Input:
Is there any way I can run the python library pytorch in pyodide? I tried installing pytorch with micropip but it gives this error message:

Couldn't find a pure Python 3 wheel for 'pytorch'

Output:
In Pyodide micropip only allows to install pure python wheels (i.e. that don't have compiled extensions). The filename for those wheels ends with none-any.whl (see PEP 427).
If you look at Pytorch wheels currently available on PyPi, their filenames ends with e.g. x86_64.whl so it means that they would only work on the x86_64 architecture and not in the WebAssembly VM.
The general solution to this is to add a package to the Pyodide build system. However in the case of pytorch, there is a blocker that cffi is currently not supported in pyodide (GH-pyodide#761),  while it's required at runtime by pytorch (see an example of build setup from conda-forge). So it is unlikely that pytorch would be availble in pyodide in the near future.


Instruction: I get a tensor of 600 values instead of 3 values for mean and std of train_loader in PyTorch

Input:
I am trying to Normalize my images data and for that I need to find the mean and std for train_loader.
mean = 0.0
std = 0.0
nb_samples = 0.0
for data in train_loader:
    images, landmarks = data["image"], data["landmarks"]
    batch_samples = images.size(0)
    images_data = images.view(batch_samples, images.size(1), -1)
    mean +=  torch.Tensor.float(images_data).mean(2).sum(0)
    std += torch.Tensor.float(images_data).std(2).sum(0)
    ###mean += images_data.mean(2).sum(0)
    ###std += images_data.std(2).sum(0)
    nb_samples += batch_samples

mean /= nb_samples
std /= nb_samples

the mean and std here are each a torch.Size([600])
When I tried (almost) same code on dataloader, it worked as expected:
# code from https://discuss.pytorch.org/t/about-normalization-using-pre-trained-vgg16-networks/23560/6?u=mona_jalal
mean = 0.0
std = 0.0
nb_samples = 0.0
for data in dataloader:
    images, landmarks = data["image"], data["landmarks"]
    batch_samples = images.size(0)

    images_data = images.view(batch_samples, images.size(1), -1)
    mean += images_data.mean(2).sum(0)
    std += images_data.std(2).sum(0)
    nb_samples += batch_samples

mean /= nb_samples
std /= nb_samples

and I got:
mean is: tensor([0.4192, 0.4195, 0.4195], dtype=torch.float64), std is: tensor([0.1182, 0.1184, 0.1186], dtype=torch.float64)
So my dataloader is:
class MothLandmarksDataset(Dataset):
    """Face Landmarks dataset."""

    def __init__(self, csv_file, root_dir, transform=None):
        """
        Args:
            csv_file (string): Path to the csv file with annotations.
            root_dir (string): Directory with all the images.
            transform (callable, optional): Optional transform to be applied
                on a sample.
        """
        self.landmarks_frame = pd.read_csv(csv_file)
        self.root_dir = root_dir
        self.transform = transform

    def __len__(self):
        return len(self.landmarks_frame)

    def __getitem__(self, idx):
        if torch.is_tensor(idx):
            idx = idx.tolist()

        img_name = os.path.join(self.root_dir, self.landmarks_frame.iloc[idx, 0])
        image = io.imread(img_name)
        landmarks = self.landmarks_frame.iloc[idx, 1:]
        landmarks = np.array([landmarks])
        landmarks = landmarks.astype('float').reshape(-1, 2)
        sample = {'image': image, 'landmarks': landmarks}

        if self.transform:
            sample = self.transform(sample)

        return sample

transformed_dataset = MothLandmarksDataset(csv_file='moth_gt.csv',
                                           root_dir='.',
                                           transform=transforms.Compose(
                                               [
                                               Rescale(256),
                                               RandomCrop(224),

                                               ToTensor()
                                               ]
                                                                        )
                                           )

dataloader = DataLoader(transformed_dataset, batch_size=3,
                        shuffle=True, num_workers=4)

and train_loader is:
# Device configuration
device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
seed = 42
np.random.seed(seed)
torch.manual_seed(seed)

# split the dataset into validation and test sets
len_valid_set = int(0.1*len(dataset))
len_train_set = len(dataset) - len_valid_set

print("The length of Train set is {}".format(len_train_set))
print("The length of Test set is {}".format(len_valid_set))

train_dataset , valid_dataset,  = torch.utils.data.random_split(dataset , [len_train_set, len_valid_set])

# shuffle and batch the datasets
train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=8, shuffle=True, num_workers=4)
test_loader = torch.utils.data.DataLoader(valid_dataset, batch_size=8, shuffle=True, num_workers=4)

Please let me know if more information is needed.
I basically need to get 3 values for mean of train_loader and 3 values for std of train_loader to use as args for Normalize.
images_data in dataloader is torch.Size([3, 3, 50176]) inside the loop and images_data in train_loader is torch.Size([8, 600, 2400])

Output:
First, the weird shape you get for your mean and std ([600]) is unsuprising, it is due to your data having the shape [8, 600, 800, 3]. Basically, the channel dimension is the last one here, so when you try to flatten your images with
# (N, 600, 800, 3) -> [view] -> (N, 600, 2400 = 800*3)
images_data = images.view(batch_samples, images.size(1), -1)

You actually perform a weird operation that fuses together the width and channel dimensions of your image which is now [8, 600, 2400]. Thus, applying
# (8, 600, 2400) -> [mean(2)] -> (8, 600) -> [sum(0)] -> (600)
data.mean(2).sum(0)

Creates a tensor of size [600] which is what you indeed get.
There are two quite simple solutions :
Either you start by permuting the dimensions to make the 2nd dimension the channel one :
batch_samples = images.size(0)
# (N, H, W, C) -> (N, C, H, W)
reordered = images.permute(0, 3, 1, 2)
# flatten image into (N, C, H*W)
images_data = reordered.view(batch_samples, reordered.size(1), -1)
# mean is now (C) = (3)
mean += images_data.mean(2).sum(0)

Or you changes the axis along which to apply mean and sum
 batch_samples = images.size(0)
# flatten image into (N, H*W, C), careful this is not what you did
images_data = images.view(batch_samples, -1, images.size(1))
# mean is now (C) = (3)
mean += images_data.mean(1).sum(0)

Finally, why did dataloaderand trainloader behave differently ? Well I think it's because one is using dataset while the other is using transformedDataset. In TransformedDataset, you apply the toTensortransform which cast a PIL image into a torch tensor, and I think that pytorch is smart enough to permute your dimensions during this operation (and put the channels in the second dimension). In other word, your two datasets just do not yield images with identical format, they differ by a permutation of the axis.


Instruction: how to fix the torch.cuda.is_available() False problem without restarting the machine?

Input:
I have:
$ python
Python 3.7.6 (default, Jan  8 2020, 19:59:22)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.cuda.is_available()
False
>>> quit()

$ nvidia-smi
Wed Oct 14 21:28:50 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.51.06    Driver Version: 450.51.06    CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce RTX 2070    Off  | 00000000:01:00.0 Off |                  N/A |
| N/A   47C    P8     9W /  N/A |   1257MiB /  7982MiB |     11%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1424      G   /usr/lib/xorg/Xorg                823MiB |
|    0   N/A  N/A      1767      G   /usr/bin/gnome-shell              407MiB |
|    0   N/A  N/A      6420      G   /usr/lib/firefox/firefox            2MiB |
|    0   N/A  N/A      6949      G   /usr/lib/firefox/firefox            2MiB |
|    0   N/A  N/A      7447      G   /usr/lib/firefox/firefox            2MiB |
|    0   N/A  N/A      8888      G   /usr/lib/firefox/firefox            2MiB |
|    0   N/A  N/A      9218      G   /usr/lib/firefox/firefox            2MiB |
|    0   N/A  N/A      9282      G   /usr/lib/firefox/firefox            2MiB |
|    0   N/A  N/A     65854      G   /usr/lib/firefox/firefox            2MiB |
|    0   N/A  N/A     70801      G   /usr/lib/firefox/firefox            2MiB |
+-----------------------------------------------------------------------------+

Is there a way I could fix this problem without having to restart my machine?
I have Ubuntu 20.04 and PyTorch 1.6.0
After I restarted the machine, here's what I get:

Output:
This happens quite often to ubuntu users (I am not so sure about other distros). I have noticed this behavior especially when I leave my machine on sleep. Without restarting you could run the following commands as mentioned in this thread
sudo rmmod nvidia_uvm
sudo modprobe nvidia_uvm


Instruction: Training Sparse Autoencoders

Input:
My dataset consists of vectors that are massive.  The data points are all mostly zeros with ~3% of the features being 1.  Essentially my data is super sparse and I am attempting to train an autoencoder however my model is learning just to recreate vectors of all zeros.
Are there any techniques to prevent this?  I have tried replacing mean squared error with dice loss but it completely stopped learning.  My other thoughts would be to use a loss function that favors guessing 1s correctly rather than zeros.  I have also tried using a sigmoid and linear last activation with no clear winner.  Any ideas would be awesome.

Output:
It seems like you are facing a severe "class imbalance" problem.

Have a look at focal loss. This loss is designed for binary classification with severe class imbalance.

Consider "hard negative mining": that is, propagate gradients only for part of the training examples - the "hard" ones.
see, e.g.:
Abhinav Shrivastava, Abhinav Gupta and Ross Girshick Training Region-based Object Detectors with Online Hard Example Mining (CVPR 2016).


Instruction: PyTorch Circular Padding in one Dimension

Input:
for a convolution i want to apply a circular padding in one dimension and a zero padding in all other dimension. How can i do this?
For the convolution there are 28 channels and fore the the data is described in spherical bins. There are 20 bins for radius times 20 bins for polar times 20 bins for inclination.
The circular padding should only be applied for the inclination.
Small Example
# Example:
x = torch.tensor([[1,2,3],
                  [4,5,6],
                  [7,8,9]])
y = sphere_pad(x, pad=(0, 1))

# y is now tensor([[3, 1, 2, 3, 1],
#                  [6, 4, 5, 6, 4],
#                  [9, 7, 8, 9, 7]])

I have tried to apply
def sphere_pad(x, pad=(1,1)):
    return x.repeat(*x.shape)[
        (x.shape[0]-pad[0]):(2*x.shape[0]+pad[0]),
        (x.shape[1]-pad[1]):(2*x.shape[1]+pad[1])]

and then apply a convolution with a normal zero padding ( and no padding in the last dimension).
This works for a small example but this method exceeds the GPU memory for the actual problem size.
Are there other ways?

Output:
Using numpy, you could do a wrap padding so the array gets wrapped along the second axis:
np.pad(x, ((0,0),(1,1)), mode='wrap')
array([[3, 1, 2, 3, 1],
       [6, 4, 5, 6, 4],
       [9, 7, 8, 9, 7]])


Instruction: Trying to use Tensorboard on Google Colab

Input:
The page below gives informations about Tensorboard:
https://pytorch.org/docs/stable/tensorboard.html
I am using Google Colab and when i write the following
instructions(which are in the link above):
!pip install tensorboard
tensorboard --logdir=runs

it sends me the following error message:
 File "<ipython-input-111-949c7e8e565e>", line 2
tensorboard --logdir=runs
                         ^
SyntaxError: can't assign to operator

so when i copy paste their
own example:
from torch.utils.tensorboard import SummaryWriter
writer = SummaryWriter()
x = range(100)
for i in x:
    writer.add_scalar('y=2x', i * 2, i)
writer.close()

it does not return the expected graph.
Could someone help me fix this problem?Thank you in advance!

Output:
As explained in How to use Tensorboard with PyTorch in Google Colab.
In Google Colab you should start Tensorboard magic at the start of your code with:
%load_ext tensorboard

and after you define a summary file you need to insatiate Tensorboard with
%tensorboard --logdir $tensorboard_dir

where Tensorboard dir is a script variable of type str indicating where the summary file is.


Instruction: How can I repartition RDD by key and then pack it to shards?

Input:
I have many files containing millions of rows in format:
id, created_date, some_value_a, some_value_b, some_value_c

This way of repartitioning was super slow and created for me over million of small ~500b files:
    rdd_df = rdd.toDF(["id", "created_time", "a", "b", "c"])
    rdd_df.write.partitionBy("id").csv("output")

I would like to achieve output files, where each file contains like 10000 unique IDs and all their rows.
How could I achieve something like this?

Output:
You can repartition by adding a Random Salt key.
val totRows = rdd_df.count

val maxRowsForAnId = rdd_df.groupBy("id").count().agg(max("count"))
val numParts1 = totRows/maxRowsForAnId

val totalUniqueIds = rdd_df.select("id").distinct.count
val numParts2 = totRows/(10000*totalUniqueIds)

val numPart = numParts1.min(numParts2)

rdd_df
 .repartition(numPart,col("id"),rand)
 .csv("output")

The main concept is each partition will be written as 1 file. SO you would have  bring your required rows in to 1 partition by repartition(numPart,col("id"),rand).
The first 4-5 operations is just to calculate how many partitions we need to achieve almost 10000 ids per file.

Calculate assuming 10000 ids per partition
Corner case : if a single id has too many rows and doesn't fit in the above calculated partition size.
Hence we calculate no of paritition according to the largest count of ID present
Take min of the 2 noOfPartitons

rand is necessary so, that we can bring multiple IDs in a single partition
NOTE : Although this will give you larger files and each file will contain a set of unique ids for sure. But this involves shuffling , due to which your operation actually might be slower than the code you have mentioned in question.


Instruction: predict the position of an image in another image

Input:
If one image is a part of another image, then how to compute the accurate location in deep learning way?
Now I could compute this by extracting and matching key points using OpenCV, but I hope to solve it with neural networks.
Any ideas to design the networks and loss functions?
Thanks very much.

Output:
This is a detection problem. The simplest approach to do it is to create a a network with two heads, one for classification and the other for the bounding box (regression).
you feed your network with the image and respective label, and sum the lossess and do a backward. train for some epochs and you'll get your self a detection model that you can use to detect what you need. but its just a simple approach and it can get much more complex.
You may as well skip this and use an existing detection architecture or better framework which simplifies your life much better.
For Tensorflow I belive you can use ObjectDetctionAPI and for Pytorch you can use Detectron, Detectron2, mmdetection among others.


Instruction: AttributeError: module 'torch' has no attribute 'hstack'

Input:
I am following this doc for hstack.
a = torch.tensor([1, 2, 3])
b = torch.tensor([4, 5, 6])
torch.hstack((a,b))

But I keep getting the error:
AttributeError: module 'torch' has no attribute 'hstack'

Here is the torch version that results in this error:
torch.__version__
'1.6.0+cpu'

What am I doing wrong?

Output:
Apparently you are calling a function that does not exist YET in your PyTorch version --this is what the error message is about.
Your link points to the help page related to developers preview: note .8.0a0+342069f version number in the top left corner. When clicking, Click here to view docs for latest stable release. link - an error message come.
This function becomes available in torch version 1.8.0. --until then consider using torch.cat with `dim=1'.
torch.cat([a,b], dim=1)  # a, b - 2d torch.Tensors


Instruction: Size mismatch in fully connected layers

Input:
I build the following simply model in pytorch as a first run and I am gettign a size mismatch error that does not make sense as out_feat always equals in_feat for the subsequent layer...
class Network(nn.Module):

    def __init__(self):
        super(Network,self).__init__()
        #first linear block

        self.fc1=nn.Linear(32,1024)
        self.b1=nn.BatchNorm1d(1024)

        #Two Linear 1
        self.fc2=nn.Linear(1024,1024)
        self.b2=nn.BatchNorm1d(1024)
        self.fc3=nn.Linear(1024,1024)
        self.b3=nn.BatchNorm1d(1024)

        #Two Linear 2
        self.fc4=nn.Linear(1024,1024)
        self.b4=nn.BatchNorm1d(1024)
        self.fc5=nn.Linear(1024,1024)
        self.b5=nn.BatchNorm1d(1024)

        #Final Linear Layer
        self.fc6=nn.Linear(1024,48)

    def forward(self,x):
        x1=self.fc1(x)
        x1=self.b1(x1)
        x1=nn.ReLU(x1)

        x2=self.fc2(x1)
        x2=self.b2(x2)
        x2=nn.ReLU(x2)
        x2=self.fc3(x2)
        x2=self.b3(x2)
        x2=nn.ReLU(x2)

        x3=x1+x2

        x4=self.fc4(x3)
        x4=self.b4(x4)
        x4=nn.ReLU(x4)
        x4=self.fc5(x4)
        x4=self.b5(x4)
        x4=nn.ReLU(x4)

        x5=x3+x4

        x6=self.fc6(x5)

        return x6

model=Network()
zeros=np.zeros((1,32))
outputs=model(torch.FloatTensor(zeros))
RuntimeError: size mismatch, m1: [1 x 32], m2: [1024 x 32] at ..\aten\src\TH/generic/THTensorMath.cpp:41

I do not understand how I am getting this error when all the dimensions match does anyone see the issue?
=================================================================
Layer (type:depth-idx)                   Param #
=================================================================
├─Linear: 1-1                            33,792
├─BatchNorm1d: 1-2                       4,096
├─Linear: 1-3                            1,049,600
├─BatchNorm1d: 1-4                       4,096
├─Linear: 1-5                            1,049,600
├─BatchNorm1d: 1-6                       4,096
├─Linear: 1-7                            1,049,600
├─BatchNorm1d: 1-8                       4,096
├─Linear: 1-9                            1,049,600
├─BatchNorm1d: 1-10                      4,096
├─Linear: 1-11                           49,200
=================================================================
Total params: 4,301,872
Trainable params: 4,301,872
Non-trainable params: 0

here is model summary

Output:
Batch normalization works when batch size is greater than 1, so an input of shape (1, 32) won't work. Try a larger batch size, like 2.
Moreover, you're trying to use ReLU in the form x = nn.ReLU(x). This is wrong, as nn.ReLU is a layer. This line of code returns you the ReLU layer itself rather than a tensor. Either define nn.ReLU() layers in your init method, or use F.relu(x) or nn.ReLU()(x). Like so:
import torch
from torch import nn
import torch.nn.functional as F

class Network(nn.Module):
    def __init__(self):
        super(Network,self).__init__()
        #first linear block

        self.fc1=nn.Linear(32,1024)
        self.b1=nn.BatchNorm1d(1024)

        #Two Linear 1
        self.fc2=nn.Linear(1024,1024)
        self.b2=nn.BatchNorm1d(1024)
        self.fc3=nn.Linear(1024,1024)
        self.b3=nn.BatchNorm1d(1024)

        #Two Linear 2
        self.fc4=nn.Linear(1024,1024)
        self.b4=nn.BatchNorm1d(1024)
        self.fc5=nn.Linear(1024,1024)
        self.b5=nn.BatchNorm1d(1024)

        #Final Linear Layer
        self.fc6=nn.Linear(1024,48)

    def forward(self,x):
        x1=self.fc1(x)
        x1=self.b1(x1)
        x1=F.relu(x1)

        x2=self.fc2(x1)
        x2=self.b2(x2)
        x2=F.relu(x2)
        x2=self.fc3(x2)
        x2=self.b3(x2)
        x2=F.relu(x2)

        x3=x1+x2

        x4=self.fc4(x3)
        x4=self.b4(x4)
        x4=F.relu(x4)
        x4=self.fc5(x4)
        x4=self.b5(x4)
        x4=F.relu(x4)

        x5=x3+x4

        x6=self.fc6(x5)

        return x6

model=Network()
zeros=torch.zeros((10, 32))
outputs=model(zeros)
print(outputs.shape)
# torch.Size([10, 48])


Instruction: Is it possible to add own function in transform.compose in pytorch

Input:
I am using a pre-trained Alex model. I am running this model on some random image dataset. I want to convert RGB images to YCbCr images before training.
I am wondering is it possible to add a function on my own to transform.compose, For example:
transform = transforms.Compose([
  ycbcr(), #something like this
  transforms.Resize((224, 224)),
  transforms.ToTensor(),
  transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
])

where,
def ycbcr(img):
   img = cv2.imread(img)
   img = cv2.cvtColor(img, cv2.COLOR_BGR2ycbcr)
   t = torch.from_numpy(img)
 return t

training_dataset = datasets.ImageFolder(link_train ,transform = transform_train)

training_loader = torch.utils.data.DataLoader(training_dataset, batch_size=96, shuffle=True)

Is this process correct? Please help me on how to proceed?

Output:
You can pass a custom transformation to torchvision.transform by defining a class.
To understand better I suggest that you read the documentations.
In your case it will be something like the following:
class ycbcr(object):
    def __call__(self, img):
        """
        :param img: (PIL): Image

        :return: ycbr color space image (PIL)
        """
        img = cv2.imread(img)
        img = cv2.cvtColor(img, cv2.COLOR_BGR2ycbcr)
        # t = torch.from_numpy(img)

        return Image.fromarray(t)

    def __repr__(self):
        return self.__class__.__name__+'()'

Notice that it gets a PIL image and return a PIL image. So you might want to adjust your code properly. But this is the general way to define a custom transformation.


Instruction: What is the difference between a .ckpt and a .pth file in Pytorch?

Input:
I am following a code from GitHub that uses Pytorch.
The model is saved using :
model.save(ARGS.working_dir + '/model_%d.ckpt' % (epoch+1)).

What is the difference between using .pth and .ckpt in Pytorch?

Output:
There is no difference. the extension in Pytorch models that you see is something random. You can choose anything.
People usually use pth to indicate a PyTorcH model (and hence .pth). but then again its completely up to you on how you want to save your model.


Instruction: OSError: libcurand.so.10: cannot open shared object file: No such file or directory

Input:
I am working on Nvidia Jetson Tx2 (with JETPACK 4.2) and installed the pytorch following this link.
When I am importing torch in python its giving me an error OSError: libcurand.so.10: cannot open shared object file: No such file or directory I have tried all the options but nothing worked.
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-10.0/lib64
export LIBRARY_PATH=$LIBRARY_PATH:/usr/local/cuda-10.0/lib64
export PATH=$PATH:/usr/local/cuda-10.0/lib64

Any guidance to debug the issue is requested. Thanks

Output:
emm
I found this answer as followed...
https://forums.developer.nvidia.com/t/mounting-cuda-onto-l4t-docker-image-issues-libcurand-so-10-cannot-open-no-such-file-or-directory/121545
The key is : "You can use JetPack4.4 for CUDA 10.2 and JetPack4.3 for CUDA 10.0."
Maybe downloading Pytorch v1.4.0 and Jetpack 4.2/4.3 would solve this question...
Anyway, it is helped for me... good luck
enter image description here


Instruction: Use CrossEntropyLoss with LogSoftmax

Input:
From the Pytorch documentation, CrossEntropyLoss combines LogSoftMax and NLLLoss together in one single class
But I am curious; what happens if we use both CrossEntropyLoss for criterion and LogSoftMax in my classifier:
model_x.fc = nn.Sequential (nn.Linear(num_ftrs, 2048, bias=True), nn.ReLU(),
                               nn.Linear(2048, 1024 ), nn.ReLU(),
                               nn.Linear(1024 ,256), nn.ReLU(),
                                nn.Linear(256 ,128), nn.ReLU(),
                               nn.Linear(128, num_labels),nn.LogSoftmax(dim = 1))

criterion = nn.CrossEntropyLoss()

Then if i have saved a trained model using the code above, how can I check the criterion used by the saved model?

Output:
TL;DR: You will decrease the expressivity of the model because it only can produce relatively flat distribution.
What you suggest in the snippet actually means applying the softmax normalization twice. This will give you a distribution with the same rank of probabilities, but it will be much flatter and it will prevent the model from using a low-entropy output distribution. The output of the linear layer can be in theory any number. In practice, the logits are both positive and negative numbers, which allows producing spiky distributions. After softmax, you have probabilities between 0 and 1, so log-softmax will give you negative numbers.
Typically, models are saved without the loss function. Unless you explicitly saved the loss as well, there is no way to find it out.


Instruction: Is there a way to convert the quint8 pytorch format to np.uint8 format?

Input:
I'm using the code below to get the quantized unsiged int 8 format in pytorch. However, I'm not able to convert the quant variable to the to np.uint8. Is there possible to do that?
import torch

quant = torch.quantize_per_tensor(torch.tensor([-1.0, 0.352, 1.321, 2.0]), 0.1, 10, torch.quint8)

Output:
This can be done using torch.int_repr()
import torch
import numpy as np

# generate a test float32 tensor
float32_tensor = torch.tensor([-1.0, 0.352, 1.321, 2.0])
print(f'{float32_tensor.dtype}\n{float32_tensor}\n')

# convert to a quantized uint8 tensor. This format keeps the values in the range of
# the float32 format, with the resolution of a uint8 format (256 possible values)
quint8_tensor = torch.quantize_per_tensor(float32_tensor, 0.1, 10, torch.quint8)
print(f'{quint8_tensor.dtype}\n{quint8_tensor}\n')

# map the quantized data to the actual uint8 values (and then to an np array)
uint8_np_ndarray = torch.int_repr(quint8_tensor).numpy()
print(f'{uint8_np_ndarray.dtype}\n{uint8_np_ndarray}')

Output
torch.float32
tensor([-1.0000,  0.3520,  1.3210,  2.0000])

torch.quint8
tensor([-1.0000,  0.4000,  1.3000,  2.0000], size=(4,), dtype=torch.quint8,
       quantization_scheme=torch.per_tensor_affine, scale=0.1, zero_point=10)

uint8
[ 0 14 23 30]


Instruction: how to convert outlogits to tokens?

Input:
i have a forward function in allenNlp given by :
  def forward(self, input_tokens, output_tokens):
    '''
    This is the main process of the Model where the actual computation happens.
    Each Instance is fed to the forward method.
    It takes dicts of tensors as input, with same keys as the fields in your Instance (input_tokens, output_tokens)
    It outputs the results of predicted tokens and the evaluation metrics as a dictionary.
    '''

    mask = get_text_field_mask(input_tokens)
    embeddings = self.embedder(input_tokens)
    rnn_hidden = self.rnn(embeddings, mask)
    out_logits = self.hidden2out(rnn_hidden)
    loss = sequence_cross_entropy_with_logits(out_logits, output_tokens['tokens'], mask)

    return {'loss': loss}

the out_logits variable contains probabilities of tokens, how to dispaly these tokens.
the outlogits gives :
 array([[ 0.02416356,  0.0195566 , -0.03279119,  0.057118  ,  0.05091334,
    -0.01906729, -0.05311333,  0.04695245,  0.06872341,  0.05173637,
    -0.03523348, -0.00537474, -0.03946163, -0.05817827, -0.04316377,
    -0.06042208,  0.01190596,  0.00574979,  0.01183304,  0.02330608,
     0.04587644,  0.02319966,  0.0020873 ,  0.03781978, -0.03975108,
    -0.0131919 ,  0.00393738,  0.04785313,  0.00159995,  0.05751844,
     0.05420169, -0.01404533, -0.02716331, -0.03871592,  0.00949999,
    -0.02924301,  0.03504215,  0.00397302, -0.0305252 , -0.00228448,
     0.04034173,  0.01458408],
   [ 0.02050283,  0.0204745 , -0.03081856,  0.06295916,  0.04601778,
    -0.0167818 , -0.05653084,  0.05017883,  0.07212739,  0.06197165,
    -0.03590995, -0.01142827, -0.03807197, -0.05942211, -0.0375165 ,
    -0.06769539,  0.01200251,  0.01012686,  0.01514241,  0.01875677,
     0.04499928,  0.02748671,  0.0012517 ,  0.04062563, -0.04049949,
    -0.01986902,  0.00630998,  0.05092276,  0.00276728,  0.05341531,
     0.05047017, -0.01111878, -0.03038253, -0.04320357,  0.01768938,
    -0.03470382,  0.03567442,  0.00776757, -0.02703476, -0.00392571,
     0.04700187,  0.01671317]] dtype=float32)}

i want to convert the last array to token ?

Output:
In allennlp you have access to the self.vocab attribute with Vocabulary. get_token_from_index.
Usually to select a token from the logits one would apply a softmax (in order to have all the probability summing to 1) and then pick the most probable one.
If you want to decode sequences from a model maybe you should look into [BeamSearch] (https://docs.allennlp.org/master/api/nn/beam_search/#beamsearch).


Instruction: What is the correct way to use a PyTorch Module inside a PyTorch Function?

Input:
We have a custom torch.autograd.Function z(x, t) which computes an output y in a way not amenable to direct automatic differentiation, and have computed the Jacobian of the operation with respect to its inputs x and t, so we can implement the backward method.
However, the operation involves making several internal calls to a neural network, which we have implemented for now as a stack of torch.nn.Linear objects, wrapped in net, a torch.nn.Module. Mathematically, these are parameterized by t.
Is there any way that we can have net itself be an input to the forward method of z? Then, we would return from our backward the list of products of the upstream gradient Dy and parameter Jacobia dydt_i, one for each of the parameters ti that are children of net (in addition to Dy*dydx, although x is data and does not need gradient accumulation).
Or do we really instead need to take t (actually a list of individual t_i), and reconstruct internally in z.forward the actions of all the Linear layers in net?

Output:
I guess you could create a custom functor that inherits torch.autograd.Function and make the forward and backward methods non-static (i.e remove the @staticmethod in this example so that net could be an attribute of your functor. that would look like
class MyFunctor(torch.nn.autograd.Function):
    def __init(net):
         self.net = net

     def forward(ctx, x, t):
         #store x and t in ctx in the way you find useful
         # not sure how t is involved here
         return self.net(x)

     def backward(ctx, grad):
         # do your backward stuff

net = nn.Sequential(nn.Linear(...), ...)
z = MyFunctor(net)
y = z(x, t)

This will yield a warning that you are using a deprecated legacy way of creating autograd functions (because of the non-static methods), and you need to be extra careful with zeroing the gradients in netafter having backpropagated. So not really convenient, but I am not aware of any better way to have a stateful autograd function.


Instruction: do I have to add softmax in def forward when I use torch.nn.CrossEntropyLoss

Input:
https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html
When I read the contents above, I understood that torch.nn.CrossEntropy already computes exp score of the last layer. So I thought the forward function doesn't have to include softmax. For example,
return self.fc(x) rather than return nn.softmax(self.fc(x)). However, I'm confused, for I've seen several implementations of ConvNet Classifiers that use both ways (they return with or without softmax while both use cross entropy loss).
Do this issues affect the performance of Classifier?? Which way is correct?

Output:
JHPark,
You are correct  - with torch.nn.CrossEntropyLoss there is no need to include softmax layer. If one does include softmax it will still lead to proper classification result, since softmax does not change which element has max score. However, if applied twice, it may distort relative levels of the outputs, making gradients less strong and potentially slowing training a bit.


Instruction: Can I use BERT as a feature extractor without any finetuning on my specific data set?

Input:
I'm trying to solve a multilabel classification task of 10 classes with a relatively balanced training set consists of ~25K samples and an evaluation set consists of ~5K samples.
I'm using the huggingface:
model = transformers.BertForSequenceClassification.from_pretrained(...

and obtain quite nice results (ROC AUC = 0.98).
However, I'm witnessing some odd behavior which I don't seem to make sense of -
I add the following lines of code:
for param in model.bert.parameters():
    param.requires_grad = False

while making sure that the other layers of the model are learned, that is:
[param[0] for param in model.named_parameters() if param[1].requires_grad == True]
gives
['classifier.weight', 'classifier.bias']

Training the model when configured like so, yields some embarrassingly poor results (ROC AUC = 0.59).
I was working under the assumption that an out-of-the-box pre-trained BERT model (without any fine-tuning) should serve as a relatively good feature extractor for the classification layers. So, where do I got it wrong?

Output:
From my experience, you are going wrong in your assumption

an out-of-the-box pre-trained BERT model (without any fine-tuning) should serve as a relatively good feature extractor for the classification layers.

I have noticed similar experiences when trying to use BERT's output layer as a word embedding value with little-to-no fine-tuning, which also gave very poor results; and this also makes sense, since you effectively have 768*num_classes connections in the simplest form of output layer. Compared to the millions of parameters of BERT, this gives you an almost negligible amount of control over intense model complexity. However, I also want to cautiously point to overfitted results when training your full model, although I'm sure you are aware of that.
The entire idea of BERT is that it is very cheap to fine-tune your model, so to get ideal results, I would advise against freezing any of the layers. The one instance in which it can be helpful to disable at least partial layers would be the embedding component, depending on the model's vocabulary size (~30k for BERT-base).


Instruction: Pytorch datatype/dimension confusion TypeError: 'Tensor' object is not callable

Input:
This piece of code is originally written in numpy and I'm trying to utilise GPU computation by rewriting it in pytorch, but as I'm new to pytorch a lot of problems occured to me. Firstly I'm confused by the dimension of the tensors. Sometimes after operating on the tensors, only transposing the tensor would fix the problem, is there anyway I can stop doing .t()? The major problem here is that in the line ar = torch.stack ... the error "TypeError: 'Tensor' object is not callable " occurs. Any suggestion/correction would be appreciated. Thxxx
def vec_datastr(vector):

    vector = vector.float()
    # Find the indices corresponding to non-zero entries
    index = torch.nonzero(vector)
    index = index.t()

    # Compute probability
    prob = vector ** 2
    if torch.sum(prob) == 0:
        prob = 0
    else:
        prob = prob / torch.sum(prob)

    d = depth(vector)
    CumProb = torch.ones((2**d-len(prob.t()),1), device ='cuda')
    cp = torch.cumsum(prob, dim=0)
    cp = cp.reshape((len(cp.t()),1))
    CumProb = torch.cat((cp, CumProb),0)
    vector = vector.t()
    prob = prob.t()

    ar = torch.stack((index, vector([index,1]), prob([index, 1]), CumProb([index, 1]))) # Problems occur here
    ar = ar.reshape((len(index), 4))

    # Store the data as a 4-dimensional array
    output = dict()
    output = {'index':ar[:,0], 'value': ar[:,1], 'prob':ar[:,2], 'CumProb': ar[:,3]}

    return output

Output:
ar = torch.stack(
    (index, vector([index, 1]), prob([index, 1]), CumProb([index, 1]))
)  # Problems occur here

vector is of type torch.Tensor. It has no __call__ defined. You are going for vector(...) (vector([index,1])) while you should slice the data directly like this: vector[index, 1]. Same goes for prob and CumProb.
Somehow, you do it correctly for ar with ar[:,0] so it might be a typo


Instruction: What makes BertGeneration and/or RobertaForCausalLM causal models? Where does the causal attention masking happen?

Input:
I am trying to use RobertaForCausalLM and/or BertGeneration for causal language modelling / next-word-prediction / left-to-right prediction. I can't seem to figure out where the causal masking is happening? I want to train teacher forcing with the ground-truth labels, but no information from future tokens to be included in the attention mechanism. For that I thought the model would need causal attention masking, but I don't see it being applied anywhere...
If anyone could point me to where this might be happening or why it is unnecessary, that would be helpful.
Thanks!

Output: I have found it. It happens in get_extended_attention_mask in modeling utils. Consider this question solved :slight_smile:


Instruction: How can I calculate FLOPs and Params without 0 weights neurons affected?

Input:
My Prune code is shown below, after running this, I will get a file named 'pruned_model.pth'.
import torch
from torch import nn
import torch.nn.utils.prune as prune
import torch.nn.functional as F
from cnn import net

ori_model = '/content/drive/My Drive/ECG_weight_prune/checkpoint_dir/model.pth'
save_path = '/content/drive/My Drive/ECG_weight_prune/checkpoint_dir/pruned_model.pth'
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

model = net().to(device)
model.load_state_dict(torch.load(ori_model))

module = model.conv1
print(list(module.named_parameters()))
print(list(module.named_buffers()))

prune.l1_unstructured(module, name="weight", amount=0.3)
prune.l1_unstructured(module, name="bias", amount=3)
print(list(module.named_parameters()))
print(list(module.named_buffers()))
print(module.bias)
print(module.weight)
print(module._forward_pre_hooks)
prune.remove(module, 'weight')
prune.remove(module, 'bias')
print(list(module.named_parameters()))
print(model.state_dict())
torch.save(model.state_dict(), save_path)

and results is :
[('weight', Parameter containing:
tensor([[[-0.0000, -0.3137, -0.3221,  ...,  0.5055,  0.3614, -0.0000]],

        [[ 0.8889,  0.2697, -0.3400,  ...,  0.8546,  0.2311, -0.0000]],

        [[-0.2649, -0.1566, -0.0000,  ...,  0.0000,  0.0000,  0.3855]],

        ...,

        [[-0.2836, -0.0000,  0.2155,  ..., -0.8894, -0.7676, -0.6271]],

        [[-0.7908, -0.6732, -0.5024,  ...,  0.2011,  0.4627,  1.0227]],

        [[ 0.4433,  0.5048,  0.7685,  ..., -1.0530, -0.8908, -0.4799]]],
       device='cuda:0', requires_grad=True)), ('bias', Parameter containing:
tensor([-0.7497, -1.3594, -1.7613, -2.0137, -1.1763,  0.4150, -1.6996, -1.5354,
         0.4330, -0.9259,  0.4156, -2.3099, -0.4282, -0.5199,  0.1188, -1.1725,
        -0.9064, -1.6639, -1.5834, -0.3655, -2.0727, -2.1078, -1.6431, -0.0694,
        -0.5435, -1.9623,  0.5481, -0.8255, -1.5108, -0.4029, -1.9759,  0.0522,
         0.0599, -2.2469, -0.5599,  0.1039, -0.4472, -1.1706, -0.0398, -1.9441,
        -1.5310, -0.0837, -1.3250, -0.2098, -0.1919,  0.4600, -0.8268, -1.0041,
        -0.8168, -0.8701,  0.3869,  0.1706, -0.0226, -1.2711, -0.9302, -2.0696,
        -1.1838,  0.4497, -1.1426,  0.0772, -2.4356, -0.3138,  0.6297,  0.2022,
        -0.4024,  0.0000, -1.2337,  0.2840,  0.4515,  0.2999,  0.0273,  0.0374,
         0.1325, -0.4890, -2.3845, -1.9663,  0.2108, -0.1144,  0.0544, -0.2629,
         0.0393, -0.6728, -0.9645,  0.3118, -0.5142, -0.4097, -0.0000, -1.5142,
        -1.2798,  0.2871, -2.0122, -0.9346, -0.4931, -1.4895, -1.1401, -0.8823,
         0.2210,  0.4282,  0.1685, -1.8876, -0.7459,  0.2505, -0.6315,  0.3827,
        -0.3348,  0.1862,  0.0806, -2.0277,  0.2068,  0.3281, -1.8045, -0.0000,
        -2.2377, -1.9742, -0.5164, -0.0660,  0.8392,  0.5863, -0.7301,  0.0778,
         0.1611,  0.0260,  0.3183, -0.9097, -1.6152,  0.4712, -0.2378, -0.4972],
       device='cuda:0', requires_grad=True))]

There are many zero weights existing. How can I calculate FLOPs and Params without counting calculations associated with these zero values?
I use the following code to calculate FLOPs and Params.
import torch
from cnn import net
from ptflops import get_model_complexity_info

ori_model = '/content/drive/My Drive/ECG_weight_prune/checkpoint_dir/model.pth'
pthfile = '/content/drive/My Drive/ECG_weight_prune/checkpoint_dir/pruned_model.pth'

model = net()
# model.load_state_dict(torch.load(ori_model))
model.load_state_dict(torch.load(pthfile))
# print(model.state_dict())

macs, params = get_model_complexity_info(model, (1, 260), as_strings=False,
                                         print_per_layer_stat=True, verbose=True)
print('{:<30}  {:<8}'.format('Computational complexity: ', macs))
print('{:<30}  {:<8}'.format('Number of parameters: ', params))

The output of both ori_model nad pthfile is the same, as follows.
Warning: module Dropout2d is treated as a zero-op.
Warning: module Flatten is treated as a zero-op.
Warning: module net is treated as a zero-op.
net(
  0.05 M, 100.000% Params, 0.001 GMac, 100.000% MACs,
  (conv1): Conv1d(0.007 M, 13.143% Params, 0.0 GMac, 45.733% MACs, 1, 128, kernel_size=(50,), stride=(3,))
  (conv2): Conv1d(0.029 M, 57.791% Params, 0.001 GMac, 50.980% MACs, 128, 32, kernel_size=(7,), stride=(1,))
  (conv3): Conv1d(0.009 M, 18.619% Params, 0.0 GMac, 0.913% MACs, 32, 32, kernel_size=(9,), stride=(1,))
  (fc1): Linear(0.004 M, 8.504% Params, 0.0 GMac, 0.404% MACs, in_features=32, out_features=128, bias=True)
  (fc2): Linear(0.001 M, 1.299% Params, 0.0 GMac, 0.063% MACs, in_features=128, out_features=5, bias=True)
  (bn1): BatchNorm1d(0.0 M, 0.515% Params, 0.0 GMac, 1.793% MACs, 128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (bn2): BatchNorm1d(0.0 M, 0.129% Params, 0.0 GMac, 0.114% MACs, 32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (dropout): Dropout2d(0.0 M, 0.000% Params, 0.0 GMac, 0.000% MACs, p=0.5, inplace=False)
  (faltten): Flatten(0.0 M, 0.000% Params, 0.0 GMac, 0.000% MACs, )
)
Computational complexity:       1013472.0
Number of parameters:           49669

Output:
One thing you could do is to exclude the weights below a certain threshold from the FLOPs computation. To do so you would have to modify the flop counter functions.
I'll provide examples for the modification for fc and conv layers below.
def linear_flops_counter_hook(module, input, output):
    input = input[0]
    output_last_dim = output.shape[-1]  # pytorch checks dimensions, so here we don't care much
    # MODIFICATION HAPPENS HERE
    num_zero_weights = (module.weight.data.abs() < 1e-9).sum()
    zero_weights_factor = 1 - torch.true_divide(num_zero_weights, module.weight.data.numel())
    module.__flops__ += int(np.prod(input.shape) * output_last_dim) * zero_weights_factor.numpy()
    # MODIFICATION HAPPENS HERE

def conv_flops_counter_hook(conv_module, input, output):
    # Can have multiple inputs, getting the first one
    input = input[0]

    batch_size = input.shape[0]
    output_dims = list(output.shape[2:])

    kernel_dims = list(conv_module.kernel_size)
    in_channels = conv_module.in_channels
    out_channels = conv_module.out_channels
    groups = conv_module.groups

    filters_per_channel = out_channels // groups
    conv_per_position_flops = int(np.prod(kernel_dims)) * in_channels * filters_per_channel

    active_elements_count = batch_size * int(np.prod(output_dims))

    # MODIFICATION HAPPENS HERE
    num_zero_weights = (conv_module.weight.data.abs() < 1e-9).sum()
    zero_weights_factor = 1 - torch.true_divide(num_zero_weights, conv_module.weight.data.numel())
    overall_conv_flops = conv_per_position_flops * active_elements_count * zero_weights_factor.numpy()
    # MODIFICATION HAPPENS HERE

    bias_flops = 0

    if conv_module.bias is not None:

        bias_flops = out_channels * active_elements_count

    overall_flops = overall_conv_flops + bias_flops

    conv_module.__flops__ += int(overall_flops)

Note that I'm using 1e-9 as a threshold for a weight counting as zero.


Instruction: Calculate covariance of torch tensor (2d feature map)

Input:
I have a torch tensor with shape (batch_size, number_maps, x_val, y_val). The tensor is normalized with a sigmoid function, so within range [0, 1]. I want to find the covariance for each map, so I want to have a tensor with shape (batch_size, number_maps, 2, 2). As far as I know, there is no torch.cov() function as in numpy. How can I efficiently calculate the covariance without converting it to numpy?
Edit:
def get_covariance(tensor):
bn, nk, w, h = tensor.shape
tensor_reshape = tensor.reshape(bn, nk, 2, -1)
x = tensor_reshape[:, :, 0, :]
y = tensor_reshape[:, :, 1, :]
mean_x = torch.mean(x, dim=2).unsqueeze(-1)
mean_y = torch.mean(y, dim=2).unsqueeze(-1)

xx = torch.sum((x - mean_x) * (x - mean_x), dim=2).unsqueeze(-1) / (h*w - 1)
xy = torch.sum((x - mean_x) * (y - mean_y), dim=2).unsqueeze(-1) / (h*w - 1)
yx = xy
yy = torch.sum((y - mean_y) * (y - mean_y), dim=2).unsqueeze(-1) / (h*w - 1)

cov = torch.cat((xx, xy, yx, yy), dim=2)
cov = cov.reshape(bn, nk, 2, 2)

return cov

I tried the following now, but I m pretty sure it's not correct.

Output:
You could try the function suggested on Github:
def cov(x, rowvar=False, bias=False, ddof=None, aweights=None):
    """Estimates covariance matrix like numpy.cov"""
    # ensure at least 2D
    if x.dim() == 1:
        x = x.view(-1, 1)

    # treat each column as a data point, each row as a variable
    if rowvar and x.shape[0] != 1:
        x = x.t()

    if ddof is None:
        if bias == 0:
            ddof = 1
        else:
            ddof = 0

    w = aweights
    if w is not None:
        if not torch.is_tensor(w):
            w = torch.tensor(w, dtype=torch.float)
        w_sum = torch.sum(w)
        avg = torch.sum(x * (w/w_sum)[:,None], 0)
    else:
        avg = torch.mean(x, 0)

    # Determine the normalization
    if w is None:
        fact = x.shape[0] - ddof
    elif ddof == 0:
        fact = w_sum
    elif aweights is None:
        fact = w_sum - ddof
    else:
        fact = w_sum - ddof * torch.sum(w * w) / w_sum

    xm = x.sub(avg.expand_as(x))

    if w is None:
        X_T = xm.t()
    else:
        X_T = torch.mm(torch.diag(w), xm).t()

    c = torch.mm(X_T, xm)
    c = c / fact

    return c.squeeze()

https://github.com/pytorch/pytorch/issues/19037


Instruction: Why multiplication on GPU is slower than on CPU?

Input:
Here is my code (simulate the feed-forward neural network):
import torch
import time

print(torch.cuda.is_available())    # True
device = torch.device('cuda:0' )

a = torch.tensor([1,2,3,4,5,6]).float().reshape(-1,1)
w1 = torch.rand(120,6)
w2 = torch.rand(1,120)
b1 = torch.rand(120,1)
b2 = torch.rand(1,1).reshape(1,1)

start = time.time()
for _ in range(100000):
    ans = torch.mm(w2, torch.mm(w1,a)+b1)+b2
end = time.time()
print(end-start)                    # 1.2725720405578613 seconds

a = a.to(device)
w1 = w1.to(device)
w2 = w2.to(device)
b1 = b1.to(device)
b2 = b2.to(device)

start = time.time()
for _ in range(100000):
    ans = torch.mm(w2, torch.mm(w1,a)+b1)+b2
end = time.time()
print(end-start)                    # 5.6569812297821045 seconds

I wonder if I did it the wrong way or what, and how can I change my code to show that GPU IS faster then CPU on matrix multiplication？

Output:
The reason can be a lot of things:

Your model is simple.
For GPU calculation there is the cost of memory transfer to and from the GPU's memory
You calculation is on a small data batch, probably with bigger data sample you should see better performance on GPU than CPU
We should not forget the caching, you calculate the same operations over and over again, maybe would be better to generate random a tensors for every run

Here is a thread on the pytorch forum: https://discuss.pytorch.org/t/cpu-faster-than-gpu/25343
Also you should use better profiler, like explaind in this thread: https://discuss.pytorch.org/t/how-to-measure-time-in-pytorch/26964


Instruction: What do you use to access CSV data on S3 and other object storage providers as a PyTorch Dataset?

Input:
My dataset is stored as a collection of CSV files in an Amazon Web Services (AWS) Simple Storage Service (S3) bucket. I'd like to train a PyTorch model based on this data but the built-in Dataset classes do not provide native support for object storage services like S3 or Google Cloud Storage (GCS), Azure Blob storage, and such. I checked the PyTorch documentation here https://pytorch.org/docs/stable/data.html# about the available Dataset classes and it comes up short when it comes to public cloud object storage support.
It looks like I have to create my own custom Dataset according to the following instructions: https://pytorch.org/tutorials/beginner/data_loading_tutorial.html#dataset-class but the effort seems overwhelming: I need to figure out how to download data from the object storage to local node, parse the CSV files to read them into PyTorch tensors, and then deal with the possibility of running out of disk space since my dataset is 100s of GBs.
Since PyTorch models are trained using gradient descent and I only need to store just a small batch of data (less than 1GB) in memory at once, is there a custom dataset implementation that can help?

Output:
Check out ObjectStorage Dataset which has support for object storage services like S3 and GCS osds.readthedocs.io/en/latest/gcs.html
You can run
pip install osds

to install it and then point it at your S3 bucket to instantiate the PyTorch Dataset and DataLoader using something like
from osds.utils import ObjectStorageDataset
from torch.utils.data import DataLoader

ds = ObjectStorageDataset(f"gcs://gs://cloud-training-demos/taxifare/large/taxi-train*.csv",
 storage_options = {'anon' : False },
 batch_size = 32768,
 worker = 4,
 eager_load_batches = False)

dl = DataLoader(ds, batch_size=None)

where you use your S3 location path instead of gcs://gs://cloud-training-demos/taxifare/large/taxi-train*.csv. So your glob for S3 would be something like s3://<bucket name>/<object path>/*.csv depending on the bucket and the bucket directory where you store your CSV objects for the dataset.


Instruction: I have an error on Pytorch and in particular with nllloss

Input:
I want to appply the criterion, where
criterion = nn.NLLLoss()
I apply it on output and labels
loss = criterion(output.view(-1,1), labels.long())

where:
*the shape of the labels
labels
tensor([ 1,  4,  1,  1,  4,  1,  2,  3,  2,  4,  2,  3,  3,  4,
         0,  4])
output
tensor([ 0.1829,  0.1959,  0.1909,  0.1895,  0.1914,  0.1883,  0.1895,
         0.1884,  0.1865,  0.1931,  0.1883,  0.1917,  0.1942,  0.1937,
         0.1897,  0.1934])

the shape of the output
torch.Size([16])
On the following line:
loss = criterion(output.view(-1,1), labels.long())
I get this error:
The error is:

RuntimeError: Assertion `cur_target >= 0 && cur_target < n_classes' failed.  at /opt/conda/conda-bld/pytorch_1524584710464/work/aten/src/THNN/generic/ClassNLLCriterion.c:97

Any ideas?

Output: Your label and output shapes must be [batch_size] and [batch_size, n_classes] respectively.


Instruction: How to index intermediate dimension with an index tensor in pytorch?

Input:
How can I index a tensor t with n dimensions with an index tensor of m < n dimensions, such that the last dimensions of t are preserved? The index tensor is shaped equal to tensor t for all dimensions before dimension m. Or in other terms, I want to index intermediate dimensions of a tensor, while keeping all the following dimensions of the selected indices preserved.
For example, lets say we have the two tensors:
t = torch.randn([3, 5, 2]) * 10
index = torch.tensor([[1, 3],[0,4],[3,2]]).long()

with t:
tensor([[[ 15.2165,  -7.9702],
         [  0.6646,   5.2844],
         [-22.0657,  -5.9876],
         [ -9.7319,  11.7384],
         [  4.3985,  -6.7058]],

        [[-15.6854, -11.9362],
         [ 11.3054,   3.3068],
         [ -4.7756,  -7.4524],
         [  5.0977, -17.3831],
         [  3.9152, -11.5047]],

        [[ -5.4265, -22.6456],
         [  1.6639,  10.1483],
         [ 13.2129,   3.7850],
         [  3.8543,  -4.3496],
         [ -8.7577, -12.9722]]])

Then the output I would like to have would have shape (3, 2, 2) and be:
tensor([[[  0.6646,   5.2844],
         [ -9.7319,  11.7384]],
        [[-15.6854, -11.9362],
         [  3.9152, -11.5047]],
        [[  3.8543,  -4.3496],
         [ 13.2129,   3.7850]]])

Another example would be that I have a tensor t of shape (40, 10, 6, 2) and an index tensor of shape (40, 10, 3). This should query dimension 3 of tensor t and the expected output shape would be (40, 10, 3, 2).
How can I achieve this in a generic way, without using loops?

Output:
In this case, you can do something like this:
t[torch.arange(t.shape[0]).unsqueeze(1), index, ...]

Full code:
import torch

t = torch.tensor([[[ 15.2165,  -7.9702],
                   [  0.6646,   5.2844],
                   [-22.0657,  -5.9876],
                   [ -9.7319,  11.7384],
                   [  4.3985,  -6.7058]],
                  [[-15.6854, -11.9362],
                   [ 11.3054,   3.3068],
                   [ -4.7756,  -7.4524],
                   [  5.0977, -17.3831],
                   [  3.9152, -11.5047]],
                  [[ -5.4265, -22.6456],
                   [  1.6639,  10.1483],
                   [ 13.2129,   3.7850],
                   [  3.8543,  -4.3496],
                   [ -8.7577, -12.9722]]])

index = torch.tensor([[1, 3],[0,4],[3,2]]).long()

output = t[torch.arange(t.shape[0]).unsqueeze(1), index, ...]

# tensor([[[  0.6646,   5.2844],
#          [ -9.7319,  11.7384]],
#
#         [[-15.6854, -11.9362],
#          [  3.9152, -11.5047]],
#
#         [[  3.8543,  -4.3496],
#          [ 13.2129,   3.7850]]])


Instruction: How to make Intel GPU available for processing through pytorch?

Input:
I'm using a laptop which has Intel Corporation HD Graphics 520.
Does anyone know how to it set up for Deep Learning, specifically Pytorch? I have seen if you have Nvidia graphics I can install cuda but what to do when you have intel GPU?

Output:
PyTorch doesn't support anything other than NVIDIA CUDA and lately AMD Rocm.
Intels support for Pytorch that were given in the other answers is exclusive to xeon line of processors and its not that scalable either with regards to GPUs.
Intel's oneAPI formerly known ad oneDNN however, has support for a wide range of hardwares including intel's integrated graphics but at the moment, the full support is not yet implemented in PyTorch as of 10/29/2020 or PyTorch 1.7.
But you still have other options. for inference you have couple of options.
DirectML is one of them. basically you convert your model into onnx, and then use directml provider to run your model on gpu (which in our case will use DirectX12 and works only on Windows for now!)
Your other Option is to use OpenVino and TVM both of which support multi platforms including Linux, Windows, Mac, etc.
All of them use ONNX models so you need to first convert your model to onnx format and then use them.


Instruction: Accessing functions in the class modules of nn.Sequential

Input: When running nn.Sequential, I include a list of class modules (which would be layers of a neural network). When running nn.Sequential, it calls forward functions of the modules. However each of the class modules also has a function which I would like to access when the nn.Sequential runs. How can I access and run this function when running nn.Sequential?

Output:
You can use a hook for that. Let's consider the following example demonstrated on VGG16:
This is the network architecture:

Say we want to monitor the input and output for layer (2)  in the features Sequential (that Conv2d layer you see above).
For this matter we register a forward hook, named my_hook which will be called on any forward pass:
import torch
from torchvision.models import vgg16

def my_hook(self, input, output):
    print('my_hook\'s output')
    print('input: ', input)
    print('output: ', output)

# Sample net:
net = vgg16()

#Register forward hook:
net.features[2].register_forward_hook(my_hook)

# Test:
img = torch.randn(1,3,512,512)
out = net(img) # Will trigger my_hook and the data you are looking for will be printed