Instruction: About cosine similarity, how to choose the loss function and the network(I have two plans)

Input:
Sorry I have no clue, I don't know where to find a solution.
I'm using two networks to construct two embeddings，I have binary target to indicate whether embeddingA and embeddingB "match" or not(1 or -1).
The dataset like this:
embA0 embB0 1.0
embA1 embB1 -1.0
embA2 embB2 1.0
...

I hope to use cosine similarity to get classification results.
But I feel confused when choosing the loss function, the two networks that generate embeddings are trained separately, now I can think of two options as follows:
Plan 1:
Construct the 3rd network, use embeddingA and embeddingB as the input of nn.cosinesimilarity() to calculate the final result (should be probability in [-1,1] ), and then select a two-category loss function.
(Sorry, I dont know which loss function to choose.)
class cos_Similarity(nn.Module):
    def __init__(self):
        super(cos_Similarity,self).__init__()
        cos=nn.CosineSimilarity(dim=2)
        embA=generator_A()
        embB=generator_B()

    def forward(self,a,b):
        output_a=embA(a)
        output_b=embB(b)
        return cos(output_a,output_b)
loss_func=nn.CrossEntropyLoss()

y=cos_Similarity(a,b)
loss=loss_func(y,target)
acc=np.int64(y>0)

Plan 2:
The two Embeddings as the output, then use nn.CosineEmbeddingLoss() as loss function, when I calculate the accuracy, I use nn.Cosinesimilarity() to output the result(probability in [-1,1]).
output_a=embA(a)
output_b=embB(b)

cos=nn.CosineSimilarity(dim=2)
loss_function = torch.nn.CosineEmbeddingLoss()

loss=loss_function(output_a,output_b,target)
acc=cos(output_a,output_b)

I really need help. How do I make a choice? Why? Or I can only make a choice for me through experimental results.
Thank you very much!
###############################addition

def train_func(train_loss_list):

    train_data=load_data('train')
    trainloader = DataLoader(train_data, batch_size=BATCH_SIZE)

    cos_smi=nn.CosineSimilarity(dim=2)
    train_loss = 0

    for step,(a,b,target) in enumerate(trainloader):

        try:
            optimizer.zero_grad()

            output_a = model_A(a) #generate embA
            output_b = model_B(b) #generate embB

            acc=cos_smi(output_a,output_b)

            loss = loss_fn(output_a,output_b, target.unsqueeze(dim=1))

            train_loss += loss.item()

            loss.backward()

            optimizer.step()

            train_loss_list.append(loss)

            if step%10==0:
                print('train:',step,'step','loss:',loss,'acc',acc)

        except Exception as e:
            print('train:',step,'step')
            print(repr(e))

    return train_loss_list,train_loss/len(trainloader)

Output:
In response to the comment thread.
The objective or pipeline seems to be:

Receive two embedding vectors (say, A and B).
Check whether these two vectors are "similar" or not (using cosine similarity).
Label is 1 if they're similar, and -1 otherwise (I recommend changing this to 0 or 1 rather than -1 and 1).

What I can think of is the following. Correct me if I misunderstood something. A disclaimer is that I'm pretty much coding this off of my intuition without knowing any details, so it's probably going to be riddled with errors if you try to run in. Let's still try and get a high-level understanding.
Model
import torch
import torch.nn as nn

class Model(nn.Module):
    def __init__(self, num_emb, emb_dim): # I'm assuming the embedding matrices are same sizes.
        self.embedding1 = nn.Embedding(num_embeddings=num_emb, embedding_dim=emb_dim)
        self.embedding2 = nn.Embedding(num_embeddings=num_emb, embedding_dim=emb_dim)
        self.cosine = nn.CosineSimilarity()
        self.sigmoid = nn.Sigmoid()

    def forward(self, a, b):
        output1 = self.embedding1(a)
        output2 = self.embedding2(b)
        similarity = self.cosine(output1, output2)
        output = self.sigmoid(similarity)

        return output

Training/Evaluation
model = Model(num_emb, emb_dim)

if torch.cuda.is_available():
    model = model.to('cuda')

model.train()

criterion = loss_function()
optimizer = some_optimizer()

for epoch in range(num_epochs):
    epoch_loss = 0
    for batch in train_loader:
        optimizer.zero_grad()

        a, b, label = batch

        if torch.cuda.is_available():
            a = a.to('cuda')
            b = b.to('cuda')
            label = label.to('cuda')

        output = model(a, b)

        loss = criterion(output, label)
        loss.backward()
        optimizer.step()

        epoch_loss += loss.cpu().item()

        print("Epoch %d \t Loss %.6f" % epoch, epoch_loss)

I omitted some detailed (e.g., hyperparameter values, loss function and optimizer, etc.). Is this overall procedure something similar to what you're looking for OP?


Instruction: Partial derivatives of Gaussian Process wrt features

Input: Given a Gaussian Process Model with multidimensional features and scalar observations, how do I compute derivatives of the output wrt to each input, in GPyTorch or GPFlow (or scikit-learn)?

Output:
If I understand your question correctly, the following should give you what you want in GPflow with TensorFlow:
import numpy as np
import tensorflow as tf
import gpflow

### Set up toy data & model -- change as appropriate:
X = np.linspace(0, 10, 5)[:, None]
Y = np.random.randn(5, 1)
data = (X, Y)
kernel = gpflow.kernels.SquaredExponential()
model = gpflow.models.GPR(data, kernel)
Xtest = np.linspace(-1, 11, 7)[:, None]  # where you want to predict

### Compute gradient of prediction with respect to input:
# TensorFlow can only compute gradients with respect to tensor objects,
# so let's convert the inputs to a tensor:
Xtest_tensor = tf.convert_to_tensor(Xtest)

with tf.GradientTape(
        persistent=True  # this allows us to compute different gradients below
) as tape:
    # By default, only Variables are watched. For gradients with respect to tensors,
    # we need to explicitly watch them:
    tape.watch(Xtest_tensor)

    mean, var = model.predict_f(Xtest_tensor)  # or any other predict function

grad_mean = tape.gradient(mean, Xtest_tensor)
grad_var = tape.gradient(var, Xtest_tensor)


Instruction: How can I get argmaxed torch tensor excluding certain index?

Input:
I wonder if I can get torch.argmax of my input excluding certain index.
For example,
target = torch.tensor([1,2])
input = torch.tensor([[0.1,0.5,0.2,0.2], [0.1,0.5,0.1,0.3]])

I want to get the maximum value in input excluding the index on the target, so that the result would be
output = torch.tensor([[0.2],[0.5]])

Output:
You can try this

Set negative infy to the target indices in temp tensor
Then use torch.max or torch.argmax

tmp_input = input.clone()
tmp_input[range(len(input)), target] = float("-Inf")

torch.max(tmp_input, dim=1).values
tensor([0.2000, 0.5000])

torch.max(tmp_input, dim=1).indices
tensor([3, 1])

torch.argmax(tmp_input, dim=1)
tensor([3, 1])


Instruction: TypeError: If no scoring is specified, the estimator passed should have a 'score' method

Input:
I have been working with a PyTorch neural network for a while now. I decided I wanted to add a permutation feature importance scorer, and this started to cause some issues.
I get" TypeError: If no scoring is specified, the estimator passed should have a 'score' method. The estimator <class 'skorch.net.NeuralNet'>[uninitialized](
module=<class 'main.run..MultiLayerPredictor'>,
) does not. " - error message. Here's my code:
class MultiLayerPredictor(torch.nn.Module):
    def __init__(self, input_shape=9152, output_shape=1, hidden_dim=1024, **kwargs):
        super().__init__()
        self.fc1 = torch.nn.Linear(in_features=input_shape, out_features=hidden_dim)
        self.fc2 = torch.nn.Linear(in_features=hidden_dim, out_features=hidden_dim)
        self.fc3 = torch.nn.Linear(in_features=hidden_dim, out_features=output_shape)

    def forward(self, x):
        l1 = torch.relu(self.fc1(x))
        l2 = torch.relu(self.fc2(l1))
        return torch.sigmoid(self.fc3(l2)).reshape(-1)

print("Moving to wrapping the neural net")
net = NeuralNet(
    MultiLayerPredictor,
    criterion=nn.MSELoss,
    max_epochs=10,
    optimizer=optim.Adam,
    lr=0.1,
    iterator_train__shuffle=True
)

print("Moving to finding optimal hyperparameters")

lr = (10**np.random.uniform(-5,-2.5,1000)).tolist()
params = {
    'optimizer__lr': lr,
    'max_epochs':[300,400,500],
    'module__num_units': [14,20,28,36,42],
    'module__drop' : [0,.1,.2,.3,.4]
}

gs = RandomizedSearchCV(net,params,refit=True,cv=3,scoring='neg_mean_squared_error',n_iter=100)
gs.fit(X_train_scaled,y_train);

def report(results, n_top=3):
    for i in range(1, n_top + 1):
        candidates = np.flatnonzero(results['rank_test_score'] == i)
    for candidate in candidates:
        print("Model with rank: {0}".format(i))
        print("Mean validation score: {0:.3f} (std: {1:.3f})".format(
              results['mean_test_score'][candidate],
              results['std_test_score'][candidate]))
        print("Parameters: {0}".format(results['params'][candidate]))
        print("")

print(report(gs.cv_results_,10))

epochs = [i for i in range(len(gs.best_estimator_.history))]
train_loss = gs.best_estimator_.history[:,'train_loss']
valid_loss = gs.best_estimator_.history[:,'valid_loss']

plt.plot(epochs,train_loss,'g-');
plt.plot(epochs,valid_loss,'r-');
plt.title('Training Loss Curves');
plt.xlabel('Epochs');
plt.ylabel('Mean Squared Error');
plt.legend(['Train','Validation']);
plt.show()

r = permutation_importance(net, X_test, y_test, n_repeats=30,random_state=0)

for i in r.importances_mean.argsort()[::-1]:
    if r.importances_mean[i] - 2 * r.importances_std[i] > 0:
        print(f"{metabolites.feature_names[i]:<8}"
              f"{r.importances_mean[i]:.3f}"
              f" +/- {r.importances_std[i]:.3f}")

y_pred_acc = gs.predict(X_test)
print('Accuracy : ' + str(accuracy_score(y_test,y_pred_acc)))

Stacktrace would point that the error stems from the line where I set the permutation importance. How can I fix this?
Full stacktrace:
*Traceback (most recent call last):
  File "//ad..fi/home/h//Desktop/neuralnet/neuralnet_wrapped.py", line 141, in <module>
    run()
  File "//ad..fi/home/h//Desktop/neuralnet/neuralnet_wrapped.py", line 119, in run
    r = permutation_importance(net, X_test, y_test,
  File "C:\Users\\AppData\Roaming\Python\Python38\site-packages\sklearn\utils\validation.py", line 73, in inner_f
    return f(**kwargs)
  File "C:\Users\\AppData\Roaming\Python\Python38\site-packages\sklearn\inspection\_permutation_importance.py", line 132, in permutation_importance
    scorer = check_scoring(estimator, scoring=scoring)
  File "C:\Users\\AppData\Roaming\Python\Python38\site-packages\sklearn\utils\validation.py", line 73, in inner_f
    return f(**kwargs)
  File "C:\Users\\AppData\Roaming\Python\Python38\site-packages\sklearn\metrics\_scorer.py", line 425, in check_scoring
    raise TypeError(
TypeError: If no scoring is specified, the estimator passed should have a 'score' method. The estimator <class 'skorch.net.NeuralNet'>[uninitialized](
  module=<class '__main__.run.<locals>.MultiLayerPredictor'>,
) does not.*

Output:
From the docs:

NeuralNet still has no score method. If you need it, you have to implement it yourself.

This is the problem. The NeuralNet has no score method, as the error says. And the documentation says that "you have to implement it yourself". You can check that looking at the source-code too.


Instruction: Questions about Batch Normalization in Pytorch

Input:
Recently when I use the BN in the PyTorch, I have several questions.

Based on the BN2d documentation in PyTorch, when inferencing(evaluation), it will automatically use the mean and variance (running estimate when training) for BN layer. However, my first question is that when we save out the model after training, does it contain the running mean and variance? I originally thought the model will only save the learnable parameters. But the running mean and variance are not really learnable?

By default, when we use eval() in PyTorch, the BN layer will use some running mean and variance from a specific variable (maybe the results from training), right? But not to calculate the mean and variance of mini-batch? (Because I see some answers mention that the bad performance of BN when inferencing is cased by the 1-size batch. But it should use the running mean and variance of training, why the batch size of testing will influence?
The third question is kind of related to the second question. Will the BN performs the same when inferencing. For example, when I iterate the test set twice. Will the results behave differently? (The maybe more direct question is that will the mean and variance change when inferencing?)
I also want to know if it is possible to use mini-batch mean & variance or running mean & variance for inferencing. If I only set BN layer to train, its learnable parameter will also update. But maybe I just want them to get new mean and variance. Is there a possible way to do that?
The last question is related to the fourth question. So is it fair to use mean & variance of all test data, or batch of test data to calculate the mean & variance? I mean fair here is that is it improper (tricky?) to use some attributes of the test set?

Look forward to your answers. I'm a kind of new bird, willing to learn and discuss with you!!
Thanks in advance!!!

Output:
I get the answers from my senior classmates, and I think it's useful for others. (If you have different points, feel free to comment)

When we save out the whole model, it will contain the running mean and variance for the BN layers. These two parameters are not learnable (not updated in backward process, but updated in the forward process)

If use .eval(), the BN layer will automatically use the running mean and variance stored in that layer, and will not update in forward process again. This means that when inferencing, BN layers use the running mean and variance calculated in the training process.

Yes, if just simply use .eval(), the BN layer will use the same mean and variance at all times.

It's possible to calculate the running mean and variance based on the test set. Just make the BN layer in train() mode. This will not influence the learnable parameters of BN layer. Because when inferencing, we only have a forward process (update the mean and variance) without a backward process. Maybe if we have to reset the mean and variance at the beginning of the evaluation or inference period to make mean and variance totally unrelated to the training process.

It's kind of some tricks. I hear that some GAN papers adopt such strategies for BN layers.


Instruction: What is the correct way to implement gradient accumulation in pytorch?

Input:
Broadly there are two ways:

Call loss.backward() on every batch, but only call optimizer.step() and optimizer.zero_grad() every N batches. Is it the case that the gradients of the N batches are summed up? Hence to maintain the same learning rate per effective batch, we have to divide the learning rate by N?

Accumulate loss instead of gradient, and call (loss / N).backward() every N batches. This is easy to understand, but does it defeat the purpose of saving memory (because the gradients of the N batches are computed at once)? The learning rate doesn't need adjusting to maintain the same learning rate per effective batch, but should be multiplied by N if you want to maintain the same learning rate per example.

Which one is better, or more commonly used in packages such as pytorch-lightning? It seems that optimizer.zero_grad() is a prefect fit for gradient accumulation, therefore (1) should be recommended.

Output: You can use PytorchLightning and you get this feature of the box, see the Trainer argument accumulate_grad_batches which you can also pair with gradient_clip_val, more in docs.


Instruction: What is the machine precision in pytorch and when should one use doubles?

Input:
I am running experiments on synthetic data (e.g. fitting a sine curve) and I get errors in pytorch that are really small. One if about 2.00e-7. I was reading about machine precision and it seems really close to the machine precision. How do I know if this is going to cause problems (or if perhaps it already has e.g. I can't differentiate between the different errors since they are "machine zero").
errors:
p = np.array([2.3078539778125768e-07,
               1.9997889411762922e-07,
               2.729681222011256e-07,
               3.2532371115080884e-07])

m = np.array([3.309504692539563e-07,
                 4.1058904888091606e-06,
                 6.8326703386053605e-06,
                 7.4616147721799645e-06])

what confuses me is that I tried adding what I thought was to small of a number so that it returned no difference but it did return a difference (i.e. I tried to do a+eps = a using eps = smaller than machine precision):
import torch

x1 = torch.tensor(1e-6)
x2 = torch.tensor(1e-7)
x3 = torch.tensor(1e-8)
x4 = torch.tensor(1e-9)

eps = torch.tensor(1e-11)

print(x1.dtype)
print(x1)
print(x1+eps)

print(x2)
print(x2+eps)

print(x3)
print(x3+eps)

print(x4)
print(x4+eps)

output:
torch.float32
tensor(1.0000e-06)
tensor(1.0000e-06)
tensor(1.0000e-07)
tensor(1.0001e-07)
tensor(1.0000e-08)
tensor(1.0010e-08)
tensor(1.0000e-09)
tensor(1.0100e-09)

I expected everything to be zero but it wasn't. Can someone explain to me what is going on? If I am getting losses close to 1e-7 should I use double rather than float? googling it seems that single is the precision for float afaik.
If I want to use doubles what are cons/pros + what is the least error prone way to change my code? Is a single change to double type enough or is there a global flag?

Useful reminder:
recall machine precision:
Machine precision is the smallest number ε such that the difference between 1 and 1 + ε is nonzero, i.e., it is the smallest difference between these two numbers that the computer recognizes. For IEEE-754 single precision this is 2-23 (approximately 10-7) while for IEEE-754 double precision it is 2-52 (approximately 10-16) .

Potential solution:
Ok let’s see if this is a good summary of what I think is correct (modulo ignoring some details that I don’t fully understand right now of floats, like the bias).
But I’ve concluded that the best thing for me is to make sure my errors/numbers have two properties:
they are within 7decimals of each other (due to the mantissa being 24 bigs like you pointed out the log_10(2^24) = 7.225)
they are far enough from the edges. For this I take the mantissa to be 23 bits away from the lower edge (point position about -128+23) and the same for the largest edge but 127-23.
As long we satisfy that more or less we avoid adding two numbers that are too small for the machine to distinguish (condition 1) and avoid overflows/underflows (condition 2).
Perhaps there is a small detail I might be missing with the bias or some other float detail (like representing infinity, NaN). But I believe that is correct.
If anyone can correct the details, that would be fantastic.

useful links:

https://www.cfd-online.com/Wiki/Machine_precision
https://discuss.pytorch.org/t/what-is-the-machine-precision-of-pytorch-with-cpus-or-gpus/9384/3
Should I use double or float?
Double precision floating values in Python?
Difference between Python float and numpy float32

Output:
I think you misunderstood how floating points work. There are many good resources (e.g.) about what floating points are, so I am not going into details here.
The key is that floating points are dynamic. They can represent the addition of very large values up to a certain accuracy, or the addition of very small values up to a certain accuracy, but not the addition of a very large value with a very small value. They adjust their ranges on-the-go.
So this is why your testing result is different than the explanation in "machine precision" -- you are adding two very small values, but that paragraph explicitly said "1+eps". 1 is a much larger value than 1e-6. The following thus will work as expected:
import torch

x1 = torch.tensor(1).float()
eps = torch.tensor(1e-11)

print(x1.dtype)
print(x1)
print(x1+eps)

Output:
torch.float32
tensor(1.)
tensor(1.)

The second question -- when should you use double?
Pros - higher accuracy.
Cons - Much slower (hardware are configured to like float most of the time), doubled memory usage.
That really depends on your application. Most of the time I would just say no. As I said, you need double when you have very large values and very small values coexist in the network. That should not be happening anyway with proper normalization of data.
(Another reason is the overflow of exponent, say when you need to represent very very very large/small values, beyond 1e-38 and 1e38)


Instruction: pytorch dataloader default_collate argument use with to(device)

Input:
Ive been trying to integrate the to(device) inside my dataloader using to(device) as seen in https://github.com/pytorch/pytorch/issues/11372
I defined it on FashionMNIST in the following way:
device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
batch_size = 32
trainset = datasets.FashionMNIST('~/.pytorch/F_MNIST_data/',
                                 download=True,
                                 train=True,
                                 transform=transforms.ToTensor())
rain_loader = torch.utils.data.DataLoader(trainset, batch_size=batch_size, shuffle=False, collate_fn=lambda x: default_collate(x).to(device))

But i get the following error:
AttributeError: 'list' object has no attribute 'to'
It seems that the output of default collate is a list of length 2 with the first element being the image tensor and the second the labels tensor (since its the output of next(iter(train_loader)) with collate_fn=None), so I tried with the following defined function:
def to_device_list(l, device):
    return [l[0].to(device), l[1].to(device)]
train_loader = torch.utils.data.DataLoader(trainset, batch_size=batch_size, shuffle=False, collate_fn=lambda x: to_device_list(x, device))

And I got the following error:
AttributeError: 'tuple' object has no attribute 'to'
Any help please on how to do it?

Output:
The fashion mnist dataset returns a tuple of img and target, where the img is tensor and target is int value for class.
Now, your dataloader takes batch size samples from dataset class to get list of samples. Note, this list of samples is now, List[Tuple[Tensor, int]](using typing annotation here). Then it calls, collate function to convert List[Tuple[Tensor, int]] into List[Tensor], where this list has 2 tensors. The first tensor is stacked array of images of size [32, 1, 28, 28], where 32 was batch size and second tensor is tensor array of int values(class labels).
The default_collate function, just converts array of structures to structures of array.
Now, when you use collate_fn=lambda x: default_collate(x).to(device), notice that default_collate returns you a list of tensors. So calling .to on list wont work and should be called on all elements of the list.
Solution
Use
collate_fn=lambda x: list(map(lambda x: x.to(device), default_collate(x))))

The map function transfers each element of list(from default_collate) to cuda, and finally, call list, since map is evaluated lazy in python3.


Instruction: HParams in Tensorboard, Run IDs and naming

Input:
I'm using SummaryWriter.add_hparams(params, values) to log hyperparameters during training of my Seq2Seq model. My runs are named with a timestamp like 2020-09-10 14-50-27. In the HParams tab in Tensorboard, everything looks fine, but the HParam Trial IDs are different; they have another string of numbers attached like this: 2020-09-10 14-50-27/1599742915.9712806. These also appear in the Scalar tab as different runs, which is quite inconvenient. Is there a way to turn of this extra naming or to stop them of appearing in the Scalars tab? I use pytorch and its summarywriter like this:
params = {
    'max_epochs' : max_epochs,
    'learning_rate': learning_rate,
    'batch_size': batch_size,
    'optimizer_name': optimizer_name,
    'dropout_fc': dropout_fc
}
values = {
    'hparam/hp_total_time': t1_stop - t0_start,
    'hparam/score' : best_score
}

tb.add_hparams(params, values)

Output:
As Aniket mentioned there is not enough in your issue description to be entirely sure what the issue is.
However, if you are using Pytorch, I suspect you may be referring to the behaviour also reported in this issue. The add_hparams method creates a new subfolder with current timestamp when called, which is 1599742915.9712806 in your case.
TensorBoard uses the hierarchical folder structure to organise (group) runs, which is why 2020-09-10 14-50-27/1599742915.9712806 and 2020-09-10 14-50-27 appear as different runs.
As per the issue I mentioned above, there does not seem to be an "official" way to modify this behaviour but if you read the comments you will find a few custom classes that have been proposed to help.


Instruction: Trying to access subset of mnist dataset in pytorch [equal samples from each class]

Input:
Trying to access subset of mnist dataset in pytorch [equal samples from each class] but getting this error
prng = RandomState(42)
random_permute = prng.permutation(np.arange(0, 6000))[0:3000]
indx = np.concatenate([np.where(np.array(mnist_data.targets) == classe)[0][random_permute] for classe in range(0,10)])

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-178-038015f76b77> in <module>
----> 1 indx = np.concatenate([np.where(np.array(mnist_data.targets) == classe)[0][random_permute] for classe in range(0,10)])

<ipython-input-178-038015f76b77> in <listcomp>(.0)
----> 1 indx = np.concatenate([np.where(np.array(mnist_data.targets) == classe)[0][random_permute] for classe in range(0,10)])

IndexError: index 5992 is out of bounds for axis 0 with size 5923

Output:
MNIST dataset does not have a uniform distribution of targets. You are getting this error because class 0 in MNIST contains 5923 samples.
nums = [0]*10
for i in range(60000):
  nums[(int(mnist_data.targets[i]))] += 1
print(nums)

This will print [5923, 6742, 5958, 6131, 5842, 5421, 5918, 6265, 5851, 5949].


Instruction: no CUDA-capable device is detected at /pytorch/aten/src/THC/THCGeneral.cpp:47 in Google Colab

Input:
I have been using their algorithm for days and I tried several, but none of them gave me this error until now.
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-5-dbd18151b569> in <module>()
      1 from demo import load_checkpoints
      2 generator, kp_detector = load_checkpoints(config_path='config/vox-256.yaml',
----> 3                             checkpoint_path='/content/gdrive/My Drive/first-order-motion-model/vox-cpk.pth.tar')

10 frames
/usr/local/lib/python3.6/dist-packages/torch/cuda/__init__.py in _lazy_init()
    188             raise AssertionError(
    189                 "libcudart functions unavailable. It looks like you have a broken build?")
--> 190         torch._C._cuda_init()
    191         # Some of the queued calls may reentrantly call _lazy_init();
    192         # we need to just return without initializing in that case.

RuntimeError: cuda runtime error (100) : no CUDA-capable device is detected at /pytorch/aten/src/THC/THCGeneral.cpp:47

Output: You have not enabled GPU on your notebook, enable it in Runtime > Change runtime.


Instruction: Saving Model State and Load in Google Colab

Input: I have 500 epochs in total to train . But it is taking 8 minutes per epoch to be completed in google colab. Can any one help me how can I save my Model state after a Particular number of epoch  completion and start the training again from where I left in google Colab ??

Output:
If you want to save the model to google drive after certain number of epochs in pytorch you can do so by using
first mount google drive
from google.colab import drive
drive.mount('/content/gdrive')

Then the run the cell in colab and authenticate. Now google drive should be mounted.
Now set the path to be
PATH = F"/content/gdrive/My Drive/{Model name}/{model_save_name}"
you can the save the model
    if(epoch%(number_epoch_to_save)==0):
        torch.save(model.state_dict(), PATH)

example documentation can be found here at https://pytorch.org/tutorials/beginner/saving_loading_models.html


Instruction: how initialize a torch of matrices

Input:
Hello I m trying to create a tensor that will have inside N matrices of n by n size. I tried to initialize it with
Q=torch.zeros(N, (n,n))

but i get the following error
zeros(): argument 'size' must be tuple of ints, but found element of type tuple at pos 2

Also I want to fill it later with random matrices with integer values and I will turn it to semidefinte so I thought of the following
for i in range(0,N):
   Q[i]=torch.randint(0,10,(n,n))
Q = Q*Q.t()

Is it correct? Is there any other faster way with a build in command?

Output:
N matrices of n x n size is equivalent to three dimensional tensor of shape [N, n, n]. You can do it like so:
import torch

N = 32
n = 10
tensor = torch.randint(0, 10, size=(N, n, n))

No need to fill it with zeros to begin with, you can create it directly.
You can also iterate over 0 dimension similar to what you did:
for i in range(0, N):
    tensor[i] = tensor[i] * tensor[i].T

See @Dishin H Goyani answer for faster approach with permutation.


Instruction: RuntimeError: Expected 3-dimensional input for 3-dimensional weight [64, 512, 1], but got 2-dimensional input of size [4, 512] instead

Input:
Hello below is the pytorch model I am trying to run. But getting error. I have posted the error trace as well. It was running very well unless I added convolution layers. I am still new to deep learning and Pytorch. So I apologize if this is silly question. I am using conv1d so why should conv1d expect 3 dimensional input and it is also getting a 2d input which is also odd.
class Net(nn.Module):
        def __init__(self):
            super().__init__()
            self.fc1 = nn.Linear(CROP_SIZE*CROP_SIZE*3, 512)
            self.conv1d1 = nn.Conv1d(in_channels=512, out_channels=64, kernel_size=1, stride=2)
            self.fc2 = nn.Linear(64, 128)
            self.conv1d2 = nn.Conv1d(in_channels=128, out_channels=64, kernel_size=1, stride=2)
            self.fc3 = nn.Linear(64, 256)
            self.conv1d3 = nn.Conv1d(in_channels=256, out_channels=64, kernel_size=1, stride=2)
            self.fc4 = nn.Linear(64, 256)
            self.fc4 = nn.Linear(256, 128)
            self.fc5 = nn.Linear(128, 64)
            self.fc6 = nn.Linear(64, 32)
            self.fc7 = nn.Linear(32, 64)

     self.fc8 = nn.Linear(64, frame['landmark_id'].nunique())

    def forward(self, x):
        x = F.relu(self.conv1d1(self.fc1(x)))
        x = F.relu(self.conv1d2(self.fc2(x)))
        x = F.relu(self.conv1d3(self.fc3(x)))
        x = F.relu(self.fc4(x))
        x = F.relu(self.fc5(x))
        x = F.relu(self.fc6(x))
        x = F.relu(self.fc7(x))
        x = self.fc8(x)
        return F.log_softmax(x, dim=1)

net = Net()

import torch.optim as optim

loss_function = nn.CrossEntropyLoss()
net.to(torch.device('cuda:0'))
for epoch in range(3): # 3 full passes over the data
    optimizer = optim.Adam(net.parameters(), lr=0.001)
    for data in tqdm(train_loader):  # `data` is a batch of data
        X = data['image'].to(device)  # X is the batch of features
        y = data['landmarks'].to(device) # y is the batch of targets.
        optimizer.zero_grad()  # sets gradients to 0 before loss calc. You will do this likely every step.
        output = net(X.view(-1,CROP_SIZE*CROP_SIZE*3))  # pass in the reshaped batch
#         print(np.argmax(output))
#         print(y)
        loss = F.nll_loss(output, y)  # calc and grab the loss value
        loss.backward()  # apply this loss backwards thru the network's parameters
        optimizer.step()  # attempt to optimize weights to account for loss/gradients

    print(loss)  # print loss. We hope loss (a measure of wrong-ness) declines!

Error trace
    ---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-42-f5ed7999ce57> in <module>
      5         y = data['landmarks'].to(device) # y is the batch of targets.
      6         optimizer.zero_grad()  # sets gradients to 0 before loss calc. You will do this likely every step.
----> 7         output = net(X.view(-1,CROP_SIZE*CROP_SIZE*3))  # pass in the reshaped batch
      8 #         print(np.argmax(output))
      9 #         print(y)

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

<ipython-input-37-6d3e34d425a0> in forward(self, x)
     16
     17     def forward(self, x):
---> 18         x = F.relu(self.conv1d1(self.fc1(x)))
     19         x = F.relu(self.conv1d2(self.fc2(x)))
     20         x = F.relu(self.conv1d3(self.fc3(x)))

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/conv.py in forward(self, input)
    210                             _single(0), self.dilation, self.groups)
    211         return F.conv1d(input, self.weight, self.bias, self.stride,
--> 212                         self.padding, self.dilation, self.groups)
    213
    214

RuntimeError: Expected 3-dimensional input for 3-dimensional weight [64, 512, 1], but got 2-dimensional input of size [4, 512] instead

Output:
You should learn how convolutions work (e.g. see this answer) and some neural network basics (this tutorial from PyTorch).
Basically, Conv1d expects inputs of shape [batch, channels, features] (where features can be some timesteps and can vary, see example).
nn.Linear expects shape [batch, features] as it is fully connected and each input feature is connected to each output feature.
You can verify those shapes by yourself, for torch.nn.Linear:
import torch

layer = torch.nn.Linear(20, 10)
data = torch.randn(64, 20)  # [batch, in_features]
layer(data).shape  # [64, 10], [batch, out_features]

For Conv1d:
layer = torch.nn.Conv1d(in_channels=20, out_channels=10, kernel_size=3, padding=1)
data = torch.randn(64, 20, 15)  # [batch, channels, timesteps]
layer(data).shape  # [64, 10, 15], [batch, out_features]

layer(torch.randn(32, 20, 25)).shape  # [32, 10, 25]

BTW. As you are working with images, you should use torch.nn.Conv2d instead.


Instruction: How to train deeplabv3 on custom dataset on pytorch?

Input:
I am trying to do image segmentation and I got to know the Google work of DeepLabv3.
This is the reference to the paper:
https://arxiv.org/abs/1706.05587
Chen, L.C., Papandreou, G., Schroff, F. and Adam, H., 2017. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587.
This architecture is trained to do segmentation of the 20+1 classes of the Pascal VOC 2012 Dataset (20 foreground and 1 background class).
Pytorch provides pre-trained deeplabv3 on Pascal dataset, I would like to train the same architecture on cityscapes. Therefore, there are different classes with respect to the Pascal VOC dataset. I would like to know what is the efficient way to do it?
For now this is the only code I wrote:
import torch
model = torch.hub.load('pytorch/vision:v0.6.0', 'deeplabv3_resnet101', pretrained=True)
model.eval()

Output:
Write custom Dataloader  class which should inherit Dataset class and implement at least 2 methods   __len__ and __getitem__.
Modify the pretrained DeeplabV3 head with your custom number of output channels.

from torchvision.models.segmentation.deeplabv3 import DeepLabHead
from torchvision.models.segmentation import deeplabv3_resnet101

def custom_DeepLabv3(out_channel):
  model = deeplabv3_resnet101(pretrained=True, progress=True)
  model.classifier = DeepLabHead(2048, out_channel)

  #Set the model in training mode
  model.train()
  return model

Train and evaluate the model.


Instruction: Pytorch: How to concatenate lists within a tensor?

Input:
I have a tensor of size (2, b, h) and I want to change it to the following size: (b, 2*h), where the corresponding lists are concatenated, for example:
a = torch.tensor([[[1, 2, 3], [4, 5, 6], [4, 4, 4]],
                  [[4, 5, 6], [7, 8, 9], [5, 5, 5]]])

I want:
b = tensor([[1, 2, 3, 4, 5, 6],
            [4, 5, 6, 7, 8, 9],
            [4, 4, 4, 5, 5, 5]])

Output:
Use permute first to change order of dimensions, then contiguous to prevent strides within the permuted tensor and finally use view to reshape the tensor.
b = a.permute(1,0,2).contiguous().view(a.shape[1],-1)


Instruction: How to create a copy of nn.Sequential in torch?

Input:
I am trying to create a copy of a nn.Sequential network. For example, the following is the easiest way to do the same-
net = nn.Sequential(
        nn.Conv2d(16, 32, 3, stride=2),
        nn.ReLU(),
        nn.Conv2d(32, 64, 3, stride=2),
        nn.ReLU(),
    )

net_copy = nn.Sequential(
        nn.Conv2d(16, 32, 3, stride=2),
        nn.ReLU(),
        nn.Conv2d(32, 64, 3, stride=2),
        nn.ReLU(),
    )

However, it is not so great to define the network again. I tried the following ways but it didn't work-

net_copy = nn.Sequential(net): In this approach, it seems that net_copy is just a shared pointer of net
net_copy = nn.Sequential(*net.modules()): In this approach, net_copy contains many more layers.

Finally, I tired deepcopy in the following way which worked fine-
net_copy = deepcopy(net)

However, I am wondering if it is the proper way. I assume it is fine because it works.

Output:
Well, I just use torch.load and torch.save with io.BytesIO
import io, torch

# write to a buffer
buffer = io.BytesIO()
torch.save(model, buffer) #<--- model is some nn.module
print(buffer.tell()) #<---- no of bytes written

del model

# read from buffer
buffer.seek(0) #<--- must see to origin every time before reading
model = torch.load(buffer)
del buffer


Instruction: LayerNorm inside nn.Sequential in torch

Input:
I am trying to use LayerNorm inside nn.Sequential in torch. This is what I am looking for-
import torch.nn as nn

class LayerNormCnn(nn.Module):
    def __init__(self):
        super(LayerNormCnn, self).__init__()
        self.net = nn.Sequential(
                nn.Conv2d(3, 32, kernel_size=3, stride=2, padding=1),
                nn.LayerNorm(),
                nn.ReLU(),
                nn.Conv2d(32, 64, kernel_size=3, stride=2, padding=1),
                nn.LayerNorm(),
                nn.ReLU(),
            )

    def forward(self, x):
        x = self.net(x)
        return x

Unfortunately, it doesn't work because LayerNorm requires normalized_shape as input. The code above throws following exception-
    nn.LayerNorm(),
TypeError: __init__() missing 1 required positional argument: 'normalized_shape'

Right now, this is how I have implemented it-
import torch.nn as nn
import torch.nn.functional as F

class LayerNormCnn(nn.Module):
    def __init__(self, state_shape):
        super(LayerNormCnn, self).__init__()
        self.conv1 = nn.Conv2d(state_shape[0], 32, kernel_size=3, stride=2, padding=1)
        self.conv2 = nn.Conv2d(32, 64, kernel_size=3, stride=2, padding=1)

        # compute shape by doing a forward pass
        with torch.no_grad():
            fake_input = torch.randn(1, *state_shape)
            out        = self.conv1(fake_input)
            bn1_size   = out.size()[1:]
            out        = self.conv2(out)
            bn2_size   = out.size()[1:]

        self.bn1 = nn.LayerNorm(bn1_size)
        self.bn2 = nn.LayerNorm(bn2_size)

    def forward(self, x):
        x = F.relu(self.bn1(self.conv1(x)))
        x = F.relu(self.bn2(self.conv2(x)))
        return x

if __name__ == '__main__':
    in_shape   = (3, 128, 128)
    batch_size = 32

    model = LayerNormCnn(in_shape)
    x = torch.randn((batch_size,) + in_shape)
    out = model(x)
    print(out.shape)

Is it possible to use LayerNorm inside nn.Sequential?

Output:
The original layer normalisation paper advised against using layer normalisation in CNNs, as receptive fields around the boundary of images will have different values as opposed to the receptive fields in the actual image content. This issue does not arise with RNNs, which is what layer norm was originally tested for. Are you sure you want to be using LayerNorm? If you're looking to compare a different normalisation technique against BatchNorm, consider GroupNorm. This gets rid of the LayerNorm assumption that all channels in a layer contribute equally to a prediction, which is problematic particularly if the layer is convolutional. Instead, each channel is divided further into groups, that still allows a GN layer to learn different statistics across channels.
Please refer here for related discussion.


Instruction: Using sigmoid output for cross entropy loss on Pytorch

Input:
I’m trying to modify Yolo v1 to work with my task which each object has only 1 class. (e.g: an obj cannot be both cat and dog)
Due to the architecture (other outputs like localization prediction must be used regression) so sigmoid was applied to the last output of the model (f.sigmoid(nearly_last_output)). And for classification, yolo 1 also use MSE as loss. But as far as I know that MSE sometimes not going well compared to cross entropy for one-hot like what I want.
And specific: GT like this: 0 0 0 0 1 (let say we have only 5 classes in total, each only has 1 class so only one number 1 in them, of course this is class 5th in this example)
and output model at classification part: 0.1 0.1 0.9 0.2 0.1
I found some suggestion use nn.BCE / nn.BCEWithLogitsLoss but I think I should ask here for more correct since I’m not good at math and maybe I’m wrong somewhere so just ask to learn more and for sure what should I use correctly?

Output:
MSE loss is usually used for regression problem.

For binary classification, you can either use BCE or BCEWithLogitsLoss. BCEWithLogitsLoss combines sigmoid with BCE loss, thus if there is sigmoid applied on the last layer, you can directly use BCE.

The GT mentioned in your case refers to 'multi-class' classification problem, and the output shown doesn't really correspond to multi-class classification. So, in this case, you can apply a CrossEntropyLoss, which combines softmax and log loss and suitable for 'multi-class' classification problem.


Instruction: Get Predictions from Trained Pytorch Model

Input:
I am using transfer learning to fine tune an inception_v3 model. After I train the model and store the best version off I am attempting to use it to generate predictions for my test set. Below is an example of my attempt on one image.
img_test=Image.open("img.png")

#Perform same transformations to image that the model used
transform_pipeline = transforms.Compose([
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])])
img_test = transform_pipeline(img_test)

# I believe this is adding in the batch size of 1, but in looking around online it looked like I needed it
img = img_test.unsqueeze(0)
img = Variable(img)

model_ft(img)

When I do the above I get
RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same

Which seems to imply my model weights are on my gpu and the variable is on the cpu, how do I move one or the other over so I can use it, or reference one that is on the opposite processor?

Output:
As the error said, it seem that the input of the model (your img_test) is in the cpu.
Try to move the image to cuda before send it through your pre-trained model:
device = torch.device('cuda' if torch.cuda.is_available())
img_test = img_test.to(device)


Instruction: How to retain 2D (or more) shape when using pytrorch masked_select

Input:
Suppose I have the following two matching shape tensors:
a = tensor([[ 0.0113, -0.1666,  0.5960, -0.0667], [-0.0977, -0.1984,  0.5153,  0.0420]])
selectors = tensor([[ True,  True, False, False], [ True,  False, True, False]])

When using torch.masked_select to find the values in a that match True indices in selectors like this:
torch.masked_select(a, selectors)

The output will be in 1D shape instead of the original 2D shape:
tensor([ 0.0113, -0.1666, -0.0977, 0.5153])

This is consistent with masked_select behavior as it is given in the documentation (torch.masked_select). However, my goal is to get a result that matches the shape of the two original tensors. I.e.:
tensor([[0.0113, -0.1666], [-0.0977, 0.5153]])

Is there a way to get this without having to loop over all the elements in the tensors and find the mask for each one? Please note that I have also looked into using torch.where, but it doesn't fit the case I have as I see it.

Output:
As @jodag pointed out, for general inputs, each row on the desired masked result might have a different number of elements, depending on how many True values there are on the same row in selectors. However, you could overcome this by allowing trailing zero padding in the result.
Basic solution:
indices = torch.masked_fill(torch.cumsum(selectors.int(), dim=1), ~selectors, 0)
masked = torch.scatter(input=torch.zeros_like(a), dim=1, index=indices, src=a)[:,1:]

Explanation:
By applying cumsum() row-wise over selectors, we compute for each unmasked element in a the target column number it should be copied to in the output tensor. Then, scatter() performs a row-wise scattering of a's elements to these computed target locations. We leave all masked elements with the index 0, so that the first element in each row of the result would contain one of the masked elements (maybe arbitrarily. we don't care which). We then ignore these un-wanted 1st values by taking the slice [:,1:]. The output resulting masked tensor has the exact same size as the input a (this is the maximum needed size, for the case where there is a row of full True values in selectors).
Usage example:
>>> a = Torch.tensor([[ 1,  2,  3,  4,  5,  6], [10, 20, 30, 40, 50, 60]])
>>> selectors = Torch.tensor([[ True, False, False,  True, False,  True], [False, False,  True,  True, False, False]])
>>> torch.cumsum(selectors.int(), dim=1)
tensor([[1, 1, 1, 2, 2, 3],
        [0, 0, 1, 2, 2, 2]])
>>> indices = torch.masked_fill(torch.cumsum(selectors.int(), dim=1), ~selectors, 0)
>>> indices
tensor([[1, 0, 0, 2, 0, 3],
        [0, 0, 1, 2, 0, 0]])
>>> torch.scatter(input=torch.zeros_like(a), dim=1, index=indices, src=a)
tensor([[ 5,  1,  4,  6,  0,  0],
        [60, 30, 40,  0,  0,  0]])
>>> torch.scatter(input=torch.zeros_like(a), dim=1, index=indices, src=a)[:,1:]
tensor([[ 1,  4,  6,  0,  0],
        [30, 40,  0,  0,  0]])

Adapting output size: Here, the length of dim=1 of the output resulting masked tensor is the max number of un-masked items in a row. For your original show-case, the output shape would be (2,2) as you desired. Note that if this number is not previously known and a is on CUDA, it would cause an additional host-device synchronization that might affect the performance.
To do so, instead of allocating input=torch.zeros_like(a) for scatter(), allocate it by a.new_zeros(size=(a.size(0), torch.max(indices).item() + 1)). The +1 is for the 1st place which is later sliced-out. The host-device synchronization would occur by accessing the result of max() to calculate the allocated output size.
Example:
>>> torch.scatter(input=a.new_zeros(size=(a.size(0), torch.max(indices).item() + 1)), dim=1, index=indices, src=a)[:,1:]
tensor([[ 1,  4,  6],
        [30, 40,  0]])

Changing the padding value: If another custom default value is wanted as a padding, one could use torch.full_like(my_custom_value) rather than torch.zeros_like() when allocating the output for scatter().


Instruction: Non-deterministic behavior for training a neural network on GPU implemented in PyTorch and with a fixed random seed

Input:
I observed a strange behavior of the final Accuracy when I run exactly the same experiment (the same code for training neural net for image classification) with the same random seed on different GPUs (machines). I use only one GPU. Precisely, When I run the experiment on one machine_1 the Accuracy is 86,37. When I run the experiment on machine_2 the Accuracy is 88,0.
There is no variability when I run the experiment multiple times on the same machine. PyTorch and CUDA versions are the same. Could you help me to figure out the reason and fix it?
Machine_1:
NVIDIA-SMI 440.82       Driver Version: 440.82       CUDA Version: 10.2
Machine_2:
NVIDIA-SMI 440.100      Driver Version: 440.100      CUDA Version: 10.2
To fix random seed I use the following code:
random.seed(args.seed)
os.environ['PYTHONHASHSEED'] = str(args.seed)
np.random.seed(args.seed)
torch.manual_seed(args.seed)
torch.cuda.manual_seed(args.seed)
torch.backends.cudnn.benchmark = False
torch.backends.cudnn.deterministic = True

Output:
This is what I use:
import torch
import os
import numpy as np
import random

def set_seed(seed):
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False
    np.random.seed(seed)
    random.seed(seed)
    os.environ['PYTHONHASHSEED'] = str(seed)

set_seed(13)

Make sure you have a single function that set's the seeds from once. If you are using Jupyter notebooks cell execution timing may cause this. Also the order of functions inside may be important. I never had problems with this code. You may call set_seed() often in code.


Instruction: requires_grad = False seems not working in my case

Input:
I received a Cannot insert a Tensor that requires grad as a constant. Consider making it a parameter or input, or detaching the gradient error with tensor W.
W has the size of (10,10) and grad_fn=<DivBackward0>. The error happens at the second line
def muy(self, x):
    V = torch.tensor(self.W - self.lambda_ * torch.eye(self.ENCODING_DIM), requires_grad=False)
    return -0.5 * V.inverse().mm(self.b + self.lambda_ * x[:, None])

Other vars, values taken at the time of the error

self.lambda_: 1.0
self.ENCODING_DIM: 10
self.b: torch.Size([10, 1]), requires_grad=True
x: torch.Size([3, 1, 10]), grad_fn=<MulBackward0>

How could I set the result of muy as just an ingredient of the leaf node, so grad through V is required?
I tried this monstrosity, to no avail
def muy(self, x):
    V_inv = np.linalg.inv(self.V.detach().numpy())
    x_numpy = x[:, None].detach().numpy()
    temp= -0.5 * np.matmul(V_inv, self.b.detach().numpy() + self.lambda_ * x_numpy)
    return temp

Why I cared about this JIT:
I wanted to use tensorboard to visualize my model, if I understand the error messages right, the visualzing models use Tracer
EDIT
This still gives the same error, W or W.detach()
with torch.no_grad():
    V = self.W - self.lambda_ * torch.eye(self.ENCODING_DIM)
    return -0.5 * V.inverse().mm(self.b + self.lambda_ * x[:, None])

Output:
V = torch.tensor(self.W - self.lambda_ * torch.eye(self.ENCODING_DIM), requires_grad=False)

What you are trying to do here doesn't make much sense. torch.tensor(value) can only be created if the value is scalar (e.g. Python's 5), while you are trying to fit torch.Tensor there.
What you should do is simply this:
V = self.W - self.lambda_ * torch.eye(self.ENCODING_DIM)

If you want to detach self.W for some reason you can do this:
V = self.W.detach() - self.lambda_ * torch.eye(self.ENCODING_DIM)

(this will make a copy of self.W with requires_grad set to False).
You could also use torch.no_grad() context manager so this operation will not be recorded on the graph which will have the same effect on the graph (but only in this case, not in general and you won't make copy of self.W so it is advised to do that):
with torch.no_grad():
    V = self.W - self.lambda_ * torch.eye(self.ENCODING_DIM)

Code to reproduce
Can't reproduce this exact issue based on your code description, see below:
import torch

lambda_ = 1.0
W = torch.randn(10, 10, requires_grad=True)

ENCODING_DIM = 10
b = torch.randn(10, 1, requires_grad=True)
x = torch.randn(3, 1, 10, requires_grad=True)

with torch.no_grad():
    V = W - lambda_ * torch.eye(ENCODING_DIM)
    result = -0.5 * V.inverse().mm(b + lambda_ * x[:, None])

print(result)

This code gives the following (different!) error:

Traceback (most recent call last):   File "foo.py", line 13, in

result = -0.5 * V.inverse().mm(b + lambda_ * x[:, None]) RuntimeError: matrices expected, got 2D, 4D tensors at
/pytorch/aten/src/TH/generic/THTensorMath.cpp:36


Instruction: Retrieve elements from a 3D tensor with a 2D index tensor

Input:
I am playing around with GPT2 and I have 2 tensors:
O: An output tensor of shaped (B, S-1, V) where B is the batch size S is the the number of timestep and V is the vocabulary size. This is the output of a generative model and is softmaxed along the 2nd dimension.
L: A 2D tensor shaped (B, S-1) where each element is the index of the correct token for each timestep for each sample. This is basically the labels.
I want to extract the predicted probability of the corresponding correct token from tensor O based on tensor L such that I will end up with a 2D tensor shaped (B, S). Is there an efficient way of doing this apart from using loops?

Output:
For reference, I based my answer on this Medium article.
Essentially, your answer lies in torch.gather, assuming that both of your tensors are just regular torch.Tensors (or can be converted to one).
import torch

# Specify some arbitrary dimensions for now
B = 3
V = 6
S = 4

# Make example reproducible
torch.manual_seed(42)

# L necessarily has to be a torch.LongTensor, otherwise indexing will fail.
L = torch.randint(0, V, size=[B, S])

O = torch.rand([B, S, V])

# Now collect the results. L needs to have similar dimension,
# except in the axis you want to collect along.
X = torch.gather(O, dim=2, index=L.unsqueeze(dim=2))

# Make sure X has no "unnecessary" dimension
X = X.squeeze(dim=2)

It is a bit difficult to see whether this produces the exact correct results, which is why I included a random seed which makes the example deterministic in the result, and you an easily verify that it gets you the desired results. However, for clarification, one could also use a lower-dimensional tensor, for which this becomes clearer what exactly torch.gather does.
Note that torch.gather also allows you to index multiple indexes in the same row theoretically. Meaning if you instead got a multiclass example for which multiple values are correct, you could similarly use a tensor L of shape [B, S, number_of_correct_samples].


Instruction: Moving a tensor to cuda device cause illegal memory access in Pytorch

Input:
I am trying the following snippet in Colab but causes the following error.
Is it wrong to move a tensor object to Cuda device?.
import torch
a = torch.Tensor(torch.randn(5,5,5))
# a.device("cuda")
device = torch.device("cuda")
class abc(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.w1 = torch.nn.Linear(5,5)

    def forward(self,x):
        return self.w1(x)
mod = abc()
a.cuda()
mod.to(device)
mod(a)

Output:
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-10-5372fb4d5512> in <module>()
     11         return self.w1(x)
     12 mod = abc()
---> 13 a.cuda()
     14 mod.to(device)
     15 mod(a)

RuntimeError: CUDA error: an illegal memory access was encountered

Output:
This works for me on Google colab:
import torch
a = torch.randn(5,5,5)
a = a.to("cuda") # or just a = torch.randn((5,5,5), device='cuda')

class abc(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.w1 = torch.nn.Linear(5,5)

    def forward(self,x):
        return self.w1(x)
mod = abc()
mod.to("cuda")
mod(a)

Output:
tensor([[[ 1.5691e+00,  8.0326e-01,  1.4352e+00,  7.3295e-01,  3.2156e-01],
         [ 5.1630e-01, -2.2816e-03,  7.1052e-01,  1.9250e-01,  8.3110e-01],
         [ 7.6572e-01, -8.9701e-01,  2.7974e-01,  7.4309e-04,  9.5218e-01],
         [ 2.0723e-01, -1.0049e+00,  1.6938e+00,  1.0019e+00,  7.9305e-01],
         [-1.0973e-02, -1.1260e-01,  1.0521e+00, -1.3839e-01, -4.2380e-01]],

        [[ 1.3870e+00,  1.1620e+00, -3.6523e-01, -5.6704e-01,  4.2481e-01],
         [ 1.6204e-01,  8.3231e-02, -5.9607e-01, -1.0912e+00, -6.1651e-01],
         [ 2.3584e-01, -5.9825e-01,  1.1670e+00,  9.3185e-01,  4.0269e-01],
         [ 1.3120e+00,  1.3967e-01, -5.5048e-01, -9.8143e-01,  3.5059e-01],
         [ 8.0019e-01, -1.8983e-02,  2.3792e-01, -5.9157e-01,  3.5816e-01]],

        [[ 3.9709e-01, -8.7349e-01, -2.9742e-01, -3.8732e-01, -1.7191e-03],
         [-8.7056e-01, -8.8214e-01,  1.0647e+00,  7.7785e-01,  6.3816e-01],
         [ 7.4920e-01, -4.0143e-01,  5.9780e-01,  2.7842e-01,  8.1991e-01],
         [-5.9389e-02, -4.9465e-01, -3.7322e-03, -7.0475e-01, -2.5955e-01],
         [ 1.5722e+00,  6.4410e-01, -5.1310e-02, -1.2716e+00, -1.4607e-01]],

        [[ 6.5152e-02, -6.8772e-01,  1.0366e+00, -2.4278e-01, -2.7106e-01],
         [ 7.0832e-01,  1.4581e-01,  1.9924e-01, -4.1930e-01,  4.0567e-01],
         [ 3.9120e-01, -1.0099e+00,  1.6907e+00,  7.2674e-01,  6.5285e-01],
         [-1.3191e-01, -8.6324e-01, -1.2734e-01, -5.6135e-01, -4.1949e-01],
         [ 5.4183e-02, -5.6837e-01,  5.1347e-02, -5.3199e-01,  2.2167e-01]],

        [[ 9.9237e-02, -5.8725e-01, -3.3042e-01, -8.7371e-01, -2.3261e-01],
         [ 5.5485e-01, -3.5022e-01,  1.1516e-01,  3.8139e-02,  4.6032e-01],
         [-7.5111e-01, -9.7203e-01,  1.7809e-01,  2.2506e-01,  3.6540e-02],
         [ 2.5590e-01,  3.0592e-01,  6.8972e-01,  1.8452e-01,  6.7794e-01],
         [-7.6091e-01, -1.3956e+00,  7.8801e-01, -1.7489e-01, -1.0143e+00]]],
       device='cuda:0', grad_fn=<AddBackward0>)


Instruction: How to avoid overfitting in deep learning when features are binary in nature

Input:
I am constructing a deep learning model using 2048 bits of binary fingerprints (0 and 1's) for some 2000 samples to predict their outputs (positive (1) OR negative(0)). The feature data is quite sparse i.e. lots of zeros and rare 1's.
I have used 'binary cross entropy' but my validation accuracy doesn't increase more than 70%. I have balanced data. The model seems to be overfitting. I can't normalize my data since fetures are binary. How can I avoid overfitting?
    earlystop = EarlyStopping(monitor='val_acc', patience=20, mode='max')
    final_model = Sequential()
    final_model.add(Dense(1012, input_dim=2048, activation = 'relu'))
    final_model.add(BatchNormalization())
    final_model.add(Dropout(0.9))
    final_model.add(Dense(512, activation='relu'))
    final_model.add(BatchNormalization())
    final_model.add(Dropout(0.9))
    final_model.add(Dense(128, activation='relu'))
    final_model.add(BatchNormalization())
    final_model.add(Dropout(0.5))
    final_model.add(Dense(32, activation='relu'))
    final_model.add(Dense(1, activation='sigmoid'))
    adam_opt = keras.optimizers.Adam(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0, amsgrad=False)
    final_model.compile(loss='binary_crossentropy', optimizer=adam_opt, metrics=['accuracy'])

Output:
If you want to do a binary classification, binary crossentropy is the loss function you are looking for.
Achieving a well generalizing model includes more than just the right loss function choice (Preprocessing Data, Finding a proper Network Architecture, Finding the right hyper parameter choice, ...).
You can find a discussion about generalization of Deep Learning Models here:
https://stats.stackexchange.com/questions/365778/what-should-i-do-when-my-neural-network-doesnt-generalize-well


Instruction: PyTorch: How to check if some weights are not changed during training?

Input:
How can I check if some weights are not changed during training in PyTorch?
As I understand one option can be just dump model weights at some epochs and check if they are changed iterating over weights, but maybe there is some simpler way?

Output:
There can be two ways around this:
First
        for name, param in model.named_parameters():
            if 'weight' in name:
                temp = torch.zeros(param.grad.shape)
                temp[param.grad != 0] += 1
                count_dict[name] += temp

This step comes in after your loss.backward() step in the training module. The count_dict[name] dictionary keeps track of the gradient updates. You can initialize it this way before the start of training:
    for name, param in model.named_parameters():
        if 'weight' in name:
            count_dict[name] = torch.zeros(param.grad.shape)

Now one more way would be to register a hook function and then create the hook function where you can even update the or modify the gradients if you want to. This is not necessary to keep track of the weight updates but then if you want to do something with the gradient, it comes in handy.
Suppose, here I am randomly sparsifying the gradients.
def hook_fn(grad):
    '''
    Randomly sparsify the gradients
    :param grad: Input gradient of the layer
    :return: grad_clone - the sparsified FC layer gradients
    '''
    grad_clone = grad.clone()
    temp = torch.cuda.FloatTensor(grad_clone.shape).uniform_()
    grad_clone[temp < 0.8] = 0
    return grad_clone

And here I give the model the hook.
for name, param in model.named_parameters():
    if 'weight' in name:
            param.register_hook(hook_fn)

So, this might just sparsify the gradients for you, and you can keep track of gradients in the hook function itself in this way:
def hook_func(module, input, output):
    temp = torch.zeros(output.shape)
    temp[output != 0] += 1
    count_dict[module] += temp

Although, I won't recommend doing this. This is generally useful in the case of visualizing the forward pass features/activations. And also, input and output can confuse because the gradient and param inputs and outputs are reversed.


Instruction: Pytorch Multi-GPU Issue

Input: I want to train my model with 2 GPU(id 5, 6), so I run my code with CUDA_VISIBLE_DEVICES=5,6 train.py. However, when I printed torch.cuda.current_device I still got the id 0 rather than 5,6. But torch.cuda.device_count is 2, which semms right. How can I use GPU5,6 correctly?

Output:
It is most likely correct. PyTorch only sees two GPUs (therefore indexed 0 and 1) which are actually your GPU 5 and 6.
Check the actual usage with nvidia-smi. If it is still inconsistent, you might need to set an environment variable:
export CUDA_DEVICE_ORDER=PCI_BUS_ID

(See Inconsistency of IDs between 'nvidia-smi -L' and cuDeviceGetName())


Instruction: Process stuck when training on multiple nodes using PyTorch DistributedDataParallel

Input:
I am trying to run the script mnist-distributed.py from Distributed data parallel training in Pytorch. I have also pasted the same code here. (I have replaced my actual MASTER_ADDR with a.b.c.d for posting here).
import os
import argparse
import torch.multiprocessing as mp
import torchvision
import torchvision.transforms as transforms
import torch
import torch.nn as nn
import torch.distributed as dist

class ConvNet(nn.Module):
    def __init__(self, num_classes=10):
        super(ConvNet, self).__init__()
        self.layer1 = nn.Sequential(
            nn.Conv2d(1, 16, kernel_size=5, stride=1, padding=2),
            nn.BatchNorm2d(16),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2))
        self.layer2 = nn.Sequential(
            nn.Conv2d(16, 32, kernel_size=5, stride=1, padding=2),
            nn.BatchNorm2d(32),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2))
        self.fc = nn.Linear(7*7*32, num_classes)

    def forward(self, x):
        out = self.layer1(x)
        out = self.layer2(out)
        out = out.reshape(out.size(0), -1)
        out = self.fc(out)
        return out

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('-n', '--nodes', default=1, type=int, metavar='N')
    parser.add_argument('-g', '--gpus', default=1, type=int,
                        help='number of gpus per node')
    parser.add_argument('-nr', '--nr', default=0, type=int,
                        help='ranking within the nodes')
    parser.add_argument('--epochs', default=2, type=int, metavar='N',
                        help='number of total epochs to run')
    args = parser.parse_args()
    args.world_size = args.gpus * args.nodes
    os.environ['MASTER_ADDR'] = 'a.b.c.d'
    os.environ['MASTER_PORT'] = '8890'
    mp.spawn(train, nprocs=args.gpus, args=(args,))

def train(gpu, args):
    rank = args.nr * args.gpus + gpu
    dist.init_process_group(
        backend='nccl',
        init_method='env://',
        world_size=args.world_size,
        rank=rank
    )

    torch.manual_seed(0)
    model = ConvNet()
    torch.cuda.set_device(gpu)
    model.cuda(gpu)
    batch_size = 100
    # define loss function (criterion) and optimizer
    criterion = nn.CrossEntropyLoss().cuda(gpu)
    optimizer = torch.optim.SGD(model.parameters(), 1e-4)

    # Wrap the model
    model = nn.parallel.DistributedDataParallel(model,
                                                device_ids=[gpu])

    # Data loading code
    train_dataset = torchvision.datasets.MNIST(
        root='./data',
        train=True,
        transform=transforms.ToTensor(),
        download=True
    )
    train_sampler = torch.utils.data.distributed.DistributedSampler(
        train_dataset,
        num_replicas=args.world_size,
        rank=rank
    )

    train_loader = torch.utils.data.DataLoader(
        dataset=train_dataset,
       batch_size=batch_size,
       shuffle=False,
       num_workers=0,
       pin_memory=True,
      sampler=train_sampler)

    total_step = len(train_loader)
    for epoch in range(args.epochs):
        for i, (images, labels) in enumerate(train_loader):
            images = images.cuda(non_blocking=True)
            labels = labels.cuda(non_blocking=True)
            # Forward pass
            outputs = model(images)
            loss = criterion(outputs, labels)

            # Backward and optimize
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
            if (i + 1) % 100 == 0 and gpu == 0:
                print('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}'.format(
                    epoch + 1,
                    args.epochs,
                    i + 1,
                    total_step,
                    loss.item())
                   )

if __name__ == '__main__':
    main()

There are 2 nodes with 2 GPUs each. I run this command from the terminal of the master node-

python mnist-distributed.py -n 2 -g 2 -nr 0

, and then this from the terminal of the other node-

python mnist-distributed.py -n 2 -g 2 -nr 1

But then my process gets stuck with no output on either terminal.
Running the same code on a single node using the following command works perfectly fine-

python mnist-distributed.py -n 1 -g 2 -nr 0

Output:
I met a similar problem. And the problem is solved by
sudo vi /etc/default/grub

Edit it:
#GRUB_CMDLINE_LINUX=""                           <----- Original commented
GRUB_CMDLINE_LINUX="iommu=soft"           <------ Change

sudo update-grub

Reboot to see the change.
Ref: https://github.com/pytorch/pytorch/issues/1637#issuecomment-338268158


Instruction: Why am I getting calculated padding input size per channel smaller than kernel size?

Input:
I have the following model but its returning an error. Not sure why. I have tried googling but not found anything so far. My input is an numpy array of 6 by 6.
class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 16, kernel_size=(3,3), stride=1, padding=0)
        self.conv2 = nn.Conv2d(16, 32, kernel_size=(3,3), stride=1, padding=0)
        self.conv3 = nn.Conv2d(32, 64, kernel_size=(3,3), stride=1, padding=0)

        self.fc1 = nn.Linear(64*4*4, 320)
        self.fc2 = nn.Linear(320, 160)
        self.out = nn.Linear(160, 2)

    def forward(self, x):
        x = self.conv1(x)
        x = F.relu(x)
        x = F.max_pool2d(x, kernel_size=2, stride=2)

        x = self.conv2(x)
        x = F.relu(x)
        x = F.max_pool2d(x, kernel_size=2, stride=2)

        x = self.conv3(x)
        x = F.relu(x)
        x = F.max_pool2d(x, kernel_size=2, stride=2)

        x = x.reshape(-1, 64*4*4)
        #x = torch.flatten(x)

        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.out(x)
        return F.softmax(x, dim=1)

My input is a 6x6 numpy array and I get the following error, any idea why?
RuntimeError: Calculated padded input size per channel: (2 x 2). Kernel size: (3 x 3). Kernel size can't be greater than actual input size

Output:
Here is what you may do and I used the padding=1 as proposed by Szymon Maszke. This padding is added to the convolution and to maxpooling.
import numpy
import torch
import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 16, kernel_size=(3,3), stride=1, padding=1)
        self.conv2 = nn.Conv2d(16, 32, kernel_size=(3,3), stride=1, padding=1)
        self.conv3 = nn.Conv2d(32, 64, kernel_size=(3,3), stride=1, padding=1)

        self.fc1 = nn.Linear(64*4*4, 320)
        self.fc2 = nn.Linear(320, 160)
        self.out = nn.Linear(160, 2)

    def forward(self, x):
        x = self.conv1(x)
        x = F.relu(x)
        x = F.max_pool2d(x, kernel_size=2, stride=2)
        x = self.conv2(x)
        x = F.relu(x)
        x = F.max_pool2d(x, kernel_size=3, stride=2, padding=1)
        x = self.conv3(x)
        x = F.relu(x)
        x = F.max_pool2d(x, kernel_size=3, stride=2, padding=1)
        x = x.reshape(-1, 64*4*4)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.out(x)
        return F.softmax(x, dim=1)

a = numpy.random.rand(6,6)
print(a)
data = torch.tensor(a).float()
print(data.shape)
# data.unsqueeze_(0).unsqueeze_(0)
data= data.expand(16, 1 ,-1,-1)
print(data.shape)

n=Net()
print("Start")
o = n(data)
print(o)

Out:
[[0.89695967 0.09447725 0.0905144  0.52694105 0.66000333 0.10537102]
 [0.32854697 0.86046884 0.29804184 0.62988374 0.5965067  0.54139821]
 [0.41561266 0.95484358 0.82919364 0.75556819 0.77373267 0.52209278]
 [0.46406436 0.6553954  0.60010151 0.86314529 0.70020608 0.16471554]
 [0.72863547 0.83846636 0.95122373 0.84322402 0.32264676 0.1233866 ]
 [0.75767067 0.56546123 0.7765021  0.35303595 0.3254407  0.84033049]]
torch.Size([6, 6])
torch.Size([16, 1, 6, 6])
Start
tensor([[0.5134, 0.4866]], grad_fn=<SoftmaxBackward>)

By default in PyTorch padding=0, so you need to explicitly set padding=1 when needed.


Instruction: How to install PyTorch with pipenv and save it to Pipfile and Pipfile.lock?

Input:
I’m currently using Pipenv to maintain the Python packages used in a specific project. Most of the downloads I’ve tried so far have worked as intended; that is, I enter pipenv install [package] and it installs the package into the virtual environment, then records the package information into both the Pipfile and Pipfile.lock.
However, I’m running into some problems installing PyTorch.
I’ve tried running pipenv install torch, but every time the locking step fails. Instead, I’ve tried forcing a download directly from the PyTorch website using
pipenv run pip install torch===1.6.0 torchvision===0.7.0 -f https://download.pytorch.org/whl/torch_stable.html

And it actually installs! If I run pipenv graph it displays both torch and torchvision with their dependencies. But one problem remains: neither torch nor torchvision are being saved into Pipfile and Pipfile.lock.
Any idea on how I can make this happen?

Output:
When you use pipenv run pip install <package>, that skips the custom pipenv operations of updating the Pipfile and the Pipfile.lock. It is basically equivalent to doing a plain pip install <package> as if you did not have/use pipenv.
The only way to also update the Pipfile's is to use pipenv install.
Unfortunately, as I'm posting this, pipenv does not have an equivalent for pip's -f/--find-links option. The best solution is to specify pytorch's "https://download.pytorch.org/whl/" URLs as an  alternative package index, by adding it as a [[source]] in your Pipfile. See this answer from Mohamad and this answer from Mitch McMabers that describes how to do it. I recommend trying out those answers instead.
A less elegant and quite bad alternative is to manually find the correct torch wheel (.whl) links you need, which usually means looking for the correct link from https://download.pytorch.org/whl/torch_stable.html. Then, create/modify the Pipfile with the specific package versions and URLs to the wheels:
[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[requires]
python_version = "3.8"

[packages]
torch = {version = "==1.6.0", file = "https://download.pytorch.org/whl/cpu/torch-1.6.0-cp38-none-macosx_10_9_x86_64.whl"}
torchvision = {version = "==0.7.0", file = "https://download.pytorch.org/whl/torchvision-0.7.0-cp38-cp38-macosx_10_9_x86_64.whl"}

Then just do normal pipenv install.
You can confirm the installation with pipenv install --verbose:
Collecting torch==1.6.0
  ...
  Looking up "https://download.pytorch.org/whl/cpu/torch-1.6.0-cp38-none-macosx_10_9_x86_64.whl" in the cache
  Current age based on date: 8
  Starting new HTTPS connection (1): download.pytorch.org:443
  https://download.pytorch.org:443 "GET /whl/cpu/torch-1.6.0-cp38-none-macosx_10_9_x86_64.whl HTTP/1.1" 304 0
  ...
  Added torch==1.6.0 from https://download.pytorch.org/whl/cpu/torch-1.6.0-cp38-none-macosx_10_9_x86_64.whl#egg=torch
...
Successfully installed torch-1.6.0

Collecting torchvision==0.7.0
  ...
  Looking up "https://download.pytorch.org/whl/torchvision-0.7.0-cp38-cp38-macosx_10_9_x86_64.whl" in the cache
  Current age based on date: 8
  Starting new HTTPS connection (1): download.pytorch.org:443
  https://download.pytorch.org:443 "GET /whl/torchvision-0.7.0-cp38-cp38-macosx_10_9_x86_64.whl HTTP/1.1" 304 0
  ...
  Added torchvision==0.7.0 from https://download.pytorch.org/whl/torchvision-0.7.0-cp38-cp38-macosx_10_9_x86_64.whl#egg=torchvision
...
Successfully installed torchvision-0.7.0

This also adds entries to Pipfile.lock:
"torch": {
    "file": "https://download.pytorch.org/whl/cpu/torch-1.6.0-cp38-none-macosx_10_9_x86_64.whl",
    "hashes": [
        ...
    ],
    "index": "pypi",
    "version": "==1.6.0"
},
"torchvision": {
    "file": "https://download.pytorch.org/whl/torchvision-0.7.0-cp38-cp38-macosx_10_9_x86_64.whl",
    "hashes": [
        ...
    ],
    "index": "pypi",
    "version": "==0.7.0"
}

With that, you now have a Pipfile and Pipfile.lock that you can check-in/commit to version control and track/manage as you develop your application.
Instead of manually editing the Pipfile, you can also do it from the command line:
(temp) $ pipenv install --verbose "https://download.pytorch.org/whl/torchvision-0.7.0-cp38-cp38-macosx_10_9_x86_64.whl"
Installing https://download.pytorch.org/whl/torchvision-0.7.0-cp38-cp38-macosx_10_9_x86_64.whl...
...
Adding torchvision to Pipfile's [packages]...
✔ Installation Succeeded

That should also add an entry to the Pipfile:
[packages]
...
torchvision = {file = "https://download.pytorch.org/whl/torchvision-0.7.0-cp38-cp38-macosx_10_9_x86_64.whl"}

Of course, this all depends on finding out which wheel you actually need. This can be done by first using a plain pip install <package> with the -f/--find-links option targeting the https://download.pytorch.org/whl/torch_stable.html URL, then checking which wheel it used.

First, let's get the correct .whl file with pip install
$ pipenv run pip install --verbose torchvision==0.7.0 -f https://download.pytorch.org/whl/torch_stable.html
Looking in links: https://download.pytorch.org/whl/torch_stable.html
...
Collecting torchvision==0.7.0
  Downloading torchvision-0.7.0-cp38-cp38-macosx_10_9_x86_64.whl (387 kB)
...

Remove pip install-ed things from the virtual environment
$ pipenv clean

Repeat installation but using pipenv install
$ pipenv install --verbose "https://download.pytorch.org/whl/torchvision-0.7.0-cp38-cp38-macosx_10_9_x86_64.whl"

Just combine "https://download.pytorch.org/whl/" + .whl filename from step 1

It might seem a bit backwards using pip install first then copying it over to pipenv, but the objective here is to let pipenv update the Pipfile and Pipfile.lock (to support deterministic builds) and to "document" your env for version control.


Instruction: Using trained BERT Model and Data Preprocessing

Input:
When using a pre-trained BERT embeddings from pytorch (which are then fine-tuned), should the text data fed into the model be pre-processed like in any standard NLP task?
For instance, should  stemming, removing low frequency words, de-captilisation, be performed or should the raw text simply be passed to `transformers.BertTokenizer'?

Output:
I think preprocessing will not change your output predictions. I will try to explain for each case you mentioned -

stemming or lemmatization :
Bert uses BPE (Byte- Pair Encoding to shrink its vocab size), so words like run and running will ultimately be decoded to run + ##ing.
So it's better not to convert running into run because, in some NLP problems, you need that information.
De-Capitalization - Bert provides two models (lowercase and uncased). One converts your sentence into lowercase, and others will not change related to the capitalization of your sentence. So you don't have to do any changes here just select the model for your use case.
Removing high-frequency words -
Bert uses the Transformer model, which works on the attention principal.
So when you finetune it on any problem, it will look only on those words which will impact the output and not on words which are common in all data.


Instruction: CNN model is overfitting to data after reaching 50% accuracy

Input:
I am trying to identify 3 (classes) mental states based on EEG connectome data. The shape of the data is 99x1x34x34x50x130 (originally graph data, but now represented as a matrix), with respectably represent [subjects, channel, height, width, freq, time series]. For the sake of this study, can only input a 1x34x34 image of the connectome data. From previous studies, it was found that the alpha band (8-1 hz) had given the most information, thus the dataset was narrowed down to 99x1x34x34x4x130. The testing set accuracy on pervious machine learning techniques such as SVMs reached a testing accuracy of ~75%. Hence, by goal is to achieve a greater accuracy given the same data (1x34x34). Since my data is very limited 1-66 for training and 66-99 for testing (fixed ratios and have a 1/3 class distribution), I thought of splitting the data along the time series axis (6th axis) and then averaging the data to a shape of 1x34x34 (from ex. 1x34x34x4x10, 10 is the random sample of time series). This gave me ~1500 samples for training, and 33 for testing (testing is fixed, the class distributions are 1/3).
Model:
SimpleCNN(
  (conv1): Conv2d(1, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (conv2): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (pool1): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (drop1): Dropout(p=0.25, inplace=False)
  (fc1): Linear(in_features=9248, out_features=128, bias=True)
  (drop2): Dropout(p=0.5, inplace=False)
  (fc2): Linear(in_features=128, out_features=3, bias=True)
)
CrossEntropyLoss()
Adam (
Parameter Group 0
    amsgrad: False
    betas: (0.9, 0.999)
    eps: 1e-08
    lr: 5e-06
    weight_decay: 0.0001
)

Results:
The training set can achieve an accuracy of 100% with enough iteration, but at the cost of the testing set accuracy. After around 20-50 epochs of testing, the model starts to overfit to the training set and the test set accuracy starts to decrease (same with loss).

What I have tried:
I have tried tuning the hyperparameters: lr=.001-000001, weight decay=0.0001-0.00001. Training to 1000 epochs (useless bc overfitting in less than 100 epochs). I have also tried increasing/decreasing the model complexity with adding adding addition fc layers and varying amount of channels in CNN layers form 8-64. I have also tried adding more CNN layers and the model did a bit worse averaging around an accuracy of ~45% on the test set. I tried manually scheduling the learning rate every 10 epochs, the results were the same. Weight decay didn’t seem to effect the results much, changed it from 0.1-0.000001.
From previous testing, I have a model that achieves 60% acc on both the testing and the training set. However, when I try to retrain it, the acc instantly goes down to ~40 on both sets (training and testing), which makes no sense. I have tried altering the learning rate from 0.01 to 0.00000001, and also tried weight decay for this.
From training the model and the graphs, it seems like the model dosn’t know what it’s doing for the first 5-10 epochs and then starts to learn rapidly to around ~50%-60% acc on both sets. This is where the model starts to overfit, form there the model’s acc increases to 100% on the training set, and the acc for the testing set goes down to 33%, which is equivalent to guessing.
Any tips?
Edit:
The model’s outputs for the test set are very very close to each other.
0.33960407972335815, 0.311821848154068, 0.34857410192489624

The average standard deviation for the whole test set between predictions for each image are (softmax):
0.017695341517654846

However, the average std for the training set is .22 so...
F1 Scores:
Micro Average: 0.6060606060606061
Macro Average: 0.5810185185185186
Weighted Average: 0.5810185185185186
Scores for each class: 0.6875 0.5 0.55555556

Here is a confusion matrix:

Output:
I have some suggestions, what I would try, maybe you've already done it:

increase the probability of dropout, that could decrease overfitting,
I did not see or I missed it but if you don't do it, shuffle all the samples,
there is not so much data, did you thought about using other NN to generate more data of the classes which are having the least score? I am not sure if it is the case here but even randomly rotating, scaling the images can produce more training examples,
another approach you can take, if you haven't done it already, use transfer learning using another popular CNN net and see how it is doing the job, then you can have some comparison, whether it is something wrong with your architecture or it's lack of examples :)
I know these are just suggestions but maybe, if you haven't try some of them, they will bring you closer to the solution.
Good luck!


Instruction: How to find the mean and the covariance of a 2d activation map (pytorch)

Input: I have a tensor of shape [h, w], which consists of a normalized, 2-dimensional activation map. Considering this to be some distribution, I want to find the mean and the covariance within this activation map in pytorch. Is there an efficient way to do that?

Output:
You can use the following code, where activation_map is a tensor of shape (h,w), with non-negative elements, and is normalised (activation_map.sum() is 1):
activation_map = torch.tensor(
    [[0.2, 0.1, 0.0],
     [0.1, 0.2, 0.4]])
h, w = activation_map.shape

range_h = torch.arange(h)
range_w = torch.arange(w)
idxs = torch.stack([
  range_w[None].repeat(h, 1),
  range_h[:, None].repeat(1, w)
  ])
map_flat = activation_map.view(-1)
idxs_flat = idxs.reshape(2, -1).T
mean = (map_flat[:, None] * idxs_flat).sum(0)
mats = idxs_flat[:, :, None] @ idxs_flat[:, None, :]
second_moments = (map_flat[:, None, None] * mats).sum(0)
covariance = second_moments - mean[:, None] @ mean[None]

# mean:
# tensor([1.1000, 0.7000])
# covariance:
# tensor([[0.6900, 0.2300],
#         [0.2300, 0.2100]])


Instruction: How to specify different layer sizes in Pytorch LSTM/GRU/RNN

Input:
so I know how to work with LSTMs in general with Pytorch. But it bugs me, that you can only specify ONE hidden_size for all your layers in the LSTM. Like this:
lstm = nn.LSTM(input_size=26, hidden_size=128, num_layers=3, dropout=dropout_chance, batch_first=True)

So for all three layers, the size will be 128. But is there really no way to say, for example, that the first layer should be 128, the second 32 and the third 128?
If I missed something in the documentation or you know a work-around, please let me know, thank ya!

Output:
Actually, it depends on the shape of your input and you can see How to decide input and hidden layer dimension to torch.nn.RNN?. Also, you have to understand what is the input and the output because there are different ways to deal with the input and the output. In the  A Beginner’s Guide on Recurrent Neural Networks with PyTorch, you can see how the input data is taken in by the model.

Your model can be
lstm = nn.LSTM(input_size=26, hidden_size=128, num_layers=3, dropout=dropout_chance, batch_first=True)
lstm2 = nn.LSTM(input_size=26, hidden_size=32, num_layers=3, dropout=dropout_chance, batch_first=True)
lstm3 = nn.LSTM(input_size=26, hidden_size=128, num_layers=3, dropout=dropout_chance, batch_first=True)

For multi-layer see this model.
# sequence classification model
class M1(nn.Module):
    def __init__(self):
        super(M1, self).__init__()

        self.recurrent_layer  = nn.LSTM(hidden_size = 100, input_size = 75, num_layers = 5)
        self.recurrent_layer1  = nn.LSTM(hidden_size = 200, input_size = 100, num_layers = 5)
        self.recurrent_layer2  = nn.LSTM(hidden_size = 300, input_size = 200, num_layers = 5)
        self.project_layer     = nn.Linear(300, 200)
        self.project_layer1    = nn.Linear(200, 100)
        self.project_layer2    = nn.Linear(100, 10)

    # the size of input is [batch_size, seq_len(15), input_dim(75)]
    # the size of logits is [batch_size, num_class]
    def forward(self, input, h_t_1=None, c_t_1=None):
        # the size of rnn_outputs is [batch_size, seq_len, rnn_size]
        # self.recurrent_layer.flatten_parameters()
        rnn_outputs, (hn, cn) = self.recurrent_layer(input)
        rnn_outputs, (hn, cn) = self.recurrent_layer1(rnn_outputs)
        rnn_outputs, (hn, cn) = self.recurrent_layer2(rnn_outputs)
        # classify the last step of rnn_outpus
        # the size of logits is [batch_size, num_class]
        logits = self.project_layer(rnn_outputs[:,-1])
        logits = self.project_layer1(logits)
        logits = self.project_layer2(logits)
        return logits


Instruction: Is there any way to get torch.mode over multidimensional tensor

Input:
is there any way torch.mode can be applied over multiple dimensions
for example
import numpy as np
import torch
x = np.random.randint(10, size=(3, 5))
y = torch.tensor(x)

lets say y has
[[6 3 7 3 0]
 [2 5 7 9 7]
 [6 1 4 6 3]]

torch.mode should return a size 3 tensor [3,7,6]
without using a loop

Output:
Use the dimension attribute in torch to select which dimension should be reduced using mode operator.
torch.mode(y, dim = 1)[0]

Will give you the desired answer.


Instruction: How can I assign a list to a torch.tensor?

Input:
Assume that I have a list [(0,0),(1,0),(1,1)] and another list [4,5,6] and a matrix X with size is (3,2). I am trying to assign the list to the matrix like X[0,0] = 4, X[1,0] = 5 and X[1,1] = 6. But it seems like I have a problem of assigning a list to tensor
x = torch.zeros(3,2)
indices = [(0,0),(1,0),(1,1)]
value = [4,5,6]
x[indices] = values

Error:
TypeError                                 Traceback (most recent call last)
<ipython-input-7-dec4e6a479a5> in <module>

   4 indices = [(0, 0), (1, 0), (1, 1)]

   5 values = [4, 5, 6]

----> 6 x[indices] = values

TypeError: can't assign a list to a torch.FloatTensor

Output:
In general, the answer to "how do I change a list to a Tensor" is to use torch.Tensor(list). But that will not solve your actual problem here.
One way would be to associate the index and value and then iterate over them:
for (i,v) in zip(indices,values) :
   x[i] = v


Instruction: Why is Loss of SGD for a dataset is not matching the pytorch code with the scratch python code for linear regression?

Input:
I'm trying to implement Multiple Linear regression on the wine dataset. But when I compare the results of Pytorch with scratch code of Python the losses are not coming same.
My Scratch Code:
Functions:
def yinfer(X, beta):
  return beta[0] + np.dot(X,beta[1:])

def cost(X, Y, beta):
  sum = 0
  m = len(Y)
  for i in range(m):
    sum = sum + ( yinfer(X[i],beta) - Y[i])*(yinfer(X[i],beta) - Y[i])
  return  sum/(1.0*m)

Main Code:
alpha = 0.005
b=[0,0.04086357 ,-0.02831656  ,0.09622949 ,-0.15162516  ,0.60188454  ,0.47528714,
  -0.6066466  ,-0.22995654 ,-0.58388734  ,0.20954669 ,-0.67851365]
beta = np.array(b)
print(beta)
iterations = 1000
arr_cost = np.zeros((iterations,2))
m = len(Y)
temp_beta = np.zeros(12)
for i in range(iterations):
  for k in range(m):
        temp_beta[0] =  yinfer(X[k,:], beta) - Y[k]
        temp_beta[1:] = (yinfer(X[k,:], beta) - Y[k])*X[k,:]
        beta = beta - alpha*temp_beta/(1.0*m)    #(m*np.linalg.norm(temp_beta))
  arr_cost[i] = [i,cost(X,Y,beta)]
  #print(cost(X,Y,beta))
plt.scatter(arr_cost[0:iterations,0], arr_cost[0:iterations,1])

I have used same weights that were used in Pytorch code
My Pytorch code:
class LinearRegression(nn.Module):
  def __init__(self,n_input_features):
    super(LinearRegression,self).__init__()
    self.linear=nn.Linear(n_input_features,1)
    # self.linear.weight.data=b.view(1,-1)
    self.linear.bias.data.fill_(0.0)
    nn.init.xavier_uniform_(self.linear.weight)
    # nn.init.xavier_normal_(self.linear.bias)
  def forward(self,x):
    y_predicted=self.linear(x)
    return y_predicted
model=LinearRegression(11)

criterion = nn.MSELoss()
num_epochs=1000
for epoch in range(num_epochs):
  for x,y in train_data:
    y_pred=model(x)
    loss=criterion(y,y_pred)
    # print(loss)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()

My DataLoader:
class Data(Dataset):
    def __init__(self):
        self.x=x_train
        self.y=y_train
        self.len=self.x.shape[0]
    def __getitem__(self,index):
      return self.x[index],self.y[index]
    def __len__(self):
        return self.len
dataset=Data()
train_data=DataLoader(dataset=dataset,batch_size=1,shuffle=False)

Can someone please tell me why is that happening or is there any faults in my code?

Output:
There were a couple of tweaks necessary to the code. I also had to create data and an optimizer, which you hadn't provided. With the changes below, both methods produce a learning function.
Of course optimal hyperparameters such as alpha or iterations might be different between the two approaches, and you might need to find them separately.
# Create data:
import sklearn
X, Y = sklearn.datasets.load_diabetes(return_X_y=True)
# Adding a random column to match your data shape:
X = np.hstack((X, np.random.randn(X.shape[0], 1)))

iterations = 500

################
# Python version
def yinfer(X, beta):
    return beta[0] + np.dot(X,beta[1:])

def cost(X, Y, beta):
    sum = 0
    m = len(Y)
    for i in range(m):
        sum = sum + ( yinfer(X[i], beta) - Y[i])*(yinfer(X[i], beta) - Y[i])
    return  sum/(1.0*m)

beta = np.array([0,0.04086357 ,-0.02831656  ,0.09622949 ,-0.15162516  ,0.60188454  ,0.47528714,
  -0.6066466  ,-0.22995654 ,-0.58388734  ,0.20954669 ,-0.67851365])
arr_cost = []
m = len(Y)
alpha = 0.1
temp_beta = np.zeros(12)
for i in range(iterations):
    for k in range(m):
        temp_beta[0] =  yinfer(X[k,:], beta) - Y[k]
        temp_beta[1:] = (yinfer(X[k,:], beta) - Y[k])*X[k,:]
        beta = beta - alpha*temp_beta/(1.0*m)
    arr_cost.append(cost(X,Y,beta))

#################
# Pytorch version
from torch import nn
from torch import optim

class LinearRegression(nn.Module):
    def __init__(self,n_input_features):
        super(LinearRegression,self).__init__()
        self.linear=nn.Linear(n_input_features,1)
        self.linear.bias.data.fill_(0.0)
        nn.init.xavier_uniform_(self.linear.weight)
    def forward(self,x):
        y_predicted=self.linear(x)
        return y_predicted

class Data(Dataset):
    def __init__(self, x_train, y_train):
        self.x=x_train
        self.y=y_train
        self.len=self.x.shape[0]
    def __getitem__(self,index):
        return self.x[index],self.y[index]
    def __len__(self):
        return self.len
train_data=DataLoader(dataset=Data(X, Y),batch_size=1,shuffle=False)

criterion = nn.MSELoss()
model=LinearRegression(11)
optimizer = optim.SGD(model.parameters(), lr=0.01)

loss_vals = []  # store results
for epoch in range(iterations):
    for x, y in train_data:
        x, y = x.float(), y.float()
        y_pred=model.forward(x)
        loss=criterion(y, y_pred)
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()
    loss_vals.append(float(loss))

##############
# Plot results
f, ax = plt.subplots(1,1, figsize=(20,5))
ax.plot(range(1, iterations+1), arr_cost, label='python')
ax.plot(range(1, iterations+1), loss_vals, label='torch')
ax.legend(); ax.set_xlabel('epochs'); ax.set_xlabel('loss');


Instruction: Getting an error(cannot import name 'BertPreTrainedModel') while importing classification model from simpletransformers

Input:
Getting the following error while  trying to import the classificationmodel from simpletransformers.

ImportError                               Traceback (most recent call last)
<ipython-input-1-29f08e6c2d87> in <module>()
----> 1 from simpletransformers.classification import ClassificationModel, ClassificationArgs

3 frames
/usr/local/lib/python3.6/dist-packages/simpletransformers/classification/transformer_models/roberta_model.py in <module>()
      2 import torch.nn as nn
      3 from torch.nn import CrossEntropyLoss, MSELoss
----> 4 from transformers.modeling_roberta import (
      5     ROBERTA_PRETRAINED_MODEL_ARCHIVE_LIST,
      6     BertPreTrainedModel,

ImportError: cannot import name 'BertPreTrainedModel'

---------------------------------------------------------------------------

Output: In this github issue the problem was an old version of simpletransformers. To get the latest version do pip install --upgrade simpletransformers. Maybe even do this for the transformers package as well.


Instruction: implement dropout layer using nn.Sequential()

Input:
I am trying to implement a Dropout layer using pytorch as follows:
class DropoutLayer(nn.Module):
    def __init__(self, p):
        super().__init__()
        self.p = p

    def forward(self, input):
        if self.training:
            u1 = (np.random.rand(*input.shape)<self.p) / self.p
            u1 *= u1
            return u1
        else:
            input *= self.p

And then calling a simple NN.sequential:
model = nn.Sequential(nn.Linear(input_size,num_classes), DropoutLayer(.7), nn.Flatten())

opt = torch.optim.Adam(model.parameters(), lr=0.005)
train(model, opt, 5) #train(model, optimizer, epochs #)

But I'm getting the following error:
TypeError: flatten() takes at most 1 argument (2 given)

Not sure what I'm doing wrong.  Still new to pytorch.  Thanks.

Output:
In the forward function of your DropoutLayer, when you enter the elsebranch, there is no return. Therefore the following layer (flatten) will have no input. However, as emphasized in the comments, that's not the actual problem.
The actual problem is that you are passing a numpy array to your Flatten layer. A Minimal code to reproduce the problem would be :
nn.Flatten()(np.random.randn(5,5))
>>> TypeError: flatten() takes at most 1 argument (2 given)

However, I cannot explain why this layer behaves like that on a numpy tensor, the behavior of the flatten function being much more understandable. I don't know what additional operations the layer performs.
torch.flatten(np.random.randn(5,5))
>>> TypeError: flatten(): argument 'input' (position 1) must be Tensor, not numpy.ndarray

Why this error is raised by your code is because in the forward pass, you create a numpy tensor, perform some operations, and return it instead of returning a tensor. If I may, you don't even touch the actual input tensor (in the first branch)