Instruction: What are some ways to speed up data loading on large sparse arrays (~1 million x 1 million, density ~0.0001) in Pytorch?

Input:
I am working on a binary classification problem. I have ~1.5 million data points, and the dimensionality of the feature space is 1 million. This dataset is stored as a sparse array, with a density of ~0.0001. For this post, I'll limit the scope to assume that the model is a shallow feedforward neural network, and also assume that the dimensionality has already been optimized (so cannot be reduced below 1 million). Naiive approaches to create mini-batches out of this data to feed into the network would take a lot of time (As an example, a basic approach of creating a TensorDataset (map style) from a torch.sparse.FloatTensor representation of the input array, and wrapping a DataLoader around it, means ~20s to get a mini-batch of 32 to the network, as opposed to say ~0.1s to perform the actual training). I am looking for ways to speed this up.
What I've tried

I first figured that reading from such a large sparse array in every iteration of the DataLoader was computationally intensive, so I broke down this sparse array into smaller sparse arrays
For the DataLoader to read from these multiple sparse arrays in an iterative fashion, I replaced the map style dataset that I had inside the DataLoader with an IterableDataset, and streamed these smaller sparse arrays into this IterableDataset like so:

from itertools import chain
from scipy import sparse

class SparseIterDataset(torch.utils.data.IterableDataset):

    def __init__(self, fpaths):
        super(SparseIter).__init__()
        self.fpaths = fpaths

    def read_from_file(self, fpath):
        data = sparse.load_npz(fpath).toarray()
        for d in data:
            yield torch.Tensor(d)

    def get_stream(self, fpaths):
        return chain.from_iterable(map(self.read_from_file, fpaths))

    def __iter__(self):
        return self.get_stream(self.fpaths)

With this approach, I was able to bring down the time from the naiive base case of ~20s to ~0.2s per minibatch of 32. However, given that my dataset has ~1.5 million samples, this still implies a lot of time spent on even making one pass through the dataset. (As a comparison, even though it's slightly apples to oranges, running a logistic regression on scikit-learn on the original sparse array takes about ~6s per iteration through the whole dataset. With pytorch, with the approach I just outlined, it would take ~3000s just to load all the minibatches in an epoch)
One thing which I am aware of but yet to try is using multiprocess data loading by setting the num_workers argument in the DataLoader. I believe this has its own catches in the case of iterable style datasets though. Plus even a 10x speedup would still mean ~300s per epoch in loading mini batches. I feel I'm being inordinately slow! Are there any other approaches/improvements/best practices that you could suggest?

Output:
Your dataset in un-sparsified form would be 1.5M x 1M x 1 byte = 1.5TB as uint8, or 1.5M x 1M x 4 byte = 6TB as float32.  Simply reading 6TB from memory to CPU could take 5-10 minutes on a modern CPU (depending on the architecture), and transfer speeds from CPU to GPU would be a bit slower than that (NVIDIA V100 on PCIe has 32GB/s theoretical).
Approaches:

Benchmark everything individually - eg in jupyter
%%timeit data = sparse.load_npz(fpath).toarray()
%%timeit dense = data.toarray() # un-sparsify for comparison
%%timeit t = torch.tensor(data) # probably about the same as the line above

Also print out the shapes and datatypes of everything to make sure they are as expected.  I haven't tried running your code but I am pretty sure that (a) sparse.load_npz is extremely fast and unlikely to be a bottleneck, but (b) torch.tensor(data) produces a dense tensor and is also quite slow here

Use torch.sparse. I think torch sparse tensors can be used as regular tensors in most cases. You'd have to do some data prep to convert from scipy.sparse to torch.sparse:

A sparse tensor is represented as a pair of dense tensors: a tensor of
values and a 2D tensor of indices.  A sparse tensor can be constructed by
providing these two tensors, as well as the size of the sparse tensor

You mention torch.sparse.FloatTensor but I'm pretty sure you're not making sparse tensors in your code - there is no reason to expect those would be constructed simply from passing a scipy.sparse array to a regular tensor constructor, since that's not how they're usually made.
If you figure out a good way to do this, I recommend you post it as a project or git on github, it would be quite useful.

If torch.sparse doesn't work out, think of other ways to either convert the data to dense only on the GPU, or avoid converting it entirely.

See also:
https://towardsdatascience.com/sparse-matrices-in-pytorch-be8ecaccae6
https://github.com/rusty1s/pytorch_sparse


Instruction: Why is the super constructor necessary in PyTorch custom modules?

Input:
Why does super(LR, self).__init__() need to be called in the code below? I get the error "AttributeError: cannot assign module before Module.init() call" otherwise. That error is caused by self.linear = nn.Linear(input_size, output_size).
I don't understand what the connection is between calling super(LR, self).__init__() and being able to assign the nn.Linear object to self.linear. nn.Linear is a separate object which can be assigned to a variable outside of any class, so why does super(LR, self).__init__() need to be called to assign a Linear object to self.linear within the class?
class LR(nn.Module):

    # Constructor
    def __init__(self, input_size, output_size):

        # Inherit from parent
        super(LR, self).__init__()
        self.test = 1
        self.linear = nn.Linear(input_size, output_size)

    # Prediction function
    def forward(self, x):
        out = self.linear(x)
        return out

Output:
When you write self.linear = nn.Linear(...) inside your custom class, you are actually calling the __setattr__ function of your class. It just happens that  when you extend nn.Module, there are a bunch of things that your class is inheriting, and one of them is the __setattr__. As you can see in the implementation (I post only the relevant part below), if nn.Linear is an instance of nn.Module, your class must have an attribute called _modules, otherwise it will throw the AttributeError you got:
def __setattr__(self, name: str, value: Union[Tensor, 'Module']) -> None:
    # [...]
    modules = self.__dict__.get('_modules')
    if isinstance(value, Module):
        if modules is None:
            raise AttributeError("cannot assign module before Module.__init__() call")
        remove_from(self.__dict__, self._parameters, self._buffers, self._non_persistent_buffers_set)
        modules[name] = value

If you take a look at the nn.Module's __init__, you'll see that self._modules is initialized there:
def __init__(self):
    """
    Initializes internal Module state, shared by both nn.Module and ScriptModule.
    """
    torch._C._log_api_usage_once("python.nn_module")

    self.training = True
    self._parameters = OrderedDict()
    self._buffers = OrderedDict()
    self._non_persistent_buffers_set = set()
    self._backward_hooks = OrderedDict()
    self._forward_hooks = OrderedDict()
    self._forward_pre_hooks = OrderedDict()
    self._state_dict_hooks = OrderedDict()
    self._load_state_dict_pre_hooks = OrderedDict()
    self._modules = OrderedDict()                         # <---- here

The same is true for buffers and parameters.


Instruction: How to fix Process finished with exit code -1073741819 (0xC0000005) when using Convolutional layers in pytorch, error on backward()

Input:
When I use Conv1d or Conv2d layers on pytorch, the process is killed unexpectedly. I am getting the error in the following line:
loss.backward()

My set up:

Windows 10
cuda 10.2
cudnn 7.6.5
RTX 2060 Super
Nvidia driver 451.67
Pycharm 2020.04

Error:
Process finished with exit code -1073741819 (0xC0000005)

In comparison, when I replace the conv layer with a dense layer, the problem doesn't occur.
For more comparison, the same project and the same code was run on Ubuntu 20.04 as well, and it worked quite well.

Output:
There seems to be a known bug around this problem that happens with Pytorch on windows, when run on GPU(with CUDA) .
Ensure all params supplied to Conv1d and Conv2d are correct especially padding value. Note that it can have different behaviour with other OS like linux/ubuntu.
And also if you are using Python-3.6 or higher version, it could be this bug. In that case try with Python-3.5


Instruction: Constant loss through epochs

Input:
I code this neural network to make a gaussian regression but I don't understand why my loss doesn't change through epochs. I set the learning rate to 1 to see the loss decreases but it does not. I chose to take 2000 poitns to train my Neural network. I watched several algorithms on this website and I don't really understand why my algorithm do not achieve what I expect.
I have already imported all libraries needed.
Thank you for your help

def f(x):
    return x * np.sin(x) # function to predict

m =2000
X_bis = np.zeros((1,m),dtype = float)
X_bis=np.random.random(m)*10

        ## Create my training,validation and test set

X_train = X_bis[0:600]
X_val = X_bis[600:800]
X_test = X_bis[800:]

y_train = f(X_train)
y_val = f(X_val)
y_test = f(X_test)

mean_X_train = np.mean(X_train)
std_X_train = np.std(X_train)

mean_y_train = np.mean(y_train)
std_y_train =np.std(y_train)

class MyDataset(data.Dataset):

    def __init__(self, data_feature, data_target):
        self.data_feature = data_feature
        self.data_target = data_target

    def __len__(self):
        return len(self.data_feature)

    def __getitem__(self, index):
        X_train_normalized = (self.data_feature[index] - mean_X_train) / std_X_train
        y_train_normalized = (self.data_target[index] - mean_y_train) / std_y_train
        return torch.from_numpy(np.array(X_train_normalized,ndmin=1)).float(), torch.from_numpy(np.array(y_train_normalized, ndmin = 1)).float()

training_set  = MyDataset(X_train,y_train)
train_loading = torch.utils.data.DataLoader(training_set, batch_size= 100)

val_set = MyDataset(X_val, y_val)
val_loading = torch.utils.data.DataLoader(val_set, batch_size= 10)

test_set  = MyDataset(X_test,y_test)
test_loading = torch.utils.data.DataLoader(test_set, batch_size= 100)

class Net(nn.Module):
  def __init__(self):
    super(Net, self).__init__()
    self.FC1 = nn.Linear(1,10)
    self.FC2 = nn.Linear(10, 1)
  def forward(self, x):
    x = F.relu(self.FC1(x))
    x = self.FC2(x)
    return x

model = Net()

criterion = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(),
lr=1, weight_decay= 0.01, momentum = 0.9)

def train(net, train_loader, optimizer, epoch):
    net.train()
    total_loss=0
    for idx,(data, target) in enumerate(train_loader, 0):
        outputs = net(data)
        loss = criterion(outputs,target)
        total_loss +=loss.cpu().item()
        optimizer.step()
    print('Epoch:', epoch , 'average training loss ', total_loss/ len(train_loader))

def test(net,test_loader):
    net.eval()
    total_loss = 0
    for idx,(data, target) in enumerate(test_loader,0):
        outputs = net(data)
        outputs = outputs * std_X_train + mean_X_train
        target = target * std_y_train + mean_y_train
        loss = criterion(outputs,target)
        total_loss += sqrt(loss.cpu().item())
    print('average testing loss', total_loss/len(test_loader))

for epoch in range(50):
    train(model,train_loading,optimizer,epoch)
    test(model,val_loading)

'''

Output: I'm wondering why you don't have loss.backward() after the line that you compute the loss (i.e., loss = criterion(outputs,target)) in your training snippet. This will help backpropagating and ultimately updating the parameters of your network upon optimizer.step(). Also, try using lower learning rates as lr=1 normally is too much in training such networks. Try using learning rates in between 0.001-0.01 to see if your network is learning the mapping between input X and target Y.


Instruction: pytorch when do I need to use `.to(device)` on a model or tensor?

Input:
I am new to Pytorch, but it seems pretty nice. My only question was when to use tensor.to(device) or Module.nn.to(device).
I was reading the documentation on this topic, and it indicates that this method will move the tensor or model to the specified device. But I was not clear for what operations this is necessary, and what kind of errors I will get if I don't use .to() at the right time?
For example, if I just create a tensor, I imagine that the tensor is stored in CPU accessible memory until I move the tensor to the GPU. Once the tensor is on the GPU, then the GPU will execute any mathematical operations on that tensor.
However, do I have to worry about accidentally transferring the data tensor to the GPU while not transferring the model to the GPU? Will this just give me straight errors, or will it engage in a lot of expensive data transfer behind the scenes. This example is easy enough for me to test, but I was just wondering about other cases where it might not be so obvious.
Any guidance would be helpful.

Output:
It is necessary to have both the model, and the data on the same device, either CPU or GPU, for the model to process data. Data on CPU and model on GPU, or vice-versa, will result in a Runtime error.
You can set a variable device to cuda if it's available, else it will be set to cpu, and then transfer data and model to device :
import torch

device = 'cuda' if torch.cuda.is_available() else 'cpu'
model.to(device)
data = data.to(device)


Instruction: Pytorch and Torchvision are compiled different CUDA versions

Input:
RuntimeError: Detected that PyTorch and torchvision were compiled with different CUDA versions. PyTorch has CUDA Version=10.2 and torchvision has CUDA Version=10.1. Please reinstall the torchvision that matches your PyTorch install.

I am trying to run YOLACT on my Google Colab and found this error. Can someone help solve this issue?

Output:
You need to upgrade your torchvision to one compiled with CUDA 10.2:
pip install --upgrade torchvision>=0.6.0

or, if you're using Conda:
conda install pytorch torchvision cudatoolkit=10.2 -c pytorch

Check here the version you should install based on your PyTorch.


Instruction: How to get occurrence count of item in array in-place

Input:
I have a array which I am trying to get weights for. It looks like this:
Target: tensor([ 1, 6, 5, 8, 10, 5, 4, 5, 10, 10, 9, 8, 10, 4, 10, 9])

I need to get the counts of each item in the array. So I would like it to return this:
Class count: [1, 1, 3, 2, 5, 3, 2, 3, 5, 5, 2, 2, 5, 2, 5, 2]

This is my code right now:
class_sample_count = np.unique(y, return_counts=True)[1]
weight = 1. / class_sample_count

print(f'Target: {y}, Class count: {class_sample_count}, Weight: {weight}')

samples_weight = weight[y]
weights = torch.from_numpy(samples_weight)

.. but the class count prints as this:
Class count: [1 2 3 1 2 2 5]

How can I get the counts of each value while keeping the length of the original array?

Output:
Use return_inverse of numpy.unique:
y = [ 1, 6, 5, 8, 10, 5, 4, 5, 10, 10, 9, 8, 10, 4, 10, 9]
uniq, inv, cnt = np.unique(y, return_inverse=True, return_counts=True)
cnt[inv]

Output:
array([1, 1, 3, 2, 5, 3, 2, 3, 5, 5, 2, 2, 5, 2, 5, 2])


Instruction: How to do hyperparameter tuning in Detectron2

Input:
Detectron2
COCO Person Keypoint Detection Baselines with Keypoint R-CNN R50-FPN
How do I do hyperparameter tuning with the model above? Which files do I have to open?
Thanks

Output:
You can use "config" to tune your model. Here is an official tutorial how you can use it (https://detectron2.readthedocs.io/tutorials/configs.html)
And here is the file of all hyperparameters that you can tune (https://github.com/facebookresearch/detectron2/blob/master/detectron2/config/defaults.py)


Instruction: How to implement a Sparse Embedding in Tensorflow 2 like Pytorch Embedding(sparse=True)?

Input:
I have a network that has a lot of items that need to be embedded.
However, in each training batch, only a very small portion of the items will actually be used.
If I use the normal tf.keras.layers.Embedding layer, it will add all the items into the network parameter, thus consuming a lot of memory and decreasing speed in distributed training significantly since in each step all the unused grads are still synchronized.
What I want is, that in each training step only the actually used embedding weights are added into the graph and be computed and synchronized.
Pytorch already provides this functionality with torch.nn.Embedding(sparse=True).
How can I implement this in Tensorflow 2?

Output: My bad... checking tf.GradientTape() tells me that gradient of tf.gather is already a sparse tensor, so this needs no bother.


Instruction: migrating from keras to pytorch

Input:
i'm newly in pytorch
it is the model with bidirectional lstm, is there any body to tel me what is the equivalent of this two different lstm & bi-lstm model?
i try some torch codes but it do not worked.because this code has suitable acc in keras,i want the exact model in torch and i unfortunately can't find it:(
fist_one:
def lstm_model(embedding_size, vocab_size):

    title = layers.Input(shape=(None,), dtype='int32', name='title')
    body = layers.Input(shape=(None,), dtype='int32', name='body')

    embedding = layers.Embedding(
        mask_zero=True,
        input_dim=vocab_size,
        output_dim=embedding_size,
        weights=[w2v_weights],
        trainable=True
    )

    lstm_1 = layers.LSTM(units=80, return_sequences=True)
    lstm_2 = layers.LSTM(units=80, return_sequences=False)
    emb_title = embedding(title)
    print("question embedding shape", emb_title.shape)
    sum_a = lstm_2(lstm_1(emb_title))
    print("q_output shape", sum_a.shape)

    emb_body = embedding(body)
    print("answer embedding shape", emb_body.shape)

    sum_b = lstm_2(lstm_1(emb_body))
    print("a_output shape", sum_a.shape)

    sim = layers.dot([sum_a, sum_b], axes=1, normalize=True)
    print("qa_similarity shape", sim.shape)

    #     sim = layers.Activation(activation='sigmoid')(sim)
    sim_model = models.Model(
        inputs=[title, body],
        outputs=[sim],
    )
    sim_model.compile(loss='mean_squared_error', optimizer='nadam', metrics=['accuracy'])

    embedding_model = models.Model(
        inputs=[title],
        outputs=[sum_a]
    )
    return sim_model, embedding_model

second one:

def bilstm_model(embedding_size, vocab_size):
    title = layers.Input(shape=(None,), dtype='int32', name='title')
    body = layers.Input(shape=(None,), dtype='int32', name='body')

    embedding = layers.Embedding(
        mask_zero=True,
        input_dim=vocab_size,
        output_dim=embedding_size,
        weights=[w2v_weights],
        trainable=True
    )

    lstm_1 = layers.Bidirectional(LSTM(activation='tanh', dropout=0.2, units=100, return_sequences=True))
    lstm_2 = layers.Bidirectional(LSTM(activation='tanh', dropout=0.2, units=100, return_sequences=False))
    sum_a = lstm_2(lstm_1(embedding(title)))
    sum_b = lstm_2(lstm_1(embedding(body)))

    sim = layers.dot([sum_a, sum_b], axes=1, normalize=True)
    #     sim = layers.Activation(activation='sigmoid')(sim)
    sim_model = models.Model(
        inputs=[title, body],
        outputs=[sim],
    )
    sim_model.compile(loss='mean_squared_error', optimizer='adam', metrics=['accuracy'])

    embedding_model = models.Model(
        inputs=[title],
        outputs=[sum_a]
    )
    return sim_model, embedding_model

i;m llokingo for true answer in weeks:(

Output:
class Model(nn.Module):
    def __init__(self, **kwargs):
        super().__init__()

        self.embeddings = nn.Embedding(num_embeddings=kwargs["vocab_size"],
                                      embedding_dim=kwargs["embedding_dim"],
                                      padding_idx=kwargs["pad_idx"])
        self.embeddings.weight.requires_grad = True  # to not refine-tune

        if kwargs["model"] == "lstm":
            self.lstm = nn.LSTM(input_size=kwargs["embedding_dim"],  # input
                                hidden_size=kwargs["lstm_units"],  # output
                                num_layers=kwargs["lstm_layers"],
                                bidirectional=False,
                                batch_first=True)
        if kwargs["model"] == "BiLSTM":
            self.lstm = nn.LSTM(input_size=kwargs["embedding_dim"],  # input
                                hidden_size=kwargs["bilstm_units"],  # output
                                num_layers=kwargs["bilstm_layers"],
                                bidirectional=True,
                                batch_first=True)

        self.dropout = nn.Dropout(kwargs["dropout"])
        self.tanh = F.tanh
        self.dropout = nn.Dropout(kwargs["dropout"])

    def forward(self):
        pass

class LSTM_Model(Model):
    """
    a class to define multiple models
    """
    def __init__(self, **kwargs):
        super().__init__(**kwargs)

    def forward(self, question, answer):
        question_embedding = self.embeddings(question)
        # print("question embedding shape:", question_embedding.shape)
        answer_embedding = self.embeddings(answer)
        # print("answer embedding shape:", answer_embedding.shape)
        q_output, (qhidden, qcell) = self.lstm(question_embedding)
        print("q_output shape:", q_output.shape)
        # print("qhidden shape:", qhidden.shape)
        # print("qcell shape:", qcell.shape)
        a_output, (ahidden, acell) = self.lstm(answer_embedding)
        print("a_output shape:", a_output.shape)
        # print("ahidden shape:", ahidden.shape)
        # print("acell shape:", acell.shape)
        # qa_similary = torch.mm(qhidden[-1], ahidden[-1])
        # qa_similary =torch.matmul((qhidden[-1]), torc.th(ahidden[-1]))
        q_output = q_output[-1]
        q_output = q_output.squeeze()
        a_output = a_output[-1]
        a_output = a_output.squeeze()
        mm = torch.mul((q_output), (a_output))
        mm -= mm.min(1, keepdim=True)[0]
        mm /= mm.max(1, keepdim=True)[0]
        qa_similary =torch.mean(mm, dim=1)
        # print("qa_similary shape:", qa_similary.shape)
        return qa_similary, qhidden

    print("**************************MODEL DEFINE & CREATED!****************************")

is this a true and completely exact implemetation of that keras code for two layer lstm?


Instruction: PyTorch multiprocessing error with Hogwild

Input:
I've encountered a mysterious bug while trying to implement Hogwild with torch.multiprocessing. In particular, one version of the code runs fine, but when I add in a seemingly unrelated bit of code before the multiprocessing step, this somehow causes an error during the multiprocessing step: RuntimeError: Unable to handle autograd's threading in combination with fork-based multiprocessing. See https://github.com/pytorch/pytorch/wiki/Autograd-and-Fork
I reproduced the error in a minimal code sample, pasted below. If I comment out the two lines of code m0 = Model(); train(m0) which carry out a non-parallel training run on a separate model instance, then everything runs fine. I can't figure out how these lines could be causing a problem.
I'm running PyTorch 1.5.1 and Python 3.7.6 on a Linux machine, training on CPU only.
import torch
import torch.multiprocessing as mp
from torch import nn

def train(model):
    opt = torch.optim.Adam(model.parameters(), lr=1e-5)
    for _ in range(10000):
        opt.zero_grad()
        # We train the model to output the value 4 (arbitrarily)
        loss = (model(0) - 4)**2
        loss.backward()
        opt.step()

# Toy model with one parameter tensor of size 3.
# Output is always the sum of the elements in the tensor,
# independent of the input
class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.x = nn.Parameter(torch.ones(3))

    def forward(self, x):
        return torch.sum(self.x)

############################################
# Create a separate Model instance and run
# a non-parallel training run.
# For some reason, this code causes the
# subsequent parallel run to fail.
m0 = Model()
train(m0)
print ('Done with preliminary run')
############################################

num_processes = 2
model = Model()
model.share_memory()
processes = []
for rank in range(num_processes):
    p = mp.Process(target=train, args=(model,))
    p.start()
    processes.append(p)
for p in processes:
    p.join()

print(model.x)

Output:
If you modify your code to create new processes like this:
processes = []
ctx = mp.get_context('spawn')
for rank in range(num_processes):
    p = ctx.Process(target=train, args=(model,))

it seems to run fine (rest of code same as yours, tested on pytorch 1.5.0 / python 3.6 / NVIDIA T4 GPU).
I'm not completely sure what is carried over from the non-parallel run to the parallel run; I tried creating a completely new model for the two runs (with its own class), and/or deleting anything from the original, and/or making sure to delete any tensors and free up memory, and none of that made any difference.
What did make a difference was making sure that .backward() never got called outside of mp.Process() before it was called by a function within mp.Process().  I think what may be carried over is an autograd thread; if the thread exists before multiprocessing with the default fork method it fails, if the thread is created after fork it seems to work okay, and if using spawn it also works okay.
Btw: That's a really interesting question - thank you especially for digesting it to a minimal example!


Instruction: Reward not increasing while training a Bipedal System

Input:
I am completely new to reinforcement learning and this is my first program in practice. I am trying to train the bipedal system in the OpenAI gym environment using the policy gradient algorithm.
However, the reward never changes, either at episode 0 or at episode 1000 and I cannot figure out what is going wrong. Could anyone please help me..
TIA.
The code is as follows:
import torch
import torch.nn as nn
import numpy
from torch.autograd import Variable
import gym

def train(model, name):
    model.train()
    env = gym.make(name)
    env.reset()
    num_episodes = 20000
    max_steps = 10000
    optim = torch.optim.Adam(model.parameters(), lr=1)
    numsteps = []
    rewards = []
    avg_numsteps = []

    min_reward = -1000

    for episode in range(num_episodes):
        state = env.reset()
        probs = []
        rewards = []

        for steps in range(max_steps):
            action, log_prob = model.action(state)
            env.render()
            state, reward, finished, _ = env.step(action.squeeze(0).detach().numpy())
            env.render()
            probs.append(log_prob)
            rewards.append(reward)
            if finished:
                break

        if finished:
            Rewards = []
            for i in range(len(rewards)):
                G = 0
                p = 0
                for reward in rewards[i:]:
                    G = G + 0.9 * p * reward
                    p = p + 1
                Rewards.append(G)

            Rewards = torch.tensor(Rewards)

            discounted_reward = (Rewards - Rewards.mean()) / (Rewards.std() + 1e-9)

            gradients = []
            for log_prob, G in zip(log_prob, discounted_reward):
                gradients.append(-log_prob * G)

            optim.zero_grad()
            policy_gradient = Variable(torch.stack(gradients).sum(), requires_grad=True)
            policy_gradient.backward()
            optim.step()
            numsteps.append(steps)
            avg_numsteps.append(numpy.mean(numsteps[-10:]))
            rewards.append(numpy.sum(rewards))

            print("episode: {}, total reward: {}, average_reward: {}, length: {}\n".format(episode,
                                                                                numpy.sum(rewards),
                                                                                numpy.round(numpy.mean(
                                                                                rewards[-10:]),
                                                                                decimals=3),
                                                                                steps))

        if numpy.sum(rewards) > min_reward:
            torch.save(model.state_dict(), '/home/atharva/policyNet.pth')
            min_reward = numpy.sum(rewards)

def test(model, name):
    env = gym.make(name)
    model.eval()
    state = env.reset()
    with torch.no_grad():
        while True:
            action, log_prob = model(state)
            state, reward, finished, _ = env.step(action.squeeze(0).numpy())
            env.render()
            if finished:
                break

class PolicyNet(nn.Module):
    def __init__(self, inputs, actions, hidden_size):
        super(PolicyNet, self).__init__()
        self.num_actions = actions
        self.layer1 = nn.Linear(inputs, hidden_size)
        self.layer2 = nn.Linear(hidden_size, hidden_size)
        self.layer3 = nn.Linear(hidden_size, 2*hidden_size)
        self.layer4 = nn.Linear(2*hidden_size, hidden_size)
        self.layer5 = nn.Linear(hidden_size, actions)

    def forward(self, x):
        x = self.layer1(x)
        x = nn.functional.relu(x)
        x = self.layer2(x)
        x = nn.functional.relu(x)
        x = self.layer3(x)
        x = nn.functional.relu(x)
        x = self.layer4(x)
        x = nn.functional.relu(x)
        x = self.layer5(x)
        return x

    def action(self, state):
        state = torch.from_numpy(state).float().unsqueeze(0)
        actions = self.forward(Variable(state))
        #prob = numpy.random.choice(self.num_actions, p=numpy.squeeze(actions.detach().numpy()))
        log_prob = torch.log(actions.squeeze(0))
        return actions, log_prob

and the calling is as follows
from REINFORCE import PolicyNet, train

model = PolicyNet(24, 4, 256)
train(model, 'BipedalWalker-v3')

Output:
Your learning rate is way too high! High learning rate may lead your model to never converge (in fact, the loss may diverge). A learning rate that is way too low will make the training process unnecessarily long. There is a trade-off you have to find for yourself.
Tunning your learning rate may have a tremendous impact on the performance of your model. I can recommend you spending some time reading this (quite well written) blog post: https://machinelearningmastery.com/learning-rate-for-deep-learning-neural-networks/
For starters, try a learning rate that is in the range [0.01, 0.00001]. For instance:
optim = torch.optim.Adam(model.parameters(), lr=0.001)


Instruction: Can I create an Upper triangular tensor in pytorch?

Input:
I want to create an upper triangular tensor in pytorch which I want that the lower half of the upper triangular tensor constant zeros. And the lower half of the upper triangular tensor have no grad.
When I use torch.triu() to get upper triangular tensor, the lower half of the upper triangular tensor have grad which means that such "zeros" are not constant.
So how to get an upper triangular tensor and let the lower half of the upper triangular tensor constant zeros?
import torch
a=torch.randn(5,5)
c=torch.randn(1,5)
b=torch.triu(a).requires_grad_()
loss=torch.matmul(c,b)
loss=loss.sum()
loss.backward()
print(b.grad)

Output:
It does appear that torch.triu() gives you an upper triangular matrix and the gradients appear to be correct.
For example, lets say that
x = torch.randn(5, 5, requires_grad=True)

produces
tensor([[ 0.1907, -0.0990,  1.0373,  0.3676, -0.2752],
        [ 1.8987,  1.0265, -0.1133,  0.1476, -3.5617],
        [ 1.2581,  0.2860, -1.9215, -0.7674, -0.1687],
        [ 0.1559, -0.9870, -0.6928,  0.1487,  0.3346],
        [ 0.6317, -0.4915,  1.2506,  0.8678,  0.6367]], requires_grad=True)

Then we can take the upper triangular part of x using
y = torch.triu(x)

and y will then be
tensor([[ 0.1907, -0.0990,  1.0373,  0.3676, -0.2752],
        [ 0.0000,  1.0265, -0.1133,  0.1476, -3.5617],
        [ 0.0000,  0.0000, -1.9215, -0.7674, -0.1687],
        [ 0.0000,  0.0000,  0.0000,  0.1487,  0.3346],
        [ 0.0000,  0.0000,  0.0000,  0.0000,  0.6367]], grad_fn=<TriuBackward>)

The claim that "the lower half of the upper triangular tensor have grad which means that such zeros are not constant" indicates there may be a little confusion about what is expected of the gradients here.
The only implication as far as gradients are concerned is that no matter what we do with y, the lower part of x will not have any impact on the final result since it has been effectively multiplied by a constant zero. Therefore the derivative of any function resulting from y with respect to any of the lower components of x will be zero. We find that this is indeed the case. For example
y.sum().backward()

populates x.grad with the gradient of y.sum() w.r.t. x which is correctly reported as
# x.grad
tensor([[1., 1., 1., 1., 1.],
        [0., 1., 1., 1., 1.],
        [0., 0., 1., 1., 1.],
        [0., 0., 0., 1., 1.],
        [0., 0., 0., 0., 1.]])


Instruction: Function with torch.mm showing error while using torch.optim

Input:
I am kind of a newbie with PyTorch. Please forgive me if the question is childish. I am trying to minimize a function using PyTorch's optim. The function includes matrix multiplication. The details are given below.
First I have a tensor:
Xv.requires_grad_()
XT.requires_grad_()

My Objective Function:
def errorFun(x):
    ax = x[0]
    ay = x[1]
    x0 = x[2]
    y0 = x[3]

    A = torch.tensor([[ax, 0., x0], [0., ay, y0], [0., 0., 1.]], dtype=torch.float64)
    B = torch.tensor([[b11, b12, b13], [b21, b22, b23], [b31, b32, b33]], dtype=torch.float64)
    H = torch.mm(A, B)
    Ps = torch.mm(H, X)
    px = Ps[0,:]
    py = Ps[1,:]
    PX = torch.stack([px, py], dim=0)
    PX.requires_grad_()
    return mseloss(PX, XT)

I am minimizing it:
for ii in range(n_optim_steps):
    optimizer.zero_grad()
    loss = errorFun(params)

    #print('Step # {}, loss: {}'.format(ii, loss.item()))
    loss.backward()
    # Access gradient if necessary
    grad = params.grad.data
    optimizer.step()

But I am getting this error:
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-54-84b874448a25> in <module>()
     77     loss.backward()
     78     # Access gradient if necessary
---> 79     grad = params.grad.data
     80     optimizer.step()
     81

AttributeError: 'NoneType' object has no attribute 'data'

Thanks in advance.

Output:
I am not sure I understand your task. But it seems you are not using pytorch the way it was designed to be used.
There are 5 things you have to have:

Data for training;
A task you are interested in per forming;
A parameterized model (eg a neural network);
A cost function to be minimized;
An optimizer;

Consider the simple example:

Data: vectors containing random numbers;
Task: sum the numbers of the vector;
Model: Linear regressor (i.e: 1 layer neural net)
Cost function: Mean Squared Error
Optimizer: Stochastic Gradient Descent

The implementation:
import torch
import torch.nn as nn
from torch.optim import SGD

input_size = 5
model = nn.Linear(input_size, 1)
opt = SGD(model.parameters(), lr=0.01)
loss_func = nn.MSELoss()

for _ range(100):
    data = torch.rand(batch_size, input_size)
    target = data.sum(dim=1)

    opt.zero_grad()
    pred = model(data)
    loss = loss_func(pred, target)
    loss.backward()
    opt.step()


Instruction: PyTorch: Is it possible to differentiate a matrix?

Input:
How do you differentiate a matrix in PyTorch? I have tried the following but neither work:
Instance 1:
a = torch.tensor([1., 2, 3], requires_grad=True)
b = torch.tensor([4., 5, 6], requires_grad=True)
c = a*b
c.backward()
#print(b.grad)

>>> RuntimeError: grad can be implicitly created only for scalar outputs

Instance 2:
a = torch.tensor([1., 2, 3], requires_grad=True)
b = torch.tensor([4., 5, 6], requires_grad=True)

c = a*b
print(b.grad)

>>> None

Output:
It is possible but it doesn't really fit into the standard use case of PyTorch where you are generally interested in the gradient of a scalar valued function.
The derivative of a matrix Y w.r.t. a matrix X can be represented as a Generalized Jacobian. For the case where both matrices are just vectors this reduces to the standard Jacobian matrix, where each row of the Jacobian is the transpose of the gradient of one element of Y with respect to X. More generally if X is shape (n1, n2, ..., nD) and Y is shape (m1, m2, ..., mE) then a natural way to represent the Generalized Jacobian of Y with respect to X is as a tensor of shape (m1, m2, ..., mE, n1, n2, ..., nD).
There are two ways to compute the Generalized Jacobian that I'm aware of in PyTorch.
Option 1
Repeated application of back-propagation on each element of Y.
import torch

def construct_jacobian(y, x, retain_graph=False):
    x_grads = []
    for idx, y_element in enumerate(y.flatten()):
        if x.grad is not None:
            x.grad.zero_()
        # if specified set retain_graph=False on last iteration to clean up
        y_element.backward(retain_graph=retain_graph or idx < y.numel() - 1)
        x_grads.append(x.grad.clone())
    return torch.stack(x_grads).reshape(*y.shape, *x.shape)

then the Jacobian for your test case may be computed using
a = torch.tensor([1., 2., 3.])
b = torch.tensor([4., 5., 6.], requires_grad=True)
c = a * b

jacobian = construct_jacobian(c, b)

print(jacobian)

which results in
tensor([[1., 0., 0.],
        [0., 2., 0.],
        [0., 0., 3.]])

Option 2
In PyTorch 1.5.1 a new autograd.functional API was introduced, including the new function torch.autograd.functional.jacobian. This produces the same results as the previous example but takes a function as an argument. Not demonstrated here, but you can provide the jacobian function a list of inputs if your function takes multiple independent tensors as input. In that case the jacobian would return a tuple containing the Generalized Jacobian for each of the input arguments.
import torch

a = torch.tensor([1., 2., 3.])

def my_fun(b):
    return a * b

b = torch.tensor([4., 5., 6.], requires_grad=True)

jacobian = torch.autograd.functional.jacobian(my_fun, b)

print(jacobian)

which also produces
tensor([[1., 0., 0.],
        [0., 2., 0.],
        [0., 0., 3.]])

As an aside, in some literature the term "gradient" is used to refer to the transpose of the Jacobian matrix. If that's what you're after then, assuming Y and X are vectors, you can simply use the code above and take the transpose of the resulting Jacobian matrix. If Y or X are higher order tensors (matrices or n-dimensional tensors) then I'm not aware of any literature that distinguishes between gradient and Generalized Jacobian. A natural way to represent such a "transpose" of the Generalized Jacobian would be to use Tensor.permute to turn it into a tensor of shape (n1, n2, ..., nD, m1, m2, ..., mE).

As another aside, the concept of the Generalized Jacobian is rarely used in literature (example usage) but is actually relatively useful in practice. This is because it basically works as a bookkeeping technique to keep track of the original dimensionality of Y and X. By this I mean you could just as easily take Y and X and flatten them into vectors, regardless of their original shape. Then the derivative would be a standard Jacobian matrix. Consequently this Jacobian matrix would be equivalent to a reshaped version of the Generalized Jacobian.


Instruction: Pytorch schedule learning rate

Input:
I am trying to re-implement one paper, which suggests to adjust the learning rate as below:

The learning rate is decreased by a factor of the regression value with patience epochs 10 on the change value of 0.0001.

Should I use the torch.optim.lr_scheduler.ReduceLROnPlateau()?
I am not sure what value should I pass to each parameter.

Is the change value in the statement denotes to the parameter threshold?

Is the factor in the statement denotes to the parameter factor?

Output:
torch.optim.lr_scheduler.ReduceLROnPlateau is indeed what you are looking for. I summarized all of the important stuff for you.
mode=min: lr will be reduced when the quantity monitored has stopped decreasing
factor: factor by which the learning rate will be reduced
patience: number of epochs with no improvement after which learning rate will be reduced
threshold: threshold for measuring the new optimum, to only focus on significant changes (change value). Say we have threshold=0.0001, if loss is 18.0 on epoch n and loss is 17.9999 on epoch n+1 then we have met our criteria to multiply the current learning rate by the factor.
criterion = torch.nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode='min',
    factor=0.1, patience=10, threshold=0.0001, threshold_mode='abs')

for epoch in range(20):
    # training loop stuff
    loss = criterion(...)
    scheduler.step(loss)

You can check more details in the documentation: https://pytorch.org/docs/stable/optim.html#torch.optim.lr_scheduler.ReduceLROnPlateau


Instruction: Does Pytorch-Lightning have a multiprocessing (or Joblib) module?

Input:
I have been googling around but can't seem to find if there is a multiprocessing module available in Pytorch-Lightning, just like how Pytorch has a torch.multiprocessing module.
Does anyone know if Pytorch-Lightning has this (or a Joblib similar) module? I am looking for a Pytorch-Lightning module which allows me to parallelize over multiple GPUs
Many thanks in advance.
Edit: To be more specific, I am looking for a multiprocessing module in Pytorch-Lightning which allows me to parallelize over multiple GPUs on non-neural network computations, such as:
import numpy as np
import torch
from torch.multiprocessing import Pool

X = np.array([[1, 3, 2, 3], [2, 3, 5, 6], [1, 2, 3, 4]])
X = torch.DoubleTensor(X)

def X_power_func(j):
    X_power = X.cuda()**j
    return X_power

if __name__ == '__main__':
  with Pool(processes = 2) as p:   # Parallelizing over 2 GPUs
    results = p.map(X_power_func, range(4))

results

Output:
Yes, basically all you have to do is to provide Trainer with appropriate argument gpus=N and specify backend:
# train on 8 GPUs (same machine (ie: node))
trainer = Trainer(gpus=8, distributed_backend='ddp')

# train on 32 GPUs (4 nodes)
trainer = Trainer(gpus=8, distributed_backend='ddp', num_nodes=4)

You can read more about it in multi-GPU training documentation.
EDIT:
What you were actually looking for is distributed module instead of multiprocessing, torch.distributed.DistributedDataParallel is usually recommended for parallelizing over multiple GPUs.


Instruction: OSError: [WinError 127] The specified procedure could not be found. pytorch

Input:
I get this error OSError: [WinError 127] The specified procedure could not be found.
OSError                                   Traceback (most recent call last)
 in
      5 import matplotlib.pyplot as plt
      6
----> 7 import torch
      8 import torch.nn as nn
      9 import torch.optim as optim

~\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\__init__.py in
     79     dlls = glob.glob(os.path.join(th_dll_path, '*.dll'))
     80     for dll in dlls:
---> 81         ctypes.CDLL(dll)
     82
     83

~\AppData\Local\Programs\Python\Python38\lib\ctypes\__init__.py in __init__(self, name, mode, handle, use_errno, use_last_error, winmode)
    371
    372         if handle is None:
--> 373             self._handle = _dlopen(self._name, mode)
    374         else:
    375             self._handle = handle

OSError: [WinError 127] La procédure spécifiée est introuvable

I installed pytorch a few days ago and hit this error for the first time today ... so I don't think it's an installation problem...
I also tried to reinstall pytorch and still get this error
I am using torch 1.5.1+cu101 and python 3.8.5 on windows 10
I have only one version of python installed.
if someone could please help me on this
thanks a lot

Output:
I got a similar error with Python 3.9.2 on Windows 10 and solved it doing the following:
I copied the CUDA 11 drivers (specifically cublas64_11.dll) from:
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.2\bin to
C:\<your python directory>\Lib\site-packages\torch\lib


Instruction: PyTorch: Running unseen text through generated model

Input:
I am trying to implement a PyTorch project, found here.
import os
from process_file import process_doc
import random
import torch
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
import numpy as np
from argparse import ArgumentDefaultsHelpFormatter, ArgumentParser
from sklearn.metrics import precision_recall_fscore_support
from sklearn.metrics import classification_report, confusion_matrix
from full_model import Classifier
import torch.nn as nn
import torch.optim as optim
import time

def get_batch(doc, ref_type='headline'):
    sent, ls, out, sids = [], [], [], []
    sent.append(doc.headline)
    ls.append(len(doc.headline))
    for sid in doc.sentences:
        if SPEECH:
            out.append(out_map[doc.sent_to_speech.get(sid, 'NA')])
        else:
            out.append(out_map[doc.sent_to_event.get(sid)])
        sent.append(doc.sentences[sid])
        ls.append(len(doc.sentences[sid]))
        sids.append(sid)
    ls = torch.LongTensor(ls)
    out = torch.LongTensor(out)
    return sent, ls, out, sids

def train(epoch, data):
    start_time = time.time()
    total_loss = 0
    global prev_best_macro

    for ind, doc in enumerate(data):
        model.train()
        optimizer.zero_grad()
        sent, ls, out, _ = get_batch(doc)
        if has_cuda:
            ls = ls.cuda()
            out = out.cuda()

        _output, _, _, _ = model.forward(sent, ls)
        loss = criterion(_output, out)
        total_loss += loss.item()
        loss.backward()
        optimizer.step()
        del sent, ls, out
        if has_cuda:
            torch.cuda.empty_cache()

    print("--Training--\nEpoch: ", epoch, "Loss: ", total_loss, "Time Elapsed: ", time.time()-start_time)
    perf = evaluate(validate_data)
    # print(perf)
    if prev_best_macro < perf:
        prev_best_macro = perf
        print ("-------------------Test start-----------------------")
        _ = evaluate(test_data, True)
        print ("-------------------Test end-----------------------")
        torch.save(model.state_dict(), 'discourse_lstm_model.pt')

def evaluate(data, is_test=False):
    y_true, y_pred = [], []
    model.eval()
    for doc in data:
        sent, ls, out, sids = get_batch(doc)
        if has_cuda:
            ls = ls.cuda()
            #out = out.cuda()

        _output, _, _, _ = model.forward(sent, ls)
        _output = _output.squeeze()
        _, predict = torch.max(_output, 1)
        y_pred += list(predict.cpu().numpy() if has_cuda else predict.numpy())
        temp_true = list(out.numpy())
        y_true += temp_true

    print("MACRO: ", precision_recall_fscore_support(y_true, y_pred, average='macro'))
    print("MICRO: ", precision_recall_fscore_support(y_true, y_pred, average='micro'))
    if is_test:
        print("Classification Report \n", classification_report(y_true, y_pred))
    print("Confusion Matrix \n", confusion_matrix(y_true, y_pred))
    return precision_recall_fscore_support(y_true, y_pred, average='macro')[2]

if __name__ == '__main__':

    parser = ArgumentParser(formatter_class=ArgumentDefaultsHelpFormatter)
    # parser.add_argument('--drop', help='DROP', default=6, type=float)
    # parser.add_argument('--learn_rate', help='LEARNING RATE', default=0, type=float)
    # parser.add_argument('--loss_wt', help='LOSS WEIGHTS', default=0, type=str)
    parser.add_argument('--seed', help='SEED', default=0, type=int)

    args = parser.parse_args()

    has_cuda = torch.cuda.is_available()

    SPEECH = 0
    if SPEECH:
        out_map = {'NA':0, 'Speech':1}
    else:
        out_map = {'NA':0,'Main':1,'Main_Consequence':2, 'Cause_Specific':3, 'Cause_General':4, 'Distant_Historical':5,
        'Distant_Anecdotal':6, 'Distant_Evaluation':7, 'Distant_Expectations_Consequences':8}

    train_data = []
    validate_data = []
    test_data = []
    for domain in ["Business", "Politics", "Crime", "Disaster", "kbp"]:
        subdir = "../data/train/"+domain
        files = os.listdir(subdir)
        for file in files:
            if '.txt' in file:
                doc = process_doc(os.path.join(subdir, file), domain) #'../data/Business/nyt_corpus_data_2007_04_27_1843240.txt'
                #print(doc.sent_to_event)
                train_data.append(doc)
        subdir = "../data/test/"+domain
        files = os.listdir(subdir)
        for file in files:
            if '.txt' in file:
                doc = process_doc(os.path.join(subdir, file), domain) #'../data/Business/nyt_corpus_data_2007_04_27_1843240.txt'
                #print(doc.sent_to_event)
                test_data.append(doc)

    subdir = "../data/validation"
    files = os.listdir(subdir)
    for file in files:
        if '.txt' in file:
            doc = process_doc(os.path.join(subdir, file), 'VAL') #'../data/Business/nyt_corpus_data_2007_04_27_1843240.txt'
            #print(doc.sent_to_event)
            validate_data.append(doc)
    print(len(train_data), len(validate_data), len(test_data))

    seed = args.seed
    np.random.seed(seed)
    torch.manual_seed(seed)
    if has_cuda:
        torch.cuda.manual_seed(seed)
    random.seed(seed)
    np.random.seed(seed)

    prev_best_macro = 0.

    model = Classifier({'num_layers': 1, 'hidden_dim': 512, 'bidirectional': True, 'embedding_dim': 1024,
                        'dropout': 0.5, 'out_dim': len(out_map)})

    if has_cuda:
        model = model.cuda()
    model.init_weights()

    criterion = nn.CrossEntropyLoss()

    print("Model Created")

    params = filter(lambda p: p.requires_grad, model.parameters())
    optimizer = optim.Adam(params, lr=5e-5, betas=[0.9, 0.999], eps=1e-8, weight_decay=0)

    try:
        for epoch in range(15):
            print("---------------------------Started Training Epoch = {0}--------------------------".format(epoch+1))
            train(epoch, train_data)

    except KeyboardInterrupt:
        print ("----------------- INTERRUPTED -----------------")
        evaluate(validate_data)
        evaluate(test_data)

Running this code, I have successfully outputted a .pt model trained on a corpus of ~400 articles, each article annotated according to its sectional content (data from the Github repo).
Now, I want to annotate a new, unseen article using this model, but I can't quite figure out how to do so. I have a feeling the classification code is already implemented in the snippet above and I'd really appreciate any help/guidance as to how I could go about classifying an unseen article using this code. Thanks so much in advance!

Output:
Well, your code does training and testing already. You just need an extra piece of code that loads the trained model and test data for inference.
It should be something like that:
# load trained model:
model = Classifier({'num_layers': 1, 'hidden_dim': 512, 'bidirectional': True, 'embedding_dim': 1024,
                        'dropout': 0.5, 'out_dim': len(out_map)})

model.load_state_dict(torch.load("PATH/TO/SAVED/MODEL.pt")

# load data:
subdir = "PATH/TO/DOCS/"
files = os.listdir(subdir)
validate_data = []
for file in files:
    if '.txt' in file:
        doc = process_doc(os.path.join(subdir, file), 'VAL')
        validate_data.append(doc)
print(len(train_data), len(validate_data), len(test_data))

# use the evaluate function with is_test=True for inference.
evaluate(data, is_test=True)


Instruction: why does batch normalization make my batches so abnormal?

Input:
I'm playing with pytorch for the first time, and I've noticed that when training my neural net, about one time in four or so the loss takes a left turn towards infinity, then nan shortly after that. I've seen a few other questions about nan-ing, but the recommendations there seem to be essentially to do normalization; but the first layer in my net below is such a normalization, and I still see this problem! The full net is a bit convoluted, but I've done some debugging to try to produce a very small, understandable net that still displays the same issue.
The code is below; it consists of sixteen inputs, 0-1, which are passed through a batch normalization and then a fully-connected layer to a single output. I'd like it to learn the function that always outputs 1, so I take the squared error from 1 for the loss.
import torch as t
import torch.nn as tn
import torch.optim as to

if __name__ == '__main__':
    board = t.rand([1,1,1,16])
    net = tn.Sequential \
        ( tn.BatchNorm2d(1)
        , tn.Conv2d(1, 1, [1,16])
        )
    optimizer = to.SGD(net.parameters(), lr=0.1)
    for i in range(10):
        net.zero_grad()
        nn_outputs = net.forward(board)
        loss = t.sum((nn_outputs - 1)**2)
        print(i, nn_outputs, loss)
        loss.backward()
        optimizer.step()

If you run it a few times, eventually you'll see a run that looks like this:
0 tensor([[[[-0.7594]]]], grad_fn=<MkldnnConvolutionBackward>) tensor(3.0953, grad_fn=<SumBackward0>)
1 tensor([[[[4.0954]]]], grad_fn=<MkldnnConvolutionBackward>) tensor(9.5812, grad_fn=<SumBackward0>)
2 tensor([[[[5.5210]]]], grad_fn=<MkldnnConvolutionBackward>) tensor(20.4391, grad_fn=<SumBackward0>)
3 tensor([[[[-3.4042]]]], grad_fn=<MkldnnConvolutionBackward>) tensor(19.3966, grad_fn=<SumBackward0>)
4 tensor([[[[823.6523]]]], grad_fn=<MkldnnConvolutionBackward>) tensor(676756.7500, grad_fn=<SumBackward0>)
5 tensor([[[[3.5471e+08]]]], grad_fn=<MkldnnConvolutionBackward>) tensor(1.2582e+17, grad_fn=<SumBackward0>)
6 tensor([[[[2.8560e+25]]]], grad_fn=<MkldnnConvolutionBackward>) tensor(inf, grad_fn=<SumBackward0>)
7 tensor([[[[inf]]]], grad_fn=<MkldnnConvolutionBackward>) tensor(inf, grad_fn=<SumBackward0>)
8 tensor([[[[nan]]]], grad_fn=<MkldnnConvolutionBackward>) tensor(nan, grad_fn=<SumBackward0>)
9 tensor([[[[nan]]]], grad_fn=<MkldnnConvolutionBackward>) tensor(nan, grad_fn=<SumBackward0>)

Why does my loss go to nan, and what can I do about it?

Output:
Welcome to pytorch!
Here is how I would set up your training. Please check the comments.
# how the comunity usually does the import:
import torch  # some people do: import torch as th
import torch.nn as nn
import torch.optim as optim

if __name__ == '__main__':

    # setting some parameters:
    batch_size = 32
    n_dims = 128

    # select GPU if available
    device = 'cuda' if torch.cuda.is_available() else 'cpu'

    # initializing a simple neural net
    net = nn.Sequential(nn.Linear(n_dims, n_dims  // 2), # Batch norm is not usually used directly on the input
                        nn.BachNorm1d(n_dims  // 2), # Batch norm is used before the activation function (it centers the input and helps make the dims of the previous layers independent of each other)
                        nn.ReLU(), # the most common activation function
                        nn.Linear(n_dims  // 2, 1)  # final layer)
    net.to(device) # model is copied to the GPU if it is availalbe

    optimizer = to.SGD(net.parameters(), lr=0.01) # it is better to start with a low lr and increase it at later experiments to avoid training divergence, the range [1.e-6, 5.e-2] is recommended.

    for i in range(10):
       # generating random data:
       board = torch.rand([batch_size, n_dims])
       # for sequences: [batch_size, channels, L]
       # for image data: [batch_size, channels, W, H]
       # for videos: [batch_size, chanels, L, W, H]
       boad = board.to(device) # data is copied to the gpu if it is available

        optimizer.zero_grad()  # the convension the comunity uses, though the result is the same as net.zero_grad()

        nn_outputs = net(board) # don't call net.forward(x), call net(x). Pytorch applies some hooks in the net.__call__(x) that are useful for backpropagation.

        loss = ((nn_outputs - 1)**2).mean() # using .mean() makes your training less sensitive to the batch size.

        print(i, nn_outputs, loss.item())

        loss.backward()
        optimizer.step()

One comment about the batch norm. Per dimension, it calculates the mean and the standard deviation of your batch (check the documentation https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html#torch.nn.BatchNorm2d):
x_normalized = (x.mean(dim=0) / (x.std(dim=0) + e-6)) * scale + shift

Where scale and shift are learnable parameters. If you only give one example per batch, x.std(0) = 0 will make x_normalized contain very very large values.


Instruction: get the index of element in NumPy array

Input:
I have a Numpy integer array with a lot of duplicate elements.
For example:
a = np.random.randint(0,5,20)
a
Out[23]:
array([3, 1, 2, 4, 1, 2, 4, 3, 2, 3, 1, 4, 4, 1, 2, 4, 2, 4, 1, 1])

There are two cases:

if one element is less than 4, get all the indexes of this element
if one element is great than or equal to 4, select four of them randomly

I solved this with a loop.
ans = np.array([])
num = 4
for i in range(1,5):
    indexes = np.where(a == i)[0] # all indexes of elements equal to i
    index_i = np.random.choice(indexes, num, False) if len(indexes) >=num else indexes
    ans = np.concatenate([ans, index_i])

np.sort(ans)

Out[57]:
array([ 0.,  1.,  2.,  5.,  6.,  7.,  8.,  9., 10., 11., 13., 14., 15.,
       17., 19.])

Can I solve this problem without a loop or more efficiently in Numpy or PyTorch?

Output:
You can do it quite easily, using Pandas.
First convert your array to a pandasonic Series:
s = pd.Series(a)

Then:

Group it by its value.
Apply to each group a function, which:

for groups of size 4 or smaller returns just this group,
for groups with more members, returns a random sample of 4 elements
from them.

Drop the 0-th level of the resulting index (added during grouping).
Sort by the (original) index, to bring back the original order (without
the dropped elements, for now we have original values with their
corresponding indices).
Return the index of the above result, as a Numpy array.

The code to do it is:
s.groupby(s).apply(lambda grp: grp if grp.size <= 4 else grp.sample(4))\
    .reset_index(level=0, drop=True).sort_index().index.values

For a sample array containg:
array([2, 2, 1, 0, 1, 0, 2, 2, 2, 3, 0, 2, 1, 0, 0, 3, 3, 0, 2, 4])

the result is:
array([ 0,  2,  4,  5,  7,  9, 10, 11, 12, 14, 15, 16, 17, 18, 19])

To show that this result is correct, I repeated the source array,
with "x" marks below the elements at the returned indices.
array([2, 2, 1, 0, 1, 0, 2, 2, 2, 3, 0, 2, 1, 0, 0, 3, 3, 0, 2, 4])
       x     x     x  x     x     x  x  x  x     x  x  x  x  x  x


Instruction: How to concat two tensors of size [B,C,13,18] and [B,C,14,18] respectively in Pytorch?

Input:
I often met this problem when the height or width of an image or a tensor becomes odd.
For example, suppose the original tensor is of size [B,C,13,18]. After forwarding a strided-2 conv and several other conv layers, its size will become [B,C,7,9]. If we upsample the output by 2 and concat it with the original feature map as most cases, the error occurs.
I found that in many source codes, they use even sizes like (512,512) for training, so this kind of problem won't happen. But for test, I use the original image size to keep fine details and often met this problem.
What should I do? Do I need to change the network architecture?

Output:
Concatenating tensors with incompatible shapes does not make sense. Information is missing, and you need to specify it by yourself. The question is, what do you are expected from this concatenation ? Usually, you pad the input with zeros, or truncate the output, in order to get compatible shapes (in the general case, being even is not the required condition). If the height and width are large enough, the edge effect should be negligible (well, except perhaps on the edge, it depends).
So if you are dealing with convolutions only, no need to change the architecture strictly speaking, just to add a padding layer somewhere it seems appropriate.


Instruction: How to make sure PyTorch has deallocated GPU memory?

Input:
Say we have a function like this:
        def trn_l(totall_lc, totall_lw, totall_li, totall_lr):
            self.model_large.cuda()
            self.model_large.train()
            self.optimizer_large.zero_grad()

            for fb in range(self.fake_batch):
                val_x, val_y = next(self.valid_loader)
                val_x, val_y = val_x.cuda(), val_y.cuda()

                logits_main, emsemble_logits_main = self.model_large(val_x)
                cel = self.criterion(logits_main, val_y)
                loss_weight = cel / (self.fake_batch)
                loss_weight.backward(retain_graph=False)
                cel = cel.cpu().detach()
                emsemble_logits_main = emsemble_logits_main.cpu().detach()

                totall_lw += float(loss_weight.item())
                val_x = val_x.cpu().detach()
                val_y = val_y.cpu().detach()

            loss_weight = loss_weight.cpu().detach()
            self._clip_grad_norm(self.model_large)
            self.optimizer_large.step()
            self.model_large.train(mode=False)
            self.model_large = self.model_large.cpu()
            return totall_lc, totall_lw, totall_li, totall_lr

On the first call, it allocates 8GB of GPU memory. On the next call, no new memory gets allocated, yet 8GBs are still occupied. I want to have after it is called and the produced first result to have 0 allocated GPU memory or as low as possible.
What I have tried: do  retain_graph=False and .cpu().detach() everywhere - no positive effects.
Memory snapshot before
|===========================================================================|
|                  PyTorch CUDA memory summary, device ID 0                 |
|---------------------------------------------------------------------------|
|            CUDA OOMs: 0            |        cudaMalloc retries: 0         |
|===========================================================================|
|        Metric         | Cur Usage  | Peak Usage | Tot Alloc  | Tot Freed  |
|---------------------------------------------------------------------------|
| Allocated memory      |   33100 KB |   33219 KB |   40555 KB |    7455 KB |
|       from large pool |    3072 KB |    3072 KB |    3072 KB |       0 KB |
|       from small pool |   30028 KB |   30147 KB |   37483 KB |    7455 KB |
|---------------------------------------------------------------------------|
| Active memory         |   33100 KB |   33219 KB |   40555 KB |    7455 KB |
|       from large pool |    3072 KB |    3072 KB |    3072 KB |       0 KB |
|       from small pool |   30028 KB |   30147 KB |   37483 KB |    7455 KB |
|---------------------------------------------------------------------------|
| GPU reserved memory   |   51200 KB |   51200 KB |   51200 KB |       0 B  |
|       from large pool |   20480 KB |   20480 KB |   20480 KB |       0 B  |
|       from small pool |   30720 KB |   30720 KB |   30720 KB |       0 B  |
|---------------------------------------------------------------------------|
| Non-releasable memory |   18100 KB |   20926 KB |   56892 KB |   38792 KB |
|       from large pool |   17408 KB |   18944 KB |   18944 KB |    1536 KB |
|       from small pool |     692 KB |    2047 KB |   37948 KB |   37256 KB |
|---------------------------------------------------------------------------|
| Allocations           |   12281    |   12414    |   12912    |     631    |
|       from large pool |       2    |       2    |       2    |       0    |
|       from small pool |   12279    |   12412    |   12910    |     631    |
|---------------------------------------------------------------------------|
| Active allocs         |   12281    |   12414    |   12912    |     631    |
|       from large pool |       2    |       2    |       2    |       0    |
|       from small pool |   12279    |   12412    |   12910    |     631    |
|---------------------------------------------------------------------------|
| GPU reserved segments |      16    |      16    |      16    |       0    |
|       from large pool |       1    |       1    |       1    |       0    |
|       from small pool |      15    |      15    |      15    |       0    |
|---------------------------------------------------------------------------|
| Non-releasable allocs |       3    |      30    |     262    |     259    |
|       from large pool |       1    |       1    |       1    |       0    |
|       from small pool |       2    |      29    |     261    |     259    |
|===========================================================================|

And after calliing function and
torch.cuda.empty_cache()
torch.cuda.synchronize()

We get:
|===========================================================================|
|                  PyTorch CUDA memory summary, device ID 0                 |
|---------------------------------------------------------------------------|
|            CUDA OOMs: 0            |        cudaMalloc retries: 0         |
|===========================================================================|
|        Metric         | Cur Usage  | Peak Usage | Tot Alloc  | Tot Freed  |
|---------------------------------------------------------------------------|
| Allocated memory      |   10957 KB |    8626 MB |  272815 MB |  272804 MB |
|       from large pool |       0 KB |    8596 MB |  272477 MB |  272477 MB |
|       from small pool |   10957 KB |      33 MB |     337 MB |     327 MB |
|---------------------------------------------------------------------------|
| Active memory         |   10957 KB |    8626 MB |  272815 MB |  272804 MB |
|       from large pool |       0 KB |    8596 MB |  272477 MB |  272477 MB |
|       from small pool |   10957 KB |      33 MB |     337 MB |     327 MB |
|---------------------------------------------------------------------------|
| GPU reserved memory   |    8818 MB |    9906 MB |   19618 MB |   10800 MB |
|       from large pool |    8784 MB |    9874 MB |   19584 MB |   10800 MB |
|       from small pool |      34 MB |      34 MB |      34 MB |       0 MB |
|---------------------------------------------------------------------------|
| Non-releasable memory |    5427 KB |    3850 MB |  207855 MB |  207850 MB |
|       from large pool |       0 KB |    3850 MB |  207494 MB |  207494 MB |
|       from small pool |    5427 KB |       5 MB |     360 MB |     355 MB |
|---------------------------------------------------------------------------|
| Allocations           |    3853    |   13391    |   34339    |   30486    |
|       from large pool |       0    |     557    |   12392    |   12392    |
|       from small pool |    3853    |   12838    |   21947    |   18094    |
|---------------------------------------------------------------------------|
| Active allocs         |    3853    |   13391    |   34339    |   30486    |
|       from large pool |       0    |     557    |   12392    |   12392    |
|       from small pool |    3853    |   12838    |   21947    |   18094    |
|---------------------------------------------------------------------------|
| GPU reserved segments |     226    |     226    |     410    |     184    |
|       from large pool |     209    |     209    |     393    |     184    |
|       from small pool |      17    |      17    |      17    |       0    |
|---------------------------------------------------------------------------|
| Non-releasable allocs |      46    |     358    |   12284    |   12238    |
|       from large pool |       0    |     212    |    7845    |    7845    |
|       from small pool |      46    |     279    |    4439    |    4393    |
|===========================================================================|

Output:
I don't think the other answer is correct. Allocation and deallocation definitely happens during runtime, the thing to note is that the CPU code runs asynchronously from the GPU code, so you need to wait for any deallocation to happen if you want to reserve more memory after it. Take a look at this:
import torch

a = torch.zeros(100,100,100).cuda()

print(torch.cuda.memory_allocated())

del a
torch.cuda.synchronize()
print(torch.cuda.memory_allocated())

Outputs
4000256
0

So you should del the tensors you don't need and call torch.cuda.synchronize() to make sure that the deallocation goes through before your CPU code continues to run.
In your specific case, after your function trn_l returns, any variables that were local to that function, and do not have references elsewhere, will be deallocated along with the corresponding GPU tensors. All you need to do is wait for this to happen by calling torch.cuda.synchronize() after the function call.


Instruction: Is there an equivalent of pytorch.nn.functional.unfold() in keras or tensorflow?

Input:
I want to perform a similar operation in keras. However, I am unable to do the unfold operation in keras. I tried it with conv1D layer, but unable to figure out. Any help would be appreciated
'''
import numpy as np
import torch

x = torch.tensor(np.random.rand(25,100,24))   # tensor of shape (batch_size, seq_length,feature_dim)
x = x.unsqueeze(1) # shape=(25,1,100,24)

import torch.nn.functional as F
x = F.unfold(x,(5, 24), stride=(1,24),dilation=(1,1)) #shape (25,120,96)

'''

Output:
I don't think there is. But you can do one thing. Use tensorly for unfolding. Make a function that unfolds the input array. Then using that funtion make a lambda layer in keras or tf2.0 . Suppose you have input  array X :
 X = np.array([[[ 0,  1],
                   [ 2,  3],
                   [ 4,  5],
                   [ 6,  7]],

                  [[ 8,  9],
                   [10, 11],
                   [12, 13],
                   [14, 15]],

                  [[16, 17],
                   [18, 19],
                   [20, 21],
                   [22, 23]]])

To unfold a tensor, simply use the unfold function from TensorLy:

> from tensorly import unfold unfold(X, 0)
>> array([[ 0,  1,  2,  3,  4,  5,  6,  7],
       [ 8,  9, 10, 11, 12, 13, 14, 15],
       [16, 17, 18, 19, 20, 21, 22, 23]])

Now create a function that takes input array and returns unfolded
array

def unfold(X):
        return unfold(X, 0)

Now use this function as a layer in keras

from keras.layers import Lambda
from keras.models import Sequential

model = Sequential()
model.add(....some_layer....)
model.add(....anotenter code hereher_layer....)
model.add(Lambda(unfold))   <<<<=== using our unfold function as keras layer
model.add(...more_layers..)

Hope this will help !


Instruction: What should be the size of input image for training a YOLOv3 Model Architecture CNN.?

Input:
I've implemented a YOLOv3 from scratch and I plan to fine-tune using MS-COCO weights for some different data.
The dataset I've chosen has images of 720*1280 size.
When I go through the YOLOv3 paper, 1st CONV2d layer is there with filter_size =3 and stride = 1, and output size is 256*256....
Can someone give me a walkthrough for how YOLO training part works in here?

Output:
From Yolov3 paper:

If best possible accuracy/mAP is what you want then use 608 x 608 as input layer size in the config.
If you want good inference/speed at the cost of accuracy then use, 320 x 320
If balanced model is what you want then use 416 x 416

Note that first layer automatically resizes your images to the size of first layer in Yolov3 CNN, so you need not convert your 1280 x 720 images to the input layer size.
Suggest you to read following things:

To understand how Yolov3 works, read this blog post.
To understand some basic stuff read from original site
Learn how to train your custom object detector here


Instruction: pytorch beginner :torch.data.new() torch.new()

Input:
I have a small question, what is the difference between tensor.data.new() and tensor.new()? It seems they all return an empty tensor with the same dtype and device as the self tensor.
Thank you

Output:
There is no difference. It's a little convoluted, but, you can think of .data as being essentially the same object as the Tensor that holds it. Every Tensor has a .data, and the .data is, itself, a Tensor, so there's some circular references going on.
The most important part is that they both always point to the same data, so all operations that don't require a gradient on either the tensor or it's .data will give you the same result.
import torch
a = torch.randn((1,2))
a.data.data_ptr() == a.data_ptr()
# True -- indicating it's precisely the same memory/buffer

The .data property is the counter-part to the .grad property. But, for convenience, as most people only care about the data, not the gradients associated with them, there's a default buffer for the Tensor that .data also points to.


Instruction: How to save and load models of custom dataset in Detectron2?

Input:
I have tried to save and load the model using:
All keys are mapped but there is no prediction in  output
#1
from detectron2.modeling import build_model
model = build_model(cfg)
torch.save(model.state_dict(), 'checkpoint.pth')
model.load_state_dict(torch.load(checkpoint_path,map_location='cpu'))

I also tried doing it using the official doc but can't understand the input format part
from detectron2.checkpoint import DetectionCheckpointer
DetectionCheckpointer(model).load(file_path_or_url)  # load a file, usually from cfg.MODEL.WEIGHTS
checkpointer = DetectionCheckpointer(model, save_dir="output")
checkpointer.save("model_999")  # save to output/model_999.pth

Output:
cfg = get_cfg()
cfg.merge_from_file(model_zoo.get_config_file('COCO-Detection/faster_rcnn_R_101_FPN_3x.yaml'))
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.5 # Set threshold for this model
cfg.MODEL.WEIGHTS = '/content/model_final.pth' # Set path model .pth
cfg.MODEL.ROI_HEADS.NUM_CLASSES = 1
predictor = DefaultPredictor(cfg)

My code to load custom model works.


Instruction: Problem of exporting batchnorm weight from pytorch to Keras

Input:
I follow Pytorch Batchnorm layer different from Keras Batchnorm,
Pytorch Batchnorm implementation, but they do not solve my problem.
I also read Wiki about Batchnorm.
And search source code from tensorflow batchnorm and from pytorch source code.
Below is my testing code, and the results between pytorch and keras are different in 1e-2 to 1e-3 order of error which is big. Function b0, b1 similar to torch result but still not quite accurate.
And b2 tries to follow the formula used in tensorflow batchnorm.
Convolution part yields the same result, but I am stucking at batchnorm layer. I also use eval(), no_grad() for pytorch and model.predict for keras model to make sure they are in inference stage.
Tensorflow implementation does not use 1/sqrt(var+eps), but sqrt(var+eps) instead. I try to transfer 1/running_var to keras.BN.moving_var but still fail the result.
import tensorflow as tf
import tensorflow.keras.layers as L
from tensorflow.keras import Model as KModel
import torch.nn as nn
import torch

def KM():
    x = L.Input((None,None,3))
    y0 = L.Concatenate(axis=-1)([x[:,::2,::2,:],x[:,::2,1::2,:],x[:,1::2,::2,:],x[:,1::2,1::2,:]])
    y1 = L.Conv2D(32,3,1,"same",use_bias=False)(y0)
    y2 = L.BatchNormalization()(y1)
    y3 = L.LeakyReLU(0.1)(y2)
    return KModel(x, [y1, y2, y3])

class YM(nn.Module):
    def __init__(self):
        super(YM, self).__init__()
        self.cat = lambda x : torch.cat([x[:,:,::2,::2],x[:,:,::2,1::2],x[:,:,1::2,::2],x[:,:,1::2,1::2]],axis=1)
        self.conv = nn.Conv2d(12,32,3,1,1,bias=False)
        self.bn = nn.BatchNorm2d(32)
        self.act = nn.LeakyReLU(0.1)

    def forward(self, x):
        y0 = ym.cat(x)
        y0 = ym.conv(y0)
        y1 = ym.bn(y0)
        y2 = ym.act(y1)
        return [y0, y1, y2]

np.random.seed(0)
img = np.random.randint(0,255,(1,12,14,3)).astype(np.float32)
img_torch = torch.from_numpy(img.transpose(0,3,1,2).astype(np.float32))
w1 = np.random.rand(32,12,3,3).astype(np.float32)*0.1
bw1 = np.random.rand(32).astype(np.float32)*0.1
bb1 = np.random.rand(32).astype(np.float32)
bm1 = np.random.rand(32).astype(np.float32)
bv1 = np.abs(np.random.rand(32).astype(np.float32))*0.1

ym = YM()
km = KM()

ym.conv.weight = nn.Parameter(torch.from_numpy(w1))
ym.bn.weight = nn.Parameter(torch.from_numpy(bw1))
ym.bn.bias = nn.Parameter(torch.from_numpy(bb1))
ym.bn.running_mean = torch.from_numpy(bm1)
ym.bn.running_var = torch.from_numpy(bv1)

km.layers[6].set_weights([w1.transpose(2,3,1,0)])
km.layers[7].set_weights([bw1, bb1, bm1, bv1])

ym.eval()
ym.bn.track_running_stats = True
with torch.no_grad():
    t0 = Ym(ym, img_torch/255.-0.5)
k0 = km.predict(img/255.-0.5)

for i in range(len(t0)):
    print(t0[i].shape, k0[i].shape)

Key = 1
print(t0[Key][0,0,:,:].detach().numpy())
print(k0[Key][0,:,:,0])

>>>>>>>>>>>
[[    0.71826     0.72964     0.73189     0.70224     0.74954     0.72928      0.7524]
 [    0.71305     0.68717     0.68581      0.7242     0.73491     0.71925     0.70781]
 [    0.70145     0.66769      0.6857     0.70804     0.73533     0.73165     0.72006]
 [     0.6758     0.69231     0.71173     0.71325     0.72097     0.71414     0.75782]
 [    0.68255     0.72283     0.71273      0.7226     0.71788     0.68119     0.72556]
 [    0.70452     0.68088     0.74389     0.73558     0.72853      0.7174     0.74389]]
[[    0.71953     0.73082     0.73306     0.70365     0.75056     0.73046     0.75339]
 [    0.71437      0.6887     0.68736     0.72543     0.73605     0.72052     0.70918]
 [    0.70287     0.66939     0.68724      0.7094     0.73647     0.73282     0.72133]
 [    0.67743      0.6938     0.71306     0.71457     0.72223     0.71545     0.75877]
 [    0.68413     0.72407     0.71405     0.72384     0.71916     0.68278     0.72678]
 [    0.70592     0.68246     0.74495     0.73671     0.72972     0.71868     0.74496]]```

tt = t0[Key].detach().numpy().transpose(0,2,3,1)
kk = k0[Key]
np.abs(tt-kk).max()
>>>>>>>>>>
0.078752756

gamma, beta = bw1[0], bb1[0]
mu, var = bm1[0], bv1[0]
x_p = t0[0][0,0,0,0]

print(gamma,beta,mu,var,x_p)

eps = 1e-10
def bn0(x_p, mu, var, gamma, beta):
    # wiki
    xhat = (x_p - mu)/np.sqrt(var + eps)
    _x = xhat * gamma + beta
    return _x

def bn1(x_p, mu, var, gamma, beta):
    # pytorch cpp
    inv_var = 1/ np.sqrt(var + eps)
    alpha_d = gamma * inv_var
    beta_d = beta - mu * inv_var * gamma
    return x_p * alpha_d + beta_d

def bn2(x_p, mu, var, gamma, beta):
    # tensorflow cpp
    inv_var = np.sqrt(var + eps)
    xhat = (x_p - mu)*inv_var
    _x = xhat * gamma + beta
    return _x
print(bn0(x_p, mu, var, gamma, beta))
print(bn1(x_p, mu, var, gamma, beta))
print(bn2(x_p, mu, var, gamma, beta))
print(bn2(x_p, mu, 1/var, gamma, beta))

>>>>>>>>
0.048011426 0.87305844 0.67954195 0.059197646 tensor(-0.26256)
tensor(0.68715)
tensor(0.68715)
tensor(0.86205)
tensor(0.68715)

Output:
Keras seems use different default value epsilon (1e-3) vs (1e-5) in Pytorch

The input "var" in source code from tensorflow seems had already been taken 1/moving_variance somewhere.

Besides batchnorm, the padding strategy of tensorflow vs pytorch may yield different result of output. It is recommended to use Zeropadding2D in tensorflow to specify padding number and using valid-conv2d afterwards when there is stride greater than 1(for transfer weighting from pytorch to tensorflow)

accumulated error can be big through the whole network. There is around 0.6 maximum error before final activation function for small network ~70 layers.


Instruction: Pytorch guide not using optim to train

Input:
I am currently working through a guide on the Pytroch website here: https://pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial.html
I have done pytorch projects before and they have always made use of an optimizer. This guide instead uses the code here:
    # Add parameters' gradients to their values, multiplied by learning rate
    for p in rnn.parameters():
        p.data.add_(p.grad.data, alpha=-learning_rate)

I was confused by this and wanted to know why this works. Additionally I tried to rewrite the code using an optimizer and it was unable to learn. It is using a recurrent neural network which may be the reason, but I am unsure why. Thanks!

Output: Why do you expect it to not work? Basically what it is doing is manually implementing an optimizer. p.data is the stored value of the parameter. It also provides an internal function add_ that calculates +=. Once loss.backward() is called, pytorch also calculates and stores the gradient. It is simply taking the the gradient value from the backward pass and updating the parameters to perform gradient descent. There is no reason an optimizer shouldn't work here either, but I can't help with that unless you give more info.


Instruction: RuntimeError: The expanded size of the tensor (7) must match the existing size (128) at non-singleton dimension 3

Input:
When I run AdaIN code
def adaptive_instance_normalization(content_feat, style_mean, style_std):
    size = content_feat.size()
    content_mean, content_std = calc_mean_std(content_feat)

    normalized_feat = (content_feat - content_mean.expand(
        size)) / content_std.expand(size)
    return normalized_feat * style_std.expand(size) + style_mean.expand(size)

I got the following error
RuntimeError: The expanded size of the tensor (7) must match the existing size (128) at non-singleton dimension 3.  Target sizes: [100, 128, 7, 7].  Tensor sizes: [100, 128]

Output:
You should be more precise and descriptive while explaining your issue. You cannot expect from people to read your mind or be familiar with your exact problem. So first, what should be the expected output and which line is failing ? I guess from the expand calls that you would like to enable broadcasting. Unfortunately, as you can read it from the official documentation, expand works the same as usual broadcasting, and add the required extra dimensions at the beginning, not the end.
So you should use reshape(size[:2] + (1, 1)) in place of expand(size).


Instruction: How should I apply a variational autoencoder in a low-dimensional real value case?

Input:
I am trying to applying VAE in a simple toy example to familiarize with its property. However, I get stuck in training the model. The total loss and the reconstruction error does not seem to decrease.
The toy example is listed below

random generate 5000 observation from a 2-dimensional multivariate normal distribution.
apply a transformation f, [x,y] --> [sin(x),sin(y)]
train a VAE with 1 hidden layer with 5 neuron in both encoder and decoder. The VAE has 2 latent variables.

In this example, I am not able to decrease the training loss to a sufficiently low level, and the reconstruction is also messy.
I have made several attempts

increase the hidden layers to 2 and 3 (this does not help)
--> I think it is not due to the complexity of the model
check the network on MNIST (the result is comparable with the example I found on other sources)
--> the model design is right
delete the KL divergence in the Loss function (the model can reconstruct well)
--> the model design is right
I try to balance the weight on KL divergence
--> when beta on KL divergence is low, it reconstruct well, but latent space is too far away from standard normal, when beta on KL divergence is high, it can not reconstruct well, but the latent space perform well.

I now suspect several potential reasons, but I can not distinguish which one could be the reason.

It seems that I need to find a balance between weight on KL divergence and reconstruction loss
Is it appropriate to use MSE loss + KL divergence as loss function?
In low-dimension, the VAE does not perform because the ELBO is not so tight?

Could any one help?
The code is attached.
This part defines the model
import torch
import torch.nn as nn
import pandas as pd
import numpy as np

class VAE_Encoder(nn.Module):

    def __init__(self,input_size,hidden_size_list,latent_size):

        """
        The class is the builder of the encoder part of VAE. It does not need to be directly called.

        :param input_size: int
        :param hidden_size_list: list(int)
        :param latent_size: int
        """

        super().__init__()

        encoder_size = [input_size]+hidden_size_list
        encoder_layers = []

        for in_size,out_size in zip(encoder_size[:-1],encoder_size[1:]):

            encoder_layers.append(nn.Linear(in_size,out_size))
            encoder_layers.append(nn.ReLU())

        self.encoder = nn.Sequential(*encoder_layers)

        self.encoder_mu = nn.Linear(encoder_size[-1],latent_size)
        self.encoder_logvar = nn.Linear(encoder_size[-1],latent_size)

    def encode(self,x):

        return self.encoder(x)

    def encode_gaussian_param(self,encode_x):

        return self.encoder_mu(encode_x),self.encoder_logvar(encode_x)

    def reparametrize(self,mu,logvar):

        std = torch.exp(0.5*logvar)
        eps = torch.randn_like(std)

        return mu+eps*std

    def forward(self,x):

        encode_x = self.encoder(x)
        mu,logvar = self.encode_gaussian_param(encode_x)
        z = self.reparametrize(mu,logvar)

        return z,mu,logvar

class VAE_Decoder(nn.Module):

    def __init__(self,input_size,hidden_size_list,latent_size):

        """
        The class is the builder of the decoder part of VAE. It does not need to be directly called.

        :param input_size: int
        :param hidden_size_list: list(int)
        :param latent_size: int

        """

        super().__init__()

        decoder_size = [latent_size] + hidden_size_list
        decoder_layers = []

        for in_size,out_size in zip(decoder_size[:-1],decoder_size[1:]):

            decoder_layers.append(nn.Linear(in_size,out_size))
            decoder_layers.append(nn.ReLU())

        decoder_layers.append(nn.Linear(decoder_size[-1],input_size))
        self.decoder = nn.Sequential(*decoder_layers)

    def forward(self,z):

        return self.decoder(z)

class VAE(nn.Module):

    def __init__(self,input_size,encoder_size,latent_size,decoder_size=None):

        """
        The class builds the whole VAE. It consists of a encoder model and a decoder model.
        The user has flexibility to choose the number of layers in each part of the model by
        setting the encoder size and decoder size.

        :param input_size: int
        :param encoder_size: list(int)
        :param latent_size: int
        :param decoder_size: list(int)

        """

        super().__init__()

        if decoder_size is None:

            decoder_size = encoder_size[::-1]

        self.encoder = VAE_Encoder(input_size,encoder_size,latent_size)
        self.decoder = VAE_Decoder(input_size,decoder_size,latent_size)

    def decode(self,z):

        return self.decoder(z)

    def forward(self,x):

        z,mu,logvar = self.encoder(x)
        x = self.decoder(z)

        return x,mu,logvar

def simple_vae_loss(real,recon,mu,logvar,penalty=1):

    MSE = nn.functional.mse_loss(recon,real,reduction="sum")
    KLD = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())

    return MSE+penalty*KLD

This part generate the toy example

def generate_value(x,y):

    v1 = np.sin(x)
    v2 = np.sin(y)

    x = np.zeros(x.shape)
    y = np.zeros(y.shape)

    return (v1,v2)

rand_number = torch.randn((5000,2))

x = rand_number.numpy()[:,0]
y = rand_number.numpy()[:,1]

new_value = generate_value(x,y)

new_x = new_value[0].reshape((5000,1))
new_y = new_value[1].reshape((5000,1))
new_data = np.concatenate((new_x,new_y),axis=1)

Output:
Just to pose a potential reason that leads to the result.
In a paper introduction the Hyperspherical Variational Auto-Encoders, the author suggests a problem of using gaussian as prior in the low dimensional setting.
The problem is called origin gravity. Quote the paper.
In low dimensions, the Gaussian density presents a concentrated probability mass around the origin, encouraging points to cluster in the center. This is particularly problematic when the data is divided into multiple clusters. Although an ideal latent space should separate clusters for each class, the normal prior will encourage all the cluster centers towards the origin. An ideal prior would only stimulate the variance of the posterior without forcing its mean to be close to the center. A prior satisfying these properties is a uniform over the entire space. Such a uniform prior, however, is not well defined on the hyperplane.
To verify whether this is the case, I generate 100 synthetic data from the VAE model. The most interesting finding is that all the latent variables concentrated on the origin (0,0).
"If we decrease the weight on the KL divergence, the latent variable starts to spread out. This is consistent with the origin gravity. When the gaussian prior is strong, the latent variable starts to cluster around origin. In this case, we have to reduce the influence of gaussian prior by reducing the weight on KL divergence."
I guess this is one of the reason.


Instruction: Is it normal that accuracy decreases when advancing to the next epoch?

Input:
I’m training a CNN to predict digits using the MNIST database. I’m doing Data Augmentation and for some reason accuracy sharply decreases when advancing to next epoch (iteration 60 in the image)

It has to do with data augmentation (transform = my_transforms in the code) because when I deactivate augmentation (transform = None) accuracy doesn't decrease when advancing to next epoch. But I can't explain why. Does anyone have an idea why this happens?
my_transforms = transforms.Compose([
    transforms.ToPILImage(),
    transforms.RandomCrop((25,25)),
    transforms.Resize((28,28)),
    transforms.RandomRotation(degrees=45, fill=255),
    transforms.RandomVerticalFlip(p=0.1),
    transforms.RandomHorizontalFlip(p=0.5),
    transforms.ToTensor()
    ])

dataset = MNISTDataset(transform = my_transforms)
train_loader = DataLoader(dataset = dataset, batch_size = 1000, shuffle=True)

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1,10,kernel_size=5)
        self.pool = nn.MaxPool2d(kernel_size=2,stride=2)
        self.conv2 = nn.Conv2d(10,20,kernel_size=5)
        self.fc1 = nn.Linear(20*4*4, 64)
        self.fc2 = nn.Linear(64, 10)

def forward(self,x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1,20*4*4)
        x = F.relu(self.fc1(x))
        x = F.softmax(self.fc2(x), dim=1)

        return x

net = Net()

loss_function=nn.NLLLoss()
optimizer=optim.Adam(net.parameters())

EPOCHS=2
iteracion = 0

for epoch in range(EPOCHS):
    for data in train_loader:
        inputs, labels = data
        inputs = inputs.view(-1,1,28,28)

        net.zero_grad()

        probabilities=net(inputs)

        matches=[torch.argmax(i)==int(j) for i,j in zip(probabilities,labels)]

        in_batch_acc=matches.count(True)/len(matches)

        loss=loss_function(torch.log(probabilities), labels)

        print('Loss:', round(float(loss), 3))
        print('In-batch acc:', round(in_batch_acc, 2))

        iteracion += 1

        loss.backward()
        optimizer.step()

Output:
I replicated your model with data augmentation and tried to plot the Accuracy and Loss and it seems that the problem is the way you are plotting.
In the following lines I attach my code and Loss and Accuracy plots:
Code:
# -- Imports -- #
import torch

from torch import nn, optim
from torch.utils.data import DataLoader
from torchvision import transforms, datasets
import torch.nn.functional as F
import matplotlib.pyplot as plt

# -- Data Loader -- #
my_transforms = transforms.Compose([
    transforms.ToPILImage(),
    transforms.RandomCrop((25,25)),
    transforms.Resize((28,28)),
    transforms.RandomRotation(degrees=45, fill=255),
    transforms.RandomVerticalFlip(p=0.1),
    transforms.RandomHorizontalFlip(p=0.5),
    transforms.ToTensor()
    ])
dataset = datasets.MNIST('../data', train=True, download=True,
                         transform = transforms.Compose([
                                            transforms.RandomCrop((25,25)),
                                            transforms.Resize((28,28)),
                                            transforms.RandomRotation(degrees=45, fill=255),
                                            transforms.RandomVerticalFlip(p=0.1),
                                            transforms.RandomHorizontalFlip(p=0.5),
                                            transforms.ToTensor()
                                            ]))
train_loader = DataLoader(dataset = dataset, batch_size = 1000, shuffle=True)

# -- Define Model -- #
class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1,10,kernel_size=5)
        self.pool = nn.MaxPool2d(kernel_size=2,stride=2)
        self.conv2 = nn.Conv2d(10,20,kernel_size=5)
        self.fc1 = nn.Linear(20*4*4, 64)
        self.fc2 = nn.Linear(64, 10)

    def forward(self,x):
            x = self.pool(F.relu(self.conv1(x)))
            x = self.pool(F.relu(self.conv2(x)))
            x = x.view(-1,20*4*4)
            x = F.relu(self.fc1(x))
            x = F.softmax(self.fc2(x), dim=1)

            return x

device = 'cuda'
net = Net()
net.to(device)

loss_function=nn.NLLLoss()
optimizer=optim.Adam(net.parameters())

EPOCHS=2
iteracion = 0
accuracy = []
loss_record = []
for epoch in range(EPOCHS):
    for data in train_loader:
        inputs, labels = data
        inputs = inputs.view(-1,1,28,28)
        inputs, labels = inputs.to(device), labels.to(device)

        # -- Forward -- #
        net.zero_grad()
        probabilities=net(inputs)
        matches=[torch.argmax(i)==int(j) for i,j in zip(probabilities,labels)]
        in_batch_acc=matches.count(True)/len(matches)
        loss=loss_function(torch.log(probabilities), labels)

        # -- Statistics -- #
        accuracy.append(in_batch_acc)
        loss_record.append(loss)
        print('Loss:', round(float(loss), 3))
        print('In-batch acc:', round(in_batch_acc, 2))

        iteracion += 1

        loss.backward()
        optimizer.step()

# -- Accuracy plot -- #
iterations = range(0,120)
plt.plot(iterations, accuracy, 'g', label='Accuracy')
plt.title('Accuracy')
plt.xlabel('Iterations')
plt.ylabel('Accuracy')
plt.legend()
plt.show()

# -- Loss plot -- #
plt.plot(iterations, loss_record, label='Loss')
plt.title('Loss')
plt.xlabel('Iterations')
plt.ylabel('Loss')
plt.legend()
plt.show()

Plots:
Accuracy plot
Loss plot
As you can see, there are not jumps when (iteration = 61) -> Next epoch


Instruction: Why Pytorch is slower than Tensorflow in read data

Input:
I tried two version code for iterate MNIST data to compare the elapsed time.
Pytorch version
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
import torch
from torchvision import datasets, transforms
import matplotlib.pyplot as plt
import time
train_loader = torch.utils.data.DataLoader(
    datasets.MNIST('~/data', train=True, download=True,
                   transform=transforms.Compose([
                       transforms.ToTensor(),
                       # transforms.Normalize((0.1307,), (0.3081,))
                   ])),
    batch_size=30000, shuffle=True,pin_memory=True,num_workers=4)
tic = time.time()
for epoch in range(0, 5):
    for batch_idx, (data, target) in enumerate(train_loader):
        continue
toc=time.time()
print('elapsed time:',toc-tic)

Tensorflow 2.x version
import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
import tensorflow as tf
from tensorflow.keras import datasets
import time
(xs, ys), (xs_, ys_) = datasets.mnist.load_data()
# print('datasets:', xs.shape, ys.shape, xs.min(), xs.max())
xs = tf.convert_to_tensor(xs, dtype=tf.float32) / 255.
db = tf.data.Dataset.from_tensor_slices((xs, ys))
db = db.batch(30000)
tic = time.time()
for epoch in range(5):
    for step, (x, y) in enumerate(db):
        continue
toc = time.time()
print('elapsed time:', toc - tic)

And the result are TF elapsed 2s and Pytorch elapsed 15s.
So, why Pytorch is slower than Tensorflow in read data?
Am I setting it wrong?
Thanks!

Output:
Shijia li,
You do everything correctly, it is just less fast code used in Pytorch (perhaps for a reason).
I looked into Pytorch source code and found the following:

train_loader generates indices for each batch(30000-long list)
train_loaderindices to fetcher.
Fetcher collects data from the dataset, but it only does it one record at a time, in cycle called by line 44 in fetch.py module:

data = [self.dataset[idx] for idx in possibly_batched_index]
I suspect that this is the main source of difference with TF - it may be producing whole batch as a single slice in one optimized operation (vs. Python cycle in Torch)

data gets compacted from list of tuples of tensors into two large tensors by collate_fn and delivered to the user.

If you want to accelerate your code - convert the data into Tensors (equivalent of transforms), then generate the indices yourself and get slices of the data and targets without calling the loader. Or pack the resulting tensors into TensorDataset which should be much faster than VisionDataset (used for MNIST).
Someone at Pytorch development team may want to take a look at this.