Instruction: Speed up training deep learning model in pytorch

Input:
I am working with a training deep learning model with the Pytorch framework. And I add torch.no_grad to speed up the training phase
model.train()
for epoch in range(epochs):
  for data, label in loader:
     data, label = data.to(device), label.to(device)
     with torch.no_grad():
       out = model(data)
     out.requires_grad = True
     #model.zero_grad(), loss(), loss.backward, optim.step

The speed is improved, but have something wrong with the gradient update, the model doesn't converge correctly. Can someone explain to me why it doesn't work?

Output:
Simply, when using the torch.no_grad context manager, the gradients are not computed, so the model cannot receive any update.
torch.no_grad is meant to be used in other cases, for example when evaluating the model. From the docs:

Disabling gradient calculation is useful for inference, when you are sure that you will not call Tensor.backward()


Instruction: Why i'm getting the message "UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors."?

Input:
I'm building my own dataset:
class MyDataset(Dataset):
    def __init__(self, folders):
        self.folders         = folders

    def __len__(self):
        return len(self.folders)

    def __getitem__(self, item):

        pos_file_list =  glob(self.folders[item] + "/*")
        positive_img  = pos_file_list[1]

        positive_img   = mpimg.imread(positive_img)
        positive_img   = np.transpose(positive_img,     (2,0,1))

        # positive_img have the type: <class 'numpy.ndarray'>, shape: (3, 128, 128)
        return positive_img

And I'm using it with:
batch_size =  128
train_ds   = MyDataset(train_folder_list)

oTrainDL   = DataLoader(train_ds, batch_size=batch_size, shuffle=True, num_workers=2)

for i, imgs in enumerate(oTrainDL):
    break

I'm getting the following warrning:
UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at  ../torch/csrc/utils/tensor_numpy.cpp:189.)
  return default_collate([torch.as_tensor(b) for b in batch])

Why I'm getting warrning messaage ? How can I fix it ?

Output:
change from return positive_img to:
return torch.tensor(positive_img)


Instruction: How can I solve this pytorch two devices error

Input:
I ran into a problem with PyTorch:
Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking arugment for argument mat1 in method wrapper_addmm)
model = nn.Sequential(
        nn.Linear(622, 512),
        nn.ReLU(),
        nn.Linear(512, 256),
        nn.ReLU(),
        nn.Linear(256, 5),
    ).to(device)

    loss_fn = nn.CrossEntropyLoss()
    optimizer = torch.optim.Adam(model.parameters(), lr=1e-4)

    train_loader = Data.DataLoader(
        dataset=train_dataset,
        batch_size=32,
        shuffle=True,
        num_workers=0,
    )

    test_loader = Data.DataLoader(
        dataset=test_dataset,
        batch_size=100,
        shuffle=True,
        num_workers=0,
    )

    best_acc = 0
    best_model = model.cpu().state_dict().copy()
    # train_acc = 0
    # test_acc = 0
    for epoch in range(20):
        for step, (batch_x, batch_y) in enumerate(train_loader):
            batch_x = batch_x.to(device)
            batch_y = batch_y.to(device)
            print(batch_x)
            print(batch_x.device, 0)
            out = model(batch_x.to(device)).cuda()
            print(out.device, 1)
            loss = loss_fn(out, batch_y.long())
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
            train_acc = np.mean((torch.argmax(out, 1) == batch_y).cpu().numpy())

            with torch.no_grad():
                for batch_x, batch_y in test_loader:
                    batch_x = batch_x.to(device)
                    batch_y = batch_y.to(device)
                    print(batch_x.device, 2)
                    out = model(batch_x)
                    print(batch_x.device, 3)
                    test_acc = np.mean((torch.argmax(out, 1) == batch_y).cpu().numpy())
            if test_acc > best_acc:
                best_acc = test_acc
                best_model = model.cpu().state_dict().copy()

Can someone help explain that， I've been working on this all day....

Output:
Note that .to() has different behavior when applied to nn.Modules and to torch.tensors: while for torch.tensor .to(device) creates a copy of the tensor on the device, with nn.Module .to(device) operates in place.
In your code, you move your model to CPU:
best_model = model.cpu().state_dict().copy()

make sure you move the model back to device after moving it to cpu.


Instruction: How to call "backward" in a loop with 2 optimizers?

Input:
I have 2 networks that I'm trying to update:
import torch
import torch.nn as nn
import torch.optim as optim
from torch.distributions import Normal
import matplotlib.pyplot as plt
from tqdm import tqdm

softplus = torch.nn.Softplus()

class Model_RL(nn.Module):
    def __init__(self):
        super(Model_RL, self).__init__()
        self.fc1 = nn.Linear(3, 20)
        self.fc2 = nn.Linear(20, 30)
        self.fc3 = nn.Linear(30, 2)

    def forward(self, x):
        x = torch.relu(self.fc1(x))
        x = torch.relu(self.fc2(x))
        x = softplus(self.fc3(x))
        return x

class Model_FA(nn.Module):
    def __init__(self):
        super(Model_FA, self).__init__()
        self.fc1 = nn.Linear(1, 20)
        self.fc2 = nn.Linear(20, 30)
        self.fc3 = nn.Linear(30, 1)

    def forward(self, x):
        x = torch.relu(self.fc1(x))
        x = torch.relu(self.fc2(x))
        x = softplus(self.fc3(x))
        return x

net_RL = Model_RL()
net_FA = Model_FA()

The training loop is
inps = torch.tensor([[1.0]])
y = torch.tensor(10.0)

opt_RL = optim.Adam(net_RL.parameters())
opt_FA = optim.Adam(net_FA.parameters())

baseline = 0
baseline_lr = 0.1

epochs = 100

for _ in tqdm(range(epochs)):

    for inp in inps:

        with torch.no_grad():
            net_FA(inp)

        for layer in range(3):
            out_RL = net_RL(torch.tensor([1.0,2.0,3.0]))
            mu, std = out_RL
            dist = Normal(mu, std)
            update_values = dist.sample()
            log_p = dist.log_prob(update_values).mean()

            out = net_FA(inp)
            reward = -torch.square((y - out))
            baseline = (1 - baseline_lr) * baseline + baseline_lr * reward

            loss_RL = - (reward - baseline) * log_p
            opt_RL.zero_grad()
            opt_FA.zero_grad()
            loss_RL.backward()
            opt_RL.step()

            out = net_FA(inp)
            loss_FA = torch.mean(torch.square(y - out))
            opt_RL.zero_grad()
            opt_FA.zero_grad()
            loss_FA.backward()
            opt_FA.step()

print("Mean: " + str(mu.detach().numpy()) + ", Goal: " + str(y))
print("Standard deviation: " + str(softplus(std).detach().numpy()) + ", Goal: 0ish")

I'm getting 2 main errors:
RuntimeError: Trying to backward through the graph a second time, but the saved intermediate results have already been freed. Specify retain_graph=True when calling .backward()...

And when I add retain_graph=True to both backward calls I get the following
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [30, 1]], which is output 0 of TBackward, is at version 5; expected version 4 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True)

My main question is how can I make this training work?
But intermediate questions are:
why does retain_graph=True is needed here if I'm using a loop? From here: "there is no need to use retain_graph=True. In each loop, a new graph is created"
Why does it seem as if the retain_graph=True makes training significantly slower (if I remove the other backward call)? This doesn't really makes sense to me as in each epoch a new computational graph should be created (and not just one that is being extended).

Output:
I think the line baseline = (1 - baseline_lr) * baseline + baseline_lr * reward causing the error. Because:

previous state of baseline is used to get new state of baseline.
PyTorch will track all these states inside a graph.
backward will flush the graph.
variable baseline of time - t + 1 will try to backpropagate through baseline of time - t.
But at time - t + 1  graph behind baseline of time - t doesn't exist.
This leads to error

Solution:
As you are not optimizing variable baseline or anything behind baseline

Initializebaseline as torch tensor.
detach it from graph before updating state.

Try this:
# intialize baseline as torch tensor
baseline = torch.tensor(0.)
baseline_lr = 0.1

epochs = 100

for _ in tqdm(range(epochs)):

    for inp in inps:

        with torch.no_grad():
            net_FA(inp)

        for layer in range(3):
            out_RL = net_RL(torch.tensor([1.0,2.0,3.0]))
            mu, std = out_RL
            dist = Normal(mu, std)
            update_values = dist.sample()
            log_p = dist.log_prob(update_values).mean()

            out = net_FA(inp)
            reward = -torch.square((y - out))

            # detach baseline from graph
            baseline = (1 - baseline_lr) * baseline.detach() + baseline_lr * reward

            loss_RL = - (reward - baseline) * log_p
            opt_RL.zero_grad()
            opt_FA.zero_grad()
            loss_RL.backward()
            opt_RL.step()

            out = net_FA(inp)
            loss_FA = torch.mean(torch.square(y - out))
            opt_RL.zero_grad()
            opt_FA.zero_grad()
            loss_FA.backward()
            opt_FA.step()

But actually I don't know why you are updating the networks, 3 times for the same input?


Instruction: I made an code with command append, to compare two tensors with pytorch. What's wrong with it?

Input:
xxx=torch.tensor([True,True,False,True])
xxxz=torch.tensor([True,False,False,True])

def zzz():
  asdf=[]
  for i in range(4):
    if xxx[i] == xxxz[i] and xxx[i] == True:
      asdf.append(i)
      return asdf

zzz()

What I expected was: [0,3].
But the result was : [0]. I don't get what's wrong with it. Did I use append wrongly?

Output:
Your return statement is inside for and if. Pull it outside at the level of function definition
def zzz():
  asdf=[]
  for i in range(4):
    if xxx[i] == xxxz[i] and xxx[i] == True:
      asdf.append(i)
      # return asdf # NOPE
  return asdf # THIS

Earlier, your function was returning just after encountering 0. That's why 3 was missing.


Instruction: PyTorch installation asks for python=3.1 . Python Version installed: 3.10.0

Input:
(pgqa) raphy@pc:~/pythonMatters/PathGenerator$ conda install pytorch torchvision torchaudio cpuonly -c pytorch
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.

PackagesNotFoundError: The following packages are not available from current channels:

  - python=3.1

Current channels:

  - https://conda.anaconda.org/pytorch/linux-64
  - https://conda.anaconda.org/pytorch/noarch
  - https://repo.anaconda.com/pkgs/main/linux-64
  - https://repo.anaconda.com/pkgs/main/noarch
  - https://repo.anaconda.com/pkgs/r/linux-64
  - https://repo.anaconda.com/pkgs/r/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.

(pgqa) raphy@pc:~/pythonMatters/PathGenerator$ python3 --version
Python 3.10.0

Tried installing PyTorch with pip but got these errors:
(pgqa) raphy@pc:~/pythonMatters/PathGenerator$ pip3 install torch==1.10.0+cpu
torchvision==0.11.1+cpu torchaudio==0.10.0+cpu -f https://download.pytorch.org
/whl/cpu/torch_stable.html
Looking in links: https://download.pytorch.org/whl/cpu/torch_stable.html
ERROR: Could not find a version that satisfies the requirement torch==1.10.0+cpu (from versions: none)
ERROR: No matching distribution found for torch==1.10.0+cpu

O.S.: Ubuntu 20.04
pip version:
(pgqa) raphy@pc:~$ pip --version
pip 21.2.4 from /home/raphy/anaconda3/envs/pgqa/lib/python3.10/site-packages/pip (python 3.10)
(pgqa) raphy@pc:~$ pip3 --version
pip 21.2.4 from /home/raphy/anaconda3/envs/pgqa/lib/python3.10/site-packages/pip (python 3.10)

Python Version:
(pgqa) raphy@pc:~$ python --version
Python 3.10.0

Why does it asks for  python3.1 if the latest stable python version is 3.10 ?
How to solve the problem?

Output:
Conda 4.10 is incompatible with python 3.10.
The reason you cannot install many packages with conda is that conda 4.10 has a known bug handling python 3.10. Update your conda to 4.11 or revert your python to 3.9 or older. Read more here in this SO answer.
As a workaround, you can still try to use pip.


Instruction: RuntimeError: mean(): input dtype should be either floating point or complex dtypes. Got Long instead

Input:
I wrote below code using pytorch and ran into a runtime error:
tns = torch.tensor([1,0,1])
tns.mean()
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-666-194e5ab56931> in <module>
----> 1 tns.mean()

RuntimeError: mean(): input dtype should be either floating point or complex dtypes. Got Long instead.

However, if I change the tensor to float, the error goes away:
tns = torch.tensor([1.,0,1])
tns.mean()
---------------------------------------------------------------------------
tensor(0.6667)

My question is why the error happens. The data type of the first tenor is int64 instead of Long, why does PyTorch take it as Long?

Output: This is because torch.int64 and torch.long both refer to the same data type, of 64-bit signed integers. See here for an overview of all data types.


Instruction: Why should I use a 2**N value and how do I choose the right one?

Input:
I'm working through the lessons on building a neural network and I'm confused as to why 512 is used for the linear_relu_stack in the example code:
class NeuralNetwork(nn.Module):
    def __init__(self):
        super(NeuralNetwork, self).__init__()
        self.flatten = nn.Flatten()
        self.linear_relu_stack = nn.Sequential(
            nn.Linear(28*28, 512),
            nn.ReLU(),
            nn.Linear(512, 512),
            nn.ReLU(),
            nn.Linear(512, 10),
            nn.ReLU()
        )

    def forward(self, x):
        x = self.flatten(x)
        logits = self.linear_relu_stack(x)
        return logits

I started googling around and saw many examples of the torch.nn.Linear function using various values of 2**N but it isn't clear to me why they are using powers of 2 nor how they are choosing which value to use.

Output:
The reason is how hardware makes the process. In deep learning matrix operations are the main computations and source of floating point operations (FLOPs).
Single Instruction Multiple Data (SIMD) operations in CPUs happen in batch sizes, which are powers of 2. Consider take a look if you are interested:
https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/37631.pdf
And for GPUs:
https://docs.nvidia.com/cuda/cuda-c-best-practices-guide/index.html

Memory allocated through the CUDA Runtime API, such as via
cudaMalloc(), is guaranteed to be aligned to at least 256 bytes.
Therefore, choosing sensible thread block sizes, such as multiples of
the warp size (i.e., 32 on current GPUs), facilitates memory accesses
by warps that are properly aligned. (Consider what would happen to the
memory addresses accessed by the second, third, and subsequent thread
blocks if the thread block size was not a multiple of warp size, for
example.)

This means that any multiple of 32 will optimize the memory access, and thus, the processing speed, while you are using a gpu.
About the right value, pyramidal shape usually works better, because as you go deeper, the neural network tends to create internal representations of the transformed data, in an expected hierarchical, thus, pyramidal shape. So a good guess is to use decreasing amounts of neurons at each layer as you come close to the output, e.g:
self.flatten = nn.Flatten()
self.linear_relu_stack = nn.Sequential(
    nn.Linear(28*28, 512),
    nn.ReLU(),
    nn.Linear(512, 128),
    nn.ReLU(),
    nn.Linear(128, 10),
    nn.ReLU()
    )

But there is no general rule and you can find whole fields of study (like Neural Architecture Search) about how to find optimal hyper-parameters for neural networks.
you can take a look here for some deeper information:
https://arxiv.org/pdf/1608.04064.pdf


Instruction: How does a gradient backpropagates through random samples?

Input:
I'm learning about policy gradients and I'm having hard time understanding how does the gradient passes through a random operation. From here: It is not possible to directly backpropagate through random samples. However, there are two main methods for creating surrogate functions that can be backpropagated through.
They have an example of the score function:
probs = policy_network(state)
# Note that this is equivalent to what used to be called multinomial
m = Categorical(probs)
action = m.sample()
next_state, reward = env.step(action)
loss = -m.log_prob(action) * reward
loss.backward()

Which I tried to create an example of:
import torch
import torch.nn as nn
import torch.optim as optim
from torch.distributions import Normal
import matplotlib.pyplot as plt
from tqdm import tqdm

softplus = torch.nn.Softplus()

class Model_RL(nn.Module):
    def __init__(self):
        super(Model_RL, self).__init__()
        self.fc1 = nn.Linear(1, 20)
        self.fc2 = nn.Linear(20, 30)
        self.fc3 = nn.Linear(30, 2)

    def forward(self, x):
        x1 = self.fc1(x)
        x = torch.relu(x1)
        x2 = self.fc2(x)
        x = torch.relu(x2)
        x3 = softplus(self.fc3(x))
        return x3, x2, x1

# basic

net_RL = Model_RL()

features = torch.tensor([1.0])
x = torch.tensor([1.0])
y = torch.tensor(3.0)

baseline = 0
baseline_lr = 0.1

epochs = 3

opt_RL = optim.Adam(net_RL.parameters(), lr=1e-3)
losses = []
xs = []
for _ in tqdm(range(epochs)):
    out_RL = net_RL(x)
    mu, std = out_RL[0]
    dist = Normal(mu, std)
    print(dist)
    a = dist.sample()
    log_p = dist.log_prob(a)

    out = features * a
    reward = -torch.square((y - out))
    baseline = (1-baseline_lr)*baseline + baseline_lr*reward

    loss = -(reward-baseline)*log_p

    opt_RL.zero_grad()
    loss.backward()
    opt_RL.step()
    losses.append(loss.item())

This seems to work magically fine which again, I don't understand how the gradient passes through as they mentioned that it can't pass through the random operation (but then somehow it does).
Now since the gradient can't flow through the random operation I tried to replace
mu, std = out_RL[0] with mu, std = out_RL[0].detach() and that caused the error:
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn. If the gradient doesn't pass through the random operation, I don't understand why would detaching a tensor before the operation matter.

Output:
It is indeed true that sampling is not a differentiable operation per se. However, there exist two (broad) ways to mitigate this - [1] The REINFORCE way and [2] The reparameterization way. Since your example is related to [1], I will stick my answer to REINFORCE.
What REINFORCE does is it entirely gets rid of sampling operation in the computation graph. However, the sampling operation remains outside the graph. So, your statement

.. how does the gradient passes through a random operation ..

isn't correct. It does not pass through any random operation. Let's see your example
mu, std = out_RL[0]
dist = Normal(mu, std)
a = dist.sample()
log_p = dist.log_prob(a)

Computation of a does not involve creating a computation graph. It is technically equivalent to plugging in some offline data from a dataset (as in supervised learning)
mu, std = out_RL[0]
dist = Normal(mu, std)
# a = dist.sample()
a = torch.tensor([1.23, 4.01, -1.2, ...], device='cuda')
log_p = dist.log_prob(a)

Since we don't have offline data beforehand, we create them on the fly and the .sample() method does merely that.
So, there is no random operation on the graph. The log_p depends on mu and std deterministically, just like any standard computation graph. If you cut the connection like this
mu, std = out_RL[0].detach()

.. of course it is going to complaint.
Also, do not get confused by this operation
dist = Normal(mu, std)
log_p = dist.log_prob(a)

as it does not contain any randomness by itself. This is merely a shortcut for writing the tedious log-likelihood formula for Normal distribution.


Instruction: How to correctly replace the real part of the FFT?

Input:
I try to reproduce a data augmentation method, which comes from the paper:
Qinwei Xu, Ruipeng Zhang, Ya Zhang, Yanfeng Wang and Qi Tian "A Fourier-based Framework for Domain Generalization" (CVPR 2021).
It is mentioned in the paper that they set the real part to a constant (the constant in the paper is 20000) to eliminate the amplitude and realize the reconstruction of the image relying only on the phase.
Below is my code:
img = process_img("./data/house.jpg", 128)
img_fft = torch.fft.fft2(img, dim=(-2, -1))
amp = torch.full(img_fft.shape, 200000)
img_fft.real = amp
img_ifft = torch.fft.ifft2(img_fft, dim=(-2, -1))

img_ifft = img_ifft.squeeze(0)
img_ifft = img_ifft.transpose(2, 0)

img_ifft = np.array(img_ifft)

cv2.imshow("", img_ifft.real)

Among them, the process_img function is only used to convert ndarray to tensor, as shown below:
loader = transforms.Compose([transforms.ToTensor()])
def process_img(img_path, img_size):
    img = cv2.imread(img_path)
    img = cv2.resize(img, (img_size, img_size))
    img = img.astype(np.float32) / 255.0
    img = loader(img)
    img = img.unsqueeze(0)
    return img

The first is the original image, the second is the image provided by the paper, and the third is the image generated by my code:

It can be seen that the images generated by my method are very different from those provided in the paper, and there are some artifacts. Why is there such a result?

Output:
You are confusing "real"/"imaginary" parts of complex numbers with "amplitude"/"phase" representation.
Here's the quick guide:
A complex number z can be expressed by either a sum of its real part x and its imaginary part y:
z = x + j y
Alternatively, once can express the same complex number z as a rotated vector with amplitude r and an angle phi:
z = r exp(j phi)
Where r = sqrt(x^2 + y^2) and phi=atan2(x,y).
This image (from Wikipedia) explain this visually:

In your code, you replace the "real" part, but in the paper, they suggest replacing the "amplitude".
If you want to replace the amplitude:
const_amp = ... # whatever the constant amplitude you want
new_fft = const_amp * torch.exp(1j * img_fft.angle())

# reconstruct the new image from the modulated Fourier:
img_ifft = torch.fft.ifft2(new_fft, dim=(-2, -1))

This results with the following image:


Instruction: PyTorch model tracing not working: We don't have an op for aten::fill_

Input:
I am stuck on tracing a PyTorch model on this specific module with an error:
RuntimeError: 0INTERNAL ASSERT FAILED at "../torch/csrc/jit/ir/alias_analysis.cpp":611, please report a bug to PyTorch. We don't have an op for aten::fill_ but it isn't a special case.  Argument types: Tensor, bool,

Candidates:
    aten::fill_.Scalar(Tensor(a!) self, Scalar value) -> (Tensor(a!))
    aten::fill_.Tensor(Tensor(a!) self, Tensor value) -> (Tensor(a!))

Here is the reduced code example to reproduce the bug:
import torch
import torch.nn.functional as F
import torch.nn as nn

class SurroundPattern(nn.Module):
    def __init__(self, crop_size=1./2):
        super(SurroundPattern, self).__init__()
        self.crop_size = crop_size

    def forward(self, x, s):
        H,W         = x.shape[2:]
        crop_h      = (int(H / 2 - self.crop_size / 2 * H), int(H / 2 + self.crop_size / 2 * H))
        crop_w      = (int(W / 2 - self.crop_size / 2 * W), int(W / 2 + self.crop_size / 2 * W))
        x_mask      = torch.zeros(H,W,device=x.device, dtype=torch.bool)
        x_mask[crop_h[0] : crop_h[1], crop_w[0] : crop_w[1]] = True

        inside_indices  = torch.where(x_mask)
        inside_part = x[:, :, inside_indices[0], inside_indices[1]]
        inside_feat = inside_part.mean(2)

        outside_indices = torch.where(~x_mask)
        outside_part    = x[:, :, outside_indices[0], outside_indices[1]]
        outside_feat    = outside_part.mean(2)
        fused = torch.stack([inside_feat, outside_feat], dim=2).unsqueeze(3)
        if s is None:
            return fused

        SH,SW       = s.shape[2:]
        crop_sh     = (int(SH / 2 - self.crop_size / 2 * SH), int(SH / 2 + self.crop_size / 2 * SH))
        crop_sw     = (int(SW / 2 - self.crop_size / 2 * SW), int(SW / 2 + self.crop_size / 2 * SW))
        s_mask      = torch.zeros(SH, SW, device=s.device, dtype=torch.bool)
        s_mask[crop_sh[0] : crop_sh[1], crop_sw[0] : crop_sw[1]] = True

        s_inside_indices = torch.where(s_mask)
        inside_sal  = s[:, :, s_inside_indices[0], s_inside_indices[1]].flatten(1)

        s_outside_indices = torch.where(~s_mask)
        outside_sal = s[:, :, s_outside_indices[0], s_outside_indices[1]].flatten(1)
        if outside_sal.shape != inside_sal.shape:
            outside_sal = F.adaptive_max_pool1d(outside_sal.unsqueeze(1), output_size=784)
            outside_sal = outside_sal.squeeze(1)
        fused_sal    = torch.stack([inside_sal, outside_sal], dim=2).unsqueeze(3)
        return fused, fused_sal

x = torch.randn(2, 512, 7, 7)
s = torch.randn(2, 1, 56, 56)

patt = SurroundPattern()

traced_cell = torch.jit.trace(patt, (x, s))
print(traced_cell)

How to figure out where exactly the problem is? Is there a way to fix it with another functions?
Thank you!

Output:
The problem is that you try to fill in a bool Tensor which is apparently not yet supported in jit (or a bug)
Replacing this:
    x_mask= torch.zeros(H,W,device=x.device, dtype=torch.bool)
    x_mask[crop_h[0] : crop_h[1], crop_w[0] : crop_w[1]] = True

with:
    x_mask= torch.zeros(H,W,device=x.device)
    x_mask[crop_h[0] : crop_h[1], crop_w[0] : crop_w[1]] = 1

should resolve the error. This of course is suboptimal to the target Tensor type but you should be able to perform any other operation you would be doing with torch.BoolTensor


Instruction: How to load custom model in pytorch

Input:
I'm trying to load my pretrained model (yolov5n) and test it with the following code in PyTorch:
import os
import torch
model = torch.load(os.getcwd()+'/weights/last.pt')

# Images
imgs = ['https://example.com/img.jpg']
# Inference
results = model(imgs)

# Results
results.print()
results.save()  # or .show()

results.xyxy[0]  # img1 predictions (tensor)
results.pandas().xyxy[0]  # img1 predictions (pandas)

and I'm getting the following error:

ModuleNotFoundError                       Traceback (most recent call
last)  in
3 import torch
4
----> 5 model = torch.load(os.getcwd()+'/weights/last.pt')

My model is located in the folder /weights/last.py, I'm not sure what I'm doing false. Could you please tell me, what it's missing in my code.

Output:
You should be able to find the weights in this directory: yolov5/runs/train/exp/weights/last.pt
Then you load the weights with a line like this:
model = torch.hub.load('ultralytics/yolov5', 'custom', path='yolov5/runs/train/exp/weights/last.pt', force_reload=True)

I have an example of a notebook that loads custom models and from that directory after training the model here https://github.com/pylabel-project/samples/blob/main/pylabeler.ipynb


Instruction: pytorch assigning to results of gather

Input:
Suppose I have the following matrix, and its top 3 results across each row:
p = torch.randn(5, 7)
val, idx = p.topk(3, dim=-1)

I wish to assign x to the top 3 results of each row where x is:
x = torch.randn(5, 3)

Now I know that doing torch.gather(p, -1, idx) will get me the correct elements that I want to replace, but I cannot replace against the function gather. What is the best way of getting the effect of:
torch.gather(p, -1, idx) = x

Output:
One solution is to use list-style indexing to p:
# create dummy indices to index the correct row (we need one value per value in idx)
row_idx = torch.arange(len(p)).unsqueeze(1).repeat(1,3)

# use flattened views
p[row_idx.view(-1),idx.view(-1)] = x.view(-1)

List-based indexing does require contiguous memory tensors, so you may pay a small computational penalty if p is non-contiguous, but I suspect any non-looping solution to this indexing task would have the same requirement.


Instruction: What is total_loss,loss_cls etc

Input:
I want to train a custom dataset on using faster_rcnn or mask_rcnn with the Pytorch and Detectron2 .Everything works well but I wanted to know I want to know what are the results I have.
[11/29 20:16:31 d2.utils.events]:  eta: 0:24:04  iter: 19  total_loss: 9.6  loss_cls: 1.5  loss_box_reg: 0.001034  loss_mask: 0.6936  loss_rpn_cls: 6.773  loss_rpn_loc: 0.5983  time: 1.4664  data_time: 0.0702  lr: 4.9953e-06  max_mem: 2447M

I have this as result and I want to know what all of this means

Output:
Those are metrics printed out at every iteration of the training loop. The most important ones are the loss values, but below are basic descriptions of them all (eta and iter are self-explanatory I think).
total_loss: This is a weighted sum of the following individual losses calculated during the iteration. By default, the weights are all one.

loss_cls: Classification loss in the ROI head. Measures the loss for box classification, i.e., how good the model is at labelling a predicted box with the correct class.

loss_box_reg: Localisation loss in the ROI head. Measures the loss for box localisation (predicted location vs true location).

loss_rpn_cls: Classification loss in the Region Proposal Network. Measures the "objectness" loss, i.e., how good the RPN is at labelling the anchor boxes as foreground or background.

loss_rpn_loc: Localisation loss in the Region Proposal Network. Measures the loss for localisation of the predicted regions in the RPN.

loss_mask: Mask loss in the Mask head. Measures how "correct" the predicted binary masks are.
For more details on the losses (1) and (2), take a look at the Fast R-CNN paper and the code.
For more details on the losses (3) and (4), take a look at the Faster R-CNN paper and the code.
For more details on the loss (5), take a look at the Mask R-CNN paper and the code.

time: Time taken by the iteration.
data_time: Time taken by the dataloader in that iteration.
lr: The learning rate in that iteration.
max_mem: Maximum GPU memory occupied by tensors in bytes.


Instruction: Parameters for Optimizer in ShuffleNet

Input:
In ShuffleNet the parameters that send to the optimizer first pass from the following function which makes a dictionary of weights:
def get_parameters(model):
    group_no_weight_decay = []
    group_weight_decay = []
    for pname, p in model.named_parameters():
        if pname.find('weight') >= 0 and len(p.size()) > 1:
            # print('include ', pname, p.size())
            group_weight_decay.append(p)
        else:
            # print('not include ', pname, p.size())
            group_no_weight_decay.append(p)
    assert len(list(model.parameters())) == len(group_weight_decay) + len(group_no_weight_decay)
    groups = [dict(params=group_weight_decay), dict(params=group_no_weight_decay, weight_decay=0.)]
    return groups

and then:
optimizer = optim.Adam(get_parameters(model), lr=0.01)

But what is the difference between this function and just using model.parameters  instead of get_parameters(model)?

Output:
With model.parameters() yo get all the parameters of the model in a single "group", and thus all hyper parameters of the optimizer are the same for all model.parameters().
In contrast, get_parameters() groups model.parameters() into two groups: group_weight_decay and group_no_weight_decay. As the names suggest, for the parameters of the second group the optimizer sets the weight_decay hyper parameter to zero.


Instruction: how to send an numpy array or a pytorch Tensor through http post request using requests module and Flask

Input:
I have an image and I want to send it to the serve. I'm using requests module to perform simple post request as following(info is a dictionary):
    import requests

    print(type(info["array_image"]))
    print(type(info["visual_features"]))
    response = requests.post("url", data=info)

output :
<class 'numpy.ndarray'>
<class 'torch.Tensor'>

on the server side I'm trying to receive them as arrays at least:
from flask import Flask, request

@app.route('/path', methods=['POST'])
def function_name():
    visual_features = request.form['visual_features']
    array_image = request.form['array_image']
    print(type(array_image))
    print(type(visual_features))

output:
<class 'str'>
<class 'str'>

I want to get a bytes array to build the image, but what I'm getting is a string...
If I didn't find a way I'll encode arrays in bas64 and then decode it in the server...

Output:
Thanks to @praba230890 for giving me an easy to follow example.
I would still write the solution here down, since the provided link doesn't fit my case exactly.
import pickle
import io

bytes_image = pickle.dumps(info["array_image"])
stream = io.BytesIO(bytes_image)
files = {"bytes_image": stream}

info["array_image"] = None

response = http.post("url", data=info, files=files)

and in the server side:
from flask import Flask, request

@app.route('/path', methods=['POST'])
def function_name():
    image = request.files.get('bytes_image')
    bytes_image = image.read()

if you want to get the image from a file, then:
requests.post("http://localhost:5000/predict",
                 files={"file": open('<PATH/TO/.jpg/FILE>/cat.jpg','rb')})

The Solution I'm currently using:
remember info["array_image"] is a numpy array, and info is a dictionary
    import io
    info["image_shape_width"] = info["array_image"].shape[0]
    info["image_shape_height"] = info["array_image"].shape[1]

    bytes_image = info["array_image"].tobytes()
    stream = io.BytesIO(bytes_image)
    files = {"bytes_image": stream}

    info["array_image"] = None

    response = http.post(self.ip + "path", data=info, files=files)

then receive it
    from flask import Flask, request
    import numpy as np

    @app.route('/path', methods=['POST'])
    def function_name():
        bytes_image = request.files.get('bytes_image')
        bytes_image = bytes_image.read()
        array_image = np.frombuffer(bytes_image, dtype=dtype)
        shape = (int(request.form['image_shape_width']), int(request.form['image_shape_height']), 3)
        array_image = np.reshape(array_image, shape)

        image = Image.fromarray(array_image)


Instruction: Projected gradient descent on probability simplex in pytorch

Input:
I have a matrix A of dimension 1000x70000.
my loss function includes A and I want to find optimal value of A using gradient descent where the constraint is that the rows of A remain in probability simplex (i.e. every row sums up to 1). I have initialised A as given below
A=np.random.dirichlet(np.ones(70000),1000)
A=torch.tensor(A,requires_grad=True)

and my training loop looks like as given below
for epoch in range(500):
    y_pred=forward(X)
    y=model(torch.mm(A.float(),X))
    l=loss(y,y_pred)
    l.backward()
    A.grad.data=-A.grad.data
    optimizer.step()
    optimizer.zero_grad()
    if epoch%2==0:
        print("Loss",l,"\n")

Output:
An easy way to accomplish that is not to use A directly for computation but use a row normalized version of A.
# you can keep 'A' unconstrained
A = torch.rand(1000, 70000, requires_grad=True)

then divide each row by its summation (keeping row sum always 1)
for epoch in range(500):
    y_pred = forward(X)
    B = A / A.sum(-1, keepdim=True) # normalize rows manually
    y = model(torch.mm(B, X))
    l = loss(y,y_pred)
    ...

So now, at each step, B is the constrained matrix - i.e. the quantity of your interest. However, the optimization would still be on (unconstrained) A.

Edit: @Umang Gupta remined me in the comment section that OP wanted to have "probability simplex" which means there would be another constraint, i.e. A >= 0.
To accomplish that, you may simply apply some appropriate activation function (e.g. torch.exp, torch.sigmoid) on A in each iteration
A_ = torch.exp(A)
B = A_ / A_.sum(-1, keepdim=True) # normalize rows

the exact choice of function depends on the behaviour of training dynamics which needs to be experimented with.


Instruction: Creating and Use a PyTorch DataLoader

Input:
I am trying to create a PyTorch Dataset and DataLoader object using a sample data.
This is the tab seperated dataset:
1 0  0.171429  1 0 0  0.966805  0
0 1  0.085714  0 1 0  0.188797  1
1 0  0.000000  0 0 1  0.690871  2
1 0  0.057143  0 1 0  1.000000  1
0 1  1.000000  0 0 1  0.016598  2
1 0  0.171429  1 0 0  0.802905  0
0 1  0.171429  1 0 0  0.966805  1
1 0  0.257143  0 1 0  0.329876  0

This is the code to create the Dataset above and DataLoader object:
import numpy as np
import torch as T
device = T.device("cpu")  # to Tensor or Module

# ---------------------------------------------------

# predictors and label in same file
# data has been normalized and encoded like:
#   sex     age      region   income    politic
#   [0]     [2]       [3]      [6]       [7]
#   1 0   0.057143   0 1 0    0.690871    2

class PeopleDataset(T.utils.data.Dataset):

  def __init__(self, src_file, num_rows=None):
    x_tmp = np.loadtxt(src_file, max_rows=num_rows,
      usecols=range(0,7), delimiter="\t",
      skiprows=0, dtype=np.float32)
    y_tmp = np.loadtxt(src_file, max_rows=num_rows,
      usecols=7, delimiter="\t", skiprows=0,
      dtype=np.long)

    self.x_data = T.tensor(x_tmp,
      dtype=T.float32).to(device)
    self.y_data = T.tensor(y_tmp,
      dtype=T.long).to(device)

  def __len__(self):
    return len(self.x_data)  # required

  def __getitem__(self, idx):
    if T.is_tensor(idx):
      idx = idx.tolist()
    preds = self.x_data[idx, 0:7]
    pol = self.y_data[idx]
    sample = \
      { 'predictors' : preds, 'political' : pol }
    return sample

# ---------------------------------------------------

def main():
  print("\nBegin PyTorch DataLoader demo ")

  # 0. miscellaneous prep
  T.manual_seed(0)
  np.random.seed(0)

  print("\nSource data looks like: ")
  print("1 0  0.171429  1 0 0  0.966805  0")
  print("0 1  0.085714  0 1 0  0.188797  1")
  print(" . . . ")

  # 1. create Dataset and DataLoader object
  print("\nCreating Dataset and DataLoader ")

  train_file = "people_train.txt"
  train_ds = PeopleDataset(train_file, num_rows=8)

  bat_size = 3
  train_ldr = T.utils.data.DataLoader(train_ds,
    batch_size=bat_size, shuffle=True)

  # 2. iterate thru training data twice
  for epoch in range(2):
    print("\n==============================\n")
    print("Epoch = " + str(epoch))
    for (batch_idx, batch) in enumerate(train_ldr):
      print("\nBatch = " + str(batch_idx))
      X = batch['predictors']  # [3,7]
      # Y = T.flatten(batch['political'])  #
      Y = batch['political']   # [3]
      print(X)
      print(Y)
  print("\n==============================")

  print("\nEnd demo ")

if __name__ == "__main__":
  main()

The code is simply saved with the filename "demo.py". The code should succesfully execute once the command 'python demo.py' is executed on a command prompt screen. I use Anaconda Prompt which has Torch (v 1.10) installed.
I have tried numerous methods to get the above working, but I only get an error which says:
Source data looks like:
1 0  0.171429  1 0 0  0.966805  0
0 1  0.085714  0 1 0  0.188797  1
 . . .

Creating Dataset and DataLoader

---------------------------------------------------------------------------

IndexError                                Traceback (most recent call last)

<ipython-input-8-cfb1177991f2> in <module>()
     81
     82 if __name__ == "__main__":
---> 83   main()

4 frames

<ipython-input-8-cfb1177991f2> in main()
     59
     60   train_file = "people_train.txt"
---> 61   train_ds = PeopleDataset(train_file, num_rows=8)
     62
     63   bat_size = 3

<ipython-input-8-cfb1177991f2> in __init__(self, src_file, num_rows)
     20     x_tmp = np.loadtxt(src_file, max_rows=num_rows,
     21       usecols=range(0,7), delimiter="\t",
---> 22       skiprows=0, dtype=np.float32)
     23     y_tmp = np.loadtxt(src_file, max_rows=num_rows,
     24       usecols=7, delimiter="\t", skiprows=0,

/usr/local/lib/python3.7/dist-packages/numpy/lib/npyio.py in loadtxt(fname, dtype, comments, delimiter, converters, skiprows, usecols, unpack, ndmin, encoding, max_rows)
   1137         # converting the data
   1138         X = None
-> 1139         for x in read_data(_loadtxt_chunksize):
   1140             if X is None:
   1141                 X = np.array(x, dtype)

/usr/local/lib/python3.7/dist-packages/numpy/lib/npyio.py in read_data(chunk_size)
   1058                 continue
   1059             if usecols:
-> 1060                 vals = [vals[j] for j in usecols]
   1061             if len(vals) != N:
   1062                 line_num = i + skiprows + 1

/usr/local/lib/python3.7/dist-packages/numpy/lib/npyio.py in <listcomp>(.0)
   1058                 continue
   1059             if usecols:
-> 1060                 vals = [vals[j] for j in usecols]
   1061             if len(vals) != N:
   1062                 line_num = i + skiprows + 1

IndexError: list index out of range

I am not able to see which part of the index is wrong, as I don't feel there seem to be anything wrong with the indexing. Can someone please help me ?

Output:
Your data seems to be space-separated, not tab-separated. So, when you specify delimiter="\t", the entire row is read as a single column. But because of usecols=range(0,7), NumPy expects there to be seven columns, and throws an error when trying to iterate over them.
To fix this, either change the whitespaces to tabs in your data, or change the delimiter argument to delimiter=" ".


Instruction: Why loss function always return zero after first epoch?

Input:
Why the loss function is always printing zero after the first epoch?
I suspect it's because of loss = loss_fn(outputs, torch.max(labels, 1)[1]).
But if I use loss = loss_fn(outputs, labels), I will get the error
RuntimeError: 0D or 1D target tensor expected, multi-target not supported

.
nepochs = 5

losses = np.zeros(nepochs)

loss_fn = nn.CrossEntropyLoss()

optimizer = optim.Adam(modell.parameters(), lr = 0.001)

for epoch in range(nepochs):

    running_loss = 0.0
    n = 0

    for data in train_loader:

        #single batch
        if(n == 1):
            break;

        inputs, labels = data

        optimizer.zero_grad()

        outputs = modell(inputs)

        #loss = loss_fn(outputs, labels)
        loss = loss_fn(outputs, torch.max(labels, 1)[1])
        loss.backward()
        optimizer.step()

        running_loss += loss.item()
        n += 1

    losses[epoch] = running_loss / n
    print(f"epoch: {epoch+1} loss: {losses[epoch] : .3f}")

The model is:
def __init__(self, labels=10):
    super(Classifier, self).__init__()
    self.fc = nn.Linear(3 * 64 * 64, labels)

def forward(self, x):
    out = x.reshape(x.size(0), -1)
    out = self.fc (out)
    return out

Any idea?
The labels are a 64 elements tensor like this:
tensor([[7],[1],[ 2],[3],[ 2],[9],[9],[8],[9],[8],[ 1],[7],[9],[2],[ 5],[1],[3],[3],[8],[3],[7],[1],[7],[9],[8],[ 8],[3],[7],[ 5],[ 1],[7],[3],[2],[1],[ 3],[3],[2],[0],[3],[4],[0],[7],[1],[ 8],[4],[1],[ 5],[ 3],[4],[3],[ 4],[8],[4],[1],[ 9],[7],[3],[ 2],[ 6],[4],[ 8],[3],[ 7],[3]])

Output:
Usually loss calculation is loss = loss_fn(outputs, labels) and here outputs is as following:
_ , outputs = torch.max(model(input), 1)
or
outputs = torch.max(predictions, 1)[0]

Common practice is modifying outputs instead of labels:

torch.max() returns a namedtuple (values, indices) where values is
the maximum value of each row of the input tensor in the given
dimension dim. And indices is the index location of each maximum value found (argmax).

In your code snippet the labels is not indices of the labels, so when you calculate the loss, the function should look like this:
loss = loss_fn(torch.max(outputs, 1)[0], labels)


Instruction: Detectron2: No instances in prediction

Input:
I'm trying to train Detectron2 on a custom dataset that I annotated with coco-annotator. After training I wanted to predict Instances of my Image, but I dont get any shown.
Training:
from detectron2.engine import DefaultTrainer

cfg = get_cfg()
cfg.merge_from_file(model_zoo.get_config_file("COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml"))
cfg.DATASETS.TRAIN = ("TrashTron_train",)
cfg.DATASETS.TEST = ("TrashTron_val",)
# cfg.DATASETS.TEST = ()
cfg.DATALOADER.NUM_WORKERS = 2
cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url("COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml")  # Let training initialize from model zoo
cfg.SOLVER.IMS_PER_BATCH = 2
cfg.SOLVER.BASE_LR = 0.00025  # pick a good LR
cfg.SOLVER.MAX_ITER = 300    # 300 iterations seems good enough for this toy dataset; you will need to train longer for a practical dataset
cfg.SOLVER.STEPS = []        # do not decay learning rate
cfg.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 512   # faster, and good enough for this toy dataset (default: 512)
cfg.MODEL.ROI_HEADS.NUM_CLASSES = 24  # only has one class (ballon). (see https://detectron2.readthedocs.io/tutorials/datasets.html#update-the-config-for-new-datasets)
# NOTE: this config means the number of classes, but a few popular unofficial tutorials incorrect uses num_classes+1 here.

os.makedirs(cfg.OUTPUT_DIR, exist_ok=True)
trainer = DefaultTrainer(cfg)
trainer.resume_or_load(resume=False)
trainer.train()

Prediction:
test_data = [{'1191.jpg': '/content/datasets/val/1191.jpg',
              'image_id': 1308}]

cfg = get_cfg()
cfg.merge_from_file(model_zoo.get_config_file("COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml"))
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.7   # set a custom testing threshold
cfg.MODEL.WEIGHTS = os.path.join(cfg.OUTPUT_DIR, "model_final.pth")  # path to the model we just trained
cfg.MODEL.ROI_HEADS.NUM_CLASSES = 24
predictor = DefaultPredictor(cfg)
outputs = predictor(im)
# print(outputs["instances"].pred_densepose)
im = cv2.imread(test_data[0]["1191.jpg"])

v = Visualizer(im[:, :, ::-1],
               metadata=MetadataCatalog.get(cfg.DATASETS.TRAIN[0]),
               scale=0.5,
               instance_mode=ColorMode.IMAGE_BW)
out = v.draw_instance_predictions(outputs["instances"].to("cpu"))
img = cv2.cvtColor(out.get_image()[:, :, ::-1], cv2.COLOR_RGBA2RGB)
plt.imshow(img)

The corresponding image is shown, but no instances.
Any suggestions? The overall evaluation scores aren't that great, but I picked the best class and there I also dont get any predictions...

Output:
I would try to lower the threshold, since you have said that overall training scores were not great.
In this answer in official repo, following code is suggested to change the threshold:
cfg.MODEL.TENSOR_MASK.SCORE_THRESH_TEST = 0.5

at another answer at the same thread, other thresholds are modified as well.
cfg.MODEL.RETINANET.SCORE_THRESH_TEST = args.confidence_threshold
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = args.confidence_threshold
cfg.MODEL.PANOPTIC_FPN.COMBINE.INSTANCES_CONFIDENCE_THRESH = args.confidence_threshold


Instruction: FiftyOne Save Image with Overlaid Detections

Input:
I am using Pytorch and Fiftyone to process image detections and then visualize these image detections around people like so:

However, I am having difficulty saving this in an easily viewable manner. I want to be able to save the processed image with the bounding boxes overlaid onto the image through the script, which I can only do now by right clicking and downloading the image from the application above. FiftyOne provides multiple options for exporting data: https://voxel51.com/docs/fiftyone/user_guide/export_datasets.html#supported-formats, but all of these export the detection for use in another script (by saving the images and detections seperately in a .txt/.json/etc file) rather than a 'final visualization' image. How can I save the image you see above (including the detection boxes) using FiftyOne? If there is no built in method, can I export it to another type of dataset and save the detections there?

Output:
FiftyOne has this capability built-in allowing you to draw labels on samples and save them to disk for any dataset, view, or even just individual samples: https://voxel51.com/docs/fiftyone/user_guide/draw_labels.html
In general, it can be as simple as:
import fiftyone as fo

# The Dataset or DatasetView containing the samples you wish to draw
dataset_or_view = fo.Dataset(...)

# The directory to which to write the annotated media
output_dir = "/path/for/output"

# The list of `Label` fields containing the labels that you wish to render on
# the source media (e.g., classifications or detections)
label_fields = ["ground_truth", "predictions"]  # for example

# Render the labels!
dataset_or_view.draw_labels(output_dir, label_fields=label_fields)

The draw_labels() method also accepts a DrawConfig that provides a lot of options for how to render the labels when drawing them.


Instruction: Colab Notebook: Cannot import name 'container_abcs' from 'torch._six'

Input:
I'm trying to run the deit colab notebook found here:
https://colab.research.google.com/github/facebookresearch/deit/blob/colab/notebooks/deit_inference.ipynb
but I'm running into an issue in the second cell, specifically the import timm line, which returns this:
ImportError: cannot import name 'container_abcs' from 'torch._six'

Output:
Issue related to this error here:

Try a specific version of timm library:
!pip install timm==0.3.2


Instruction: Pretrained lightning-bolts VAE not doing proper inference on training dataset

Input:
I'm using the CIFAR-10 pre-trained VAE from lightning-bolts. It should be able to regenerate images with the quality shown on this picture taken from the docs (LHS are the real images, RHS are the generated)

However, when I write a simple script that loads the model, the weights, and tests it over the training set, I get a much worse reconstruction (top row are real images, bottom row are the generated ones):

Here is a link to a self-contained colab notebook that reproduces the steps I've followed to produce the pictures.
Am I doing something wrong on my inference process? Could it be that the weights are not as "good" as the docs claim?
Thanks!

Output:
First, the image from the docs you show is for the AE, not the VAE.  The results for the VAE look much worse:

https://pl-bolts-weights.s3.us-east-2.amazonaws.com/vae/vae-cifar10/vae_output.png
Second, the docs state "Both input and generated images are normalized versions as the training was done with such images."  So when you load the data you should specify normalize=True.  When you plot your data, you will need to 'unnormalize' the data as well:
from pl_bolts.datamodules import CIFAR10DataModule
from pl_bolts.models.autoencoders import VAE
from pytorch_lightning import Trainer
import matplotlib.pyplot as plt
import numpy as np
import torch
from torchvision import transforms

torch.manual_seed(17)
np.random.seed(17)

vae = VAE(32, lr=0.00001)
vae = vae.from_pretrained("cifar10-resnet18")

dm = CIFAR10DataModule(".", normalize=True)
dm.prepare_data()
dm.setup("fit")
dataloader = dm.train_dataloader()

print(dm.default_transforms())
mean = torch.tensor(dm.default_transforms().transforms[1].mean)
std = torch.tensor(dm.default_transforms().transforms[1].std)
unnormalize = transforms.Normalize((-mean / std).tolist(), (1.0 / std).tolist())

X, _ = next(iter(dataloader))
vae.eval()
X_hat = vae(X)

fig, axes = plt.subplots(2, 10, figsize=(10, 2))
for i in range(10):
  ax_real = axes[0][i]
  ax_real.imshow(np.transpose(unnormalize(X[i]), (1, 2, 0)))
  ax_real.get_xaxis().set_visible(False)
  ax_real.get_yaxis().set_visible(False)

  ax_gen = axes[1][i]
  ax_gen.imshow(np.transpose(unnormalize(X_hat[i]).detach().numpy(), (1, 2, 0)))
  ax_gen.get_xaxis().set_visible(False)
  ax_gen.get_yaxis().set_visible(False)

Which gives something like this:

Without normalization it looks like:


Instruction: BERT Domain Adaptation

Input:
I am using transformers.BertForMaskedLM to further pre-train the BERT model on my custom dataset. I first serialize all the text to a .txt file by separating the words by a whitespace. Then, I am using transformers.TextDataset to load the serialized data with a BERT tokenizer given as tokenizer argument. Then, I am using BertForMaskedLM.from_pretrained() to load the pre-trained model (which is what transformers library presents). Then, I am using transformers.Trainer to further pre-train the model on my custom dataset, i.e., domain adaptation, for 3 epochs. I save the model with trainer.save_model(). Then, I want to load the further pre-trained model to get the embeddings of the words in my custom dataset. To load the model, I am using AutoModel.from_pretrained() but this pops up a warning.
Some weights of the model checkpoint at {path to my further pre-trained model} were not used when initializing BertModel

So, I know why this pops up. Because I further pre-trained using transformers.BertForMaskedLM but when I load with transformers.AutoModel, it loads it as transformers.BertModel. What I do not understand is if this is a problem or not. I just want to get the embeddings, e.g., embedding vector with a size of 768.

Output:
You saved a BERT model with LM head attached. Now you are going to load the serialized file into a standalone BERT structure without any extra element and the warning is issued. This is pretty normal and there is no Fatal error to do so! You can check the list of unloaded params like below:
from transformers import BertTokenizer, BertModel
from transformers import BertTokenizer, BertLMHeadModel, BertConfig
import torch

lmbert = BertLMHeadModel.from_pretrained('bert-base-cased', config=config)
lmbert.save_pretrained('you_desired_path/BertLMHeadModel')
lmbert_params = []
for name, param in lmbert.named_parameters():
    lmbert_params.append(name)
bert = BertModel.from_pretrained('you_desired_path/BertLMHeadModel')
bert_params = []
for name, param in bert.named_parameters():
    bert_params.append(name)
params_ralated_to_lm_head = [param_name for param_name in lmbert_params if param_name.replace('bert.', '') not in bert_params]
params_ralated_to_lm_head

output:
['cls.predictions.bias',
 'cls.predictions.transform.dense.weight',
 'cls.predictions.transform.dense.bias',
 'cls.predictions.transform.LayerNorm.weight',
 'cls.predictions.transform.LayerNorm.bias']


Instruction: manually computing cross entropy loss in pytorch

Input:
I am trying to compute cross_entropy loss manually in Pytorch for an encoder-decoder model.
I used the code posted here to compute it: Cross Entropy in PyTorch
I updated the code to discard padded tokens (-100). The final code is this:
class compute_crossentropyloss_manual:
    """
    y0 is the vector with shape (batch_size,C)
    x shape is the same (batch_size), whose entries are integers from 0 to C-1
    """
    def __init__(self, ignore_index=-100) -> None:
        self.ignore_index=ignore_index

    def __call__(self, y0, x):
        loss = 0.
        n_batch, n_class = y0.shape
        # print(n_class)
        for y1, x1 in zip(y0, x):
            class_index = int(x1.item())
            if class_index == self.ignore_index:  # <------ I added this if-statement
                continue
            loss = loss + torch.log(torch.exp(y1[class_index])/(torch.exp(y1).sum()))
        loss = - loss/n_batch
        return loss

To verify that it works fine, I tested it on a text generation task, and I computed the loss using pytorch.nn implementation and using this code.
The loss values are not identical:
using nn.CrossEntropyLoss:

Using the code from the link above:

Am I missing something?
I tried to get the source code of nn.CrossEntropyLoss but I wasn't able. In this link nn/functional.py at line 2955, you will see that the function points to another cross_entropy loss called torch._C._nn.cross_entropy_loss; I can't find this function in the repo.
Edit:
I noticed that the differences appear only when I have -100 tokens in the gold.
Demo example:
y = torch.randint(1, 50, (100, 50), dtype=float)
x = torch.randint(1, 50, (100,))

x[40:] = -100
print(criterion(y, x).item())
print(criterion2(y, x).item())
> 25.55788695847976
> 10.223154783391905

and when we don't have -100:
x[40:] = 30 # any positive number
print(criterion(y, x).item())
print(criterion2(y, x).item())
> 24.684453267596453
> 24.684453267596453

Output:
I solved the problem by updating the code. I discarded before the -100 tokens (the if-statement above), but I forgot to reduce the hidden_state size (which is called n_batch in the code above). After doing that, the loss numbers are identical to the nn.CrossEntropyLoss values. The final code:
class CrossEntropyLossManual:
    """
    y0 is the vector with shape (batch_size,C)
    x shape is the same (batch_size), whose entries are integers from 0 to C-1
    """
    def __init__(self, ignore_index=-100) -> None:
        self.ignore_index=ignore_index

    def __call__(self, y0, x):
        loss = 0.
        n_batch, n_class = y0.shape
        # print(n_class)
        for y1, x1 in zip(y0, x):
            class_index = int(x1.item())
            if class_index == self.ignore_index:
                n_batch -= 1
                continue
            loss = loss + torch.log(torch.exp(y1[class_index])/(torch.exp(y1).sum()))
        loss = - loss/n_batch
        return loss


Instruction: pytorch derivative returns none on .grad

Input:
i1 = tr.tensor(0.0, requires_grad=True)
i2 = tr.tensor(0.0, requires_grad=True)
x =  tr.tensor(2*(math.cos(i1)*math.cos(i2) - math.sin(i1)*math.sin(i2)) + 3*math.cos(i1),requires_grad=True)
y =  tr.tensor(2*(math.sin(i1)*math.cos(i2) + math.cos(i1)*math.sin(i2)) + 3*math.sin(i1),requires_grad=True)

z = (x - (-2))**2 + (y - 3)**2
z.backward()
dz_t1 = i1.grad
dz_t2 = i2.grad
print(dz_t1)
print(dz_t2)

im trying to run the following code, but im facing an issue after z.backward(). i1.grad and i1.grad return none. from what i understand the cause of this issue is with the way backward() is evaluated in torch. so something along the lines of i1.retain_grad() has to be used to avoid this issue, i tried doing that but i still get none. i1.retain_grad and i2.retain_grad() were placed before z.backward() and after z.backward() and i still get none as an answer. whats happening exactly and how do i fix it? y.grad and x.grad work fine.

Output:
Use:
i1 = tr.tensor(0.0, requires_grad=True)
i2 = tr.tensor(0.0, requires_grad=True)
x =  2*(torch.cos(i1)*torch.cos(i2) - torch.sin(i1)*torch.sin(i2)) + 3*torch.cos(i1)
y =  2*(torch.sin(i1)*torch.cos(i2) + torch.cos(i1)*torch.sin(i2)) + 3*torch.sin(i1)
z = (x - (-2))**2 + (y - 3)**2
z.backward()
dz_t1 = i1.grad
dz_t2 = i2.grad
print(dz_t1)
print(dz_t2)

Here, using torch.sin and torch.cos ensures that the output is a torch tensor that is connected to i1 and i2 in the computational graph. Also, creating x and y using torch.tensor like you did detaches them from the existing graph, which again prevents gradients from flowing back through to i1 and i2.


Instruction: Pytorch Geometric Expected all tensors to be on the same device [demo code not working]

Input:
I wanted to try out the link prediction functionality demonstrated here. Here are my versions:
PyTorch Geometric v2.0.2
PyTorch v1.9.0+cu111

I'm very baffled why cuda:0 is printed for every tensor yet I see the error when I pass the data through RandomLinkSplit.
import torch

import torch_geometric.transforms as T
from torch_geometric.nn import GCNConv
from torch_geometric.datasets import Planetoid
from torch_geometric.utils import negative_sampling

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
transform = T.Compose([
    T.NormalizeFeatures(),
    T.ToDevice(device),
])

dataset = Planetoid(root='/tmp/Planetoid', name='Cora', transform=transform)

data = dataset[0]
print(data.to_dict())
print(data.keys)

transform = T.RandomLinkSplit(num_val=0.05, num_test=0.1, is_undirected=True,)
train_data, val_data, test_data = transform(data)

output:
{'x': tensor([[0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        ...,
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.]], device='cuda:0'), 'edge_index': tensor([[   0,    0,    0,  ..., 2707, 2707, 2707],
        [ 633, 1862, 2582,  ...,  598, 1473, 2706]], device='cuda:0'), 'y': tensor([3, 4, 4,  ..., 3, 3, 3], device='cuda:0'), 'train_mask': tensor([ True,  True,  True,  ..., False, False, False], device='cuda:0'), 'val_mask': tensor([False, False, False,  ..., False, False, False], device='cuda:0'), 'test_mask': tensor([False, False, False,  ...,  True,  True,  True], device='cuda:0')}
['val_mask', 'test_mask', 'edge_index', 'train_mask', 'x', 'y']

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
/tmp/ipykernel_72/414574324.py in <module>
     20
     21 transform = T.RandomLinkSplit(num_val=0.05, num_test=0.1, is_undirected=True,)
---> 22 train_data, val_data, test_data = transform(data)

/usr/local/lib/python3.7/dist-packages/torch_geometric/transforms/random_link_split.py in __call__(self, data)
    204                 train_edges,
    205                 neg_edge_index[:, num_neg_val + num_neg_test:],
--> 206                 out=train_store,
    207             )
    208             self._create_label(

/usr/local/lib/python3.7/dist-packages/torch_geometric/transforms/random_link_split.py in _create_label(self, store, index, neg_edge_index, out)
    284             if neg_edge_index.numel() > 0:
    285                 edge_label = torch.cat([edge_label, neg_edge_label], dim=0)
--> 286                 edge_index = torch.cat([edge_index, neg_edge_index], dim=-1)
    287             out[self.key] = edge_label
    288             out[f'{self.key}_index'] = edge_index

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking arugment for argument tensors in method wrapper__cat)

Output: The issue was indeed a bug. Thank you for reporting it.


Instruction: Give two inputs to torch::jit::script::Module forward method

Input:
I am trying to build and train a network in python using pytorch . My forward method takes two inputs as follows :
def forward(self, x1, x2):

I trained this model in python and saved using torch.jit.script .
Then I load this model in c++ using the torch::jit::load.
How do I now pass the inputs to the model in c++ ?
If I try passing two separate tensors to the forward method like the following
    std::vector<torch::jit::IValue> inputs1{tensor1};
    std::vector<torch::jit::IValue> inputs2{tensor2};
    at::Tensor output = module.forward(inputs1,inputs2).toTensor();

then I receive an error saying that the method forward expects 1 argument, 2 provided.
I can't also concatenate the two tensors since the shapes are different in all axis.

Output:
The problem is by concatenating the two tensors and giving the concatenated tensor as input to the model. Then in the forward method, we can create two separate tensors using the concatenated tensor and use them separately for the  output computation.
For concatenation to work, I appended the tensors with 0's so that they are of the same size in all axis except the one in which concatenation is to be done.


Instruction: RuntimeError: mat1 and mat2 shapes cannot be multiplied (1280x5 and 6400x4096)?

Input:
Defining Alexnet using the following code,I can train successfully.But when I want to see the output of each layer，it will be an error ‘RuntimeError: mat1 and mat2 shapes cannot be multiplied (1280x5 and 6400x4096)?’
class AlexNet(nn.Module):
    def __init__(self):
      super(AlexNet, self).__init__()
      self.conv = nn.Sequential(
          nn.Conv2d(1, 96, 11, 4),
          nn.ReLU(),
          nn.MaxPool2d(3, 2),
          nn.Conv2d(96, 256, 5, 1, 2),
          nn.ReLU(),
          nn.MaxPool2d(3, 2),
          nn.Conv2d(256, 384, 3, 1, 1),
          nn.ReLU(),
          nn.Conv2d(384, 384, 3, 1, 1),
          nn.ReLU(),
          nn.Conv2d(384, 256, 3, 1, 1),
          nn.ReLU(),
          nn.MaxPool2d(3, 2)
         )
      self.fc = nn.Sequential(
          nn.Linear(256*5*5, 4096),
          nn.ReLU(),
          nn.Dropout(0.5),
          nn.Linear(4096, 4096),
          nn.ReLU(),
          nn.Dropout(0.5),
          nn.Linear(4096, 10)
        )

    def forward(self, img):
      feature = self.conv(img)
      output = self.fc(feature.view(img.shape[0], -1))
      return output

X=torch.randn(1,1,224,224)
for name,layer in net.named_children():
  X=layer(X)
  print(name,X.shape)

Could u help me？

Output:
You forgot to flatten the output array of self.conv in the for cycle. You can split it into two cycles, one for the convolution layers, and one for the fully connected ones.

X = torch.randn(1, 1, 224, 224)
for name, layer in net.conv.named_children():
  X = layer(X)
  print(name, X.shape)

X = X.flatten()  # or X = X.view(X.shape[0], -1)

for name, layer in net.fc.named_children():
  X = layer(X)
  print(name, X.shape)


Instruction: Pytorch is throwing an error RuntimeError: result type Float can't be cast to the desired output type Long

Input:
How should I get rid of the following error?
>>> t = torch.tensor([[1, 0, 1, 1]]).T
>>> p = torch.rand(4,1)
>>> torch.nn.BCEWithLogitsLoss()(p, t)

The above code is throwing the following error:
RuntimeError: result type Float can't be cast to the desired output type Long

Output:
BCEWithLogitsLoss requires its target to be a float tensor, not long. So you should specify the type of t tensor by dtype=torch.float32:
import torch

t = torch.tensor([[1, 0, 1, 1]], dtype=torch.float32).T
p = torch.rand(4,1)
loss_fn = torch.nn.BCEWithLogitsLoss()

print(loss_fn(p, t))

Output:
tensor(0.5207)


Instruction: OSError: [Errno 22] Invalid argument | _pickle.UnpicklingError: pickle data was truncated

Input:
When trying to train a ResNet I get this error. Any help as to why this happens would be appreciated.
This happens when I try to iterate through the Dataloader:
File "C:\Users\JCout\AppData\Local\Temp/ipykernel_2540/2174299330.py", line 1, in <module>
    runfile('C:/Users/JCout/Documents/GitHub/Hybrid_resnet/transfer_learning.py', wdir='C:/Users/JCout/Documents/GitHub/Hybrid_resnet')

  File "C:\Users\JCout\anaconda3\lib\site-packages\debugpy\_vendored\pydevd\_pydev_bundle\pydev_umd.py", line 167, in runfile
    execfile(filename, namespace)

  File "C:\Users\JCout\anaconda3\lib\site-packages\debugpy\_vendored\pydevd\_pydev_imps\_pydev_execfile.py", line 25, in execfile
    exec(compile(contents + "\n", file, 'exec'), glob, loc)

  File "C:/Users/JCout/Documents/GitHub/Hybrid_resnet/transfer_learning.py", line 24, in <module>
    model, train_loss, test_loss = train.train_model(training, testing,

  File "C:\Users\JCout\Documents\GitHub\Hybrid_resnet\train.py", line 70, in train_model
    train_stats = train.train_step(model, criterion, optimizer, train_loader)

  File "C:\Users\JCout\Documents\GitHub\Hybrid_resnet\train.py", line 121, in train_step
    for i, (x_imgs, labels) in enumerate(train_loader):

  File "C:\Users\JCout\anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 359, in __iter__
    return self._get_iterator()

  File "C:\Users\JCout\anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 305, in _get_iterator
    return _MultiProcessingDataLoaderIter(self)

  File "C:\Users\JCout\anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 918, in __init__
    w.start()

  File "C:\Users\JCout\anaconda3\lib\multiprocessing\process.py", line 121, in start
    self._popen = self._Popen(self)

  File "C:\Users\JCout\anaconda3\lib\multiprocessing\context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)

  File "C:\Users\JCout\anaconda3\lib\multiprocessing\context.py", line 327, in _Popen
    return Popen(process_obj)

  File "C:\Users\JCout\anaconda3\lib\multiprocessing\popen_spawn_win32.py", line 93, in __init__
    reduction.dump(process_obj, to_child)

  File "C:\Users\JCout\anaconda3\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)

OSError: [Errno 22] Invalid argument

I'm also getting this traceback after the error which I find a bit weird since I'm not using pickle at all. The data consists of 2 .tif files and 2 .mat files for the data/targets.
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\JCout\anaconda3\lib\multiprocessing\spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "C:\Users\JCout\anaconda3\lib\multiprocessing\spawn.py", line 126, in _main
    self = reduction.pickle.load(from_parent)
_pickle.UnpicklingError: pickle data was truncated
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\JCout\anaconda3\lib\multiprocessing\spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "C:\Users\JCout\anaconda3\lib\multiprocessing\spawn.py", line 126, in _main
    self = reduction.pickle.load(from_parent)
_pickle.UnpicklingError: pickle data was truncated

Output:
Not entirely sure what was happening, but I had
pin_memory = True
num_workers = 4

within the DataLoader. Removing these removed the error.


Instruction: I want to see data from torch.utils.data.DataLoader. How Can I?

Input:
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import torchvision
import torchvision.transforms as transforms

train_set = torchvision.datasets.MNIST(root = './data/MNIST',train = True,download = True,\transform = transfroms.Compose([transfroms.ToTensor()])

print(len(train_set))
# 60000

train_loader = torch.utils.data.DataLoader(train_set, batch_size=100)
print(len(train_loader))
# 600

It seems like because of the batch_size, length of train_loader decreased.
I think there are 100 tensors and one classification in a batch.
I just want to see the elements or shape of it. How can I do?
Also,
### Model Omitted ###
model = ConvNet().to(device)
criterion = nn.CrossEntropyLoss().to(device)
optimizer = torch.optim.Adam(model.parameters(), lr = learning_rate)

for epoch in range(5):
    avg_cost = 0
    for data, target in train_loader:
        data = data.to(device)
        target = target.to(device)
        optimizer.zero_grad()
        hypothesis = model(data)
        cost = criterion(hypothesis, target)
        cost.backward()
        optimizer.step()
        avg_cost += cost / len(train_loader)

    print('[Epoch: {:>4}] cost = {:>.9}'.format(epoch + 1, avg_cost))

I think the training per epoch trains with 60,000 tensors right? Then I think the avg_cost should be divided by 60,000, not 600(which is len(train_loader))... Am I wrong with it?

Output:
You can get one batch of train data from trainloader using the code below and you can easily check it's shape. I hope this may help to get what you want.
batch= iter(trainloader)
images, labels = batch.next()

print(images.shape)
# torch.Size([num_samples, in_channels, H, W])

print(labels.shape)


Instruction: convert a python list to tensor pytorch

Input:
I want to convert a list of pixel values to tensor, but I got an error. My code calculate the pixel values (RGB) for each detected object in the image. How we can convert the list to tensor??
my code:
         cropped_images =[]

         imgs = PIL.Image.open(img_path).convert('RGB')
         #print(img_path)
         image_width, image_height = imgs.size
         imgArrays = np.array(imgs)

         X = (xCenter*image_width)
         Y = (yCenter*image_height)
         W = (Width*image_width)
         H = (Height*image_height)
         cropped_image = np.zeros((image_height, image_width))

         for i in range(len(X)):
                 x1, y1, w, h = X[i], Y[i], W[i], H[i]
                 x_start = int(x1 - (w/2))
                 y_start = int(y1 - (h/2))
                 x_end = int(x_start + w)
                 y_end = int(y_start + h)

                 temp = imgArrays[y_start: y_end, x_start: x_end]

                 cropped_image_pixels = torch.as_tensor(temp)
                 cropped_images.append(cropped_image_pixels)
                 stacked_tensor = torch.stack(cropped_images)

                 print(stacked_tensor)

the error:
RuntimeError                              Traceback (most recent call last)
<ipython-input-82-653a155c3b71> in <module>()
    130
    131 if __name__=="__main__":
--> 132     main()

2 frames
<ipython-input-80-670335a0656c> in __getitem__(self, idx)
     76                  cropped_image_pixels = torch.as_tensor(temp)
     77                  cropped_images.append(cropped_image_pixels)
---> 78                  stacked_tensor = torch.stack(cropped_images)
     79
     80                  print(stacked_tensor)

RuntimeError: stack expects each tensor to be equal size, but got [506, 343, 3] at entry 0 and [520, 334, 3] at entry 1

Output:
list of tensors has two tensors
and it's clear that both don't have same size
torch.stack(tensors, dim=0, *, out=None) → Tensor
Concatenates a sequence of tensors along a new dimension.

All tensors need to be of the same size.

you can use this pseudo code
import torchvision.transforms as transforms
.
.
.
.
temp=[]
for img_name in LIST:
    img=cv2.resize(img,(H,W))
    temp.append(img)

train_x=np.asarray(temp)

transform = transforms.Compose(
    [transforms.ToTensor(),

check doc


Instruction: What are the main reasons why some network parameters might become nan after calling optimizer.step in Pytorch?

Input:
I am trying to understand why one or two parameters in my Pytorch neural network occasionally become nan after calling optimizer.step().
I have already checked the gradients after calling .backward() and just before calling the optimizer, and they neither contain nans nor are very large. I am doing gradient clipping, but I don't think that this can be responsible since the gradients still look fine after clipping. I am using single-precision floats everywhere.
This behavior happens randomly every hundred thousand epochs or so, and is proving very difficult to debug. Unfortunately the code is too long to reproduce here and I haven't been able to replicate the problem in a smaller example.
If anyone can suggest possible issues I haven't mentioned above, that would be super helpful.
Thanks!

Output: This ended up being ignorance on my part: there were Infs in the gradients that were evading my diagnostic code, as I didn't realize Pytorch's .isnan() method doesn't detect them.


Instruction: pytorch: log_softmax base 2?

Input:
I want to get surprisal values from logit outputs from PyTorch, using log base 2.
One way to do this, given a logits tensor, is:
probs = nn.functional.softmax(logits, dim = 2)
surprisals = -torch.log2(probs)

However, PyTorch provides a function that combines log and softmax, which is faster than the above:
surprisals = -nn.functional.log_softmax(logits, dim = 2)

But this seems to return values in base e, which I don't want. Is there a function like log_softmax, but which uses base 2? I have tried log2_softmax and log_softmax2, neither of which seems to work, and haven't had any luck finding documentation online.

Output:
How about just using the fact that logarithm bases can be easily altered by the following mathematical identity

 is what F.log_softmax() is giving you. All you need to do is
surprisals = - (1 / torch.log(2.)) * nn.functional.log_softmax(logits, dim = 2)

Its just a scalar multiplication. So, it hardly has any performance penalty.


Instruction: Output Dimensions of convolution in PyTorch

Input:
The size of my input images are 68 x 224 x 3 (HxWxC), and the first Conv2d layer is defined as
conv1 = torch.nn.Conv2d(3, 16, stride=4, kernel_size=(9,9)).
Why is the size of the output feature volume 16 x 15 x 54? I get that there are 16 filters, so there is a 16 in the front, but if I use [(W−K+2P)/S]+1 to calculate dimensions, the dimensions are not divisible.
Can someone please explain?

Output:
The calculation of feature maps is [(W−K+2P)/S]+1 and here [] brackets means floor division. In your example padding is zero, so the calculation is [(68-9+2*0)/4]+1 ->[14.75]=14 -> [14.75]+1 = 15 and [(224-9+2*0)/4]+1 -> [53.75]=53 -> [53.75]+1 = 54.
import torch

conv1 = torch.nn.Conv2d(3, 16, stride=4, kernel_size=(9,9))
input = torch.rand(1, 3, 68, 224)

print(conv1(input).shape)
# torch.Size([1, 16, 15, 54])

You may see different formulas too calculate feature maps.
In PyTorch:

In general, you may see this:

However the result of both cases are the same


Instruction: Sorting a tensor list in ascending order

Input:
I am working on a facial comparison app that will give me the closest n number of faces to my target face.
I have done this with dlib/face_recognition as it uses numpy arrays, however i am now trying to do the same thing with facenet/pytorch and running into an issue because it uses tensors.
I have created a database of embeddings and I am giving the function one picture to compare to them. What i would like is for it to sort the list from lowest distances to highest, and give me the lowest 5 results or so.
here is the code I am working on that is doing the comparison. at this point i am feeding it a photo and asking it to compare against the embedding database.
def face_match(img_path, data_path): # img_path= location of photo, data_path= location of data.pt
    # getting embedding matrix of the given img
    img_path = (os.getcwd()+'/1.jpg')
    img = Image.open(img_path)
    face = mtcnn(img) # returns cropped face and probability
    emb = resnet(face.unsqueeze(0)).detach() # detech is to make required gradient false

    saved_data = torch.load('data.pt') # loading data.pt file
    embedding_list = saved_data[0] # getting embedding data
    name_list = saved_data[1] # getting list of names
    dist_list = [] # list of matched distances, minimum distance is used to identify the person

    for idx, emb_db in enumerate(embedding_list):
        dist = torch.dist(emb, emb_db)
        dist_list.append(dist)

    namestodistance = list(zip(name_list,dist_list))

    print(namestodistance)

face_match('1.jpg', 'data.pt')

This results in giving me all the names and their distance from the target photo in alphabetical order of the names, in the form of (Adam Smith, tensor(1.2123432)), Brian Smith, tensor(0.6545464) etc. If the 'tensor' wasn't part of every entry I think it would be no problem to sort it. I don't quite understand why its being appended to the entries. I can cut this down to the best 5 by adding [0:5] at the end of dist_list, but I can't figure out how to sort the list, I think the problem is the word tensor being in every entry.
I have tried
for idx, emb_db in enumerate(embedding_list): dist = torch.dist(emb, emb_db) sorteddist = torch.sort(dist)
but for whatever reason this only returns one distance value, and it isn't the smallest one.
idx_min = dist_list.index(min(dist_list)), this works fine in giving me the lowest value and then matching it to a name using namelist[idx_min], therefore giving the best match, but I would like the best 5 matches in order as opposed to just the best match.
Anyone able to solve this ?

Output:
Unfortunately I cannot test your code, but to me it seems like you are operation on a python list of tuples. You can sort that by using a key:
namestodistance = [('Alice', .1), ('Bob', .3), ('Carrie', .2)]
names_top = sorted(namestodistance, key=lambda x: x[1])
print(names_top[:2])

Of course you have to modify the anonymous function in key to return a sortable value instead of e.g. a torch.tensor.
This can be done by using key = lambda x: x[1].item().
Edit: To answer the question that crept up in the comments, we can refactor our code a little. Namely
namestodistance = list(map(lambda x: (x[0], x[1].item()), namestodistance)
names_top = sorted(namestodistance, key=lambda x: x[1])
print(names_top[:2])


Instruction: Intersection over Union used as Metric or Loss

Input: Im currently struggling to understand the use of the IoU. Is the IoU just a Metric to monitor the quality of a network, or is used as a loss function where the value has some impact on the backprop?

Output:
For a measure to be used as a loss function, it must be differentiable, with non-trivial gradients.
For instance, in image classification, accuracy is the most common measure of success. However, if you try to differentiate accuracy, you'll see that the gradients are zero almost everywhere and therefore one cannot train a model with accuracy as a loss function.
Similarly, IoU, in its native form, also has meaningless gradients and cannot be used as a loss function. However, extensions to IoU that preserve gradients exist and can be effectively used as a loss function for training.


Instruction: Extracting Autoencoder features from the hidden layer

Input:
I have developed some code to apply Autoencoder on my dataset, in order to extract hidden features from it. I have a dataset that consists of 84 variables, and they have been normalised.
epochs = 10
batch_size = 128
lr = 0.008

# Convert Input and Output data to Tensors and create a TensorDataset
input = torch.Tensor(input.to_numpy())
output = torch.tensor(output.to_numpy())
data = torch.utils.data.TensorDataset(input, output)

# Split to Train, Validate and Test sets using random_split
number_rows = len(input)    # The size of our dataset or the number of rows in excel table.

test_split = int(number_rows*0.3)
train_split = number_rows - test_split
train_set, test_set = random_split(data, [train_split, test_split])

# Create Dataloader to read the data within batch sizes and put into memory.
train_loader = torch.utils.data.DataLoader(train_set, batch_size=batch_size, shuffle = True)
test_loader = torch.utils.data.DataLoader(test_set, batch_size=batch_size)

The model structure:
# Model structure
class AutoEncoder(nn.Module):
    def __init__(self):
        super(AutoEncoder, self).__init__()

        # Encoder
        self.encoder = nn.Sequential(
            nn.Linear(84, 128),
            nn.Tanh(),
            nn.Linear(128, 64),
            nn.Tanh(),
            nn.Linear(64, 16),
            nn.Tanh(),
            nn.Linear(16, 2),
        )

        # Decoder
        self.decoder = nn.Sequential(
            nn.Linear(2, 16),
            nn.Tanh(),
            nn.Linear(16, 64),
            nn.Tanh(),
            nn.Linear(64, 128),
            nn.Tanh(),
            nn.Linear(128, 84),
            nn.Sigmoid()
        )

    def forward(self, inputs):
        codes = self.encoder(inputs)
        decoded = self.decoder(codes)

        return codes, decoded

Optimiser and Loss function
# Optimizer and loss function
model = AutoEncoder()
optimizer = torch.optim.Adam(model.parameters(), lr=lr)
loss_function = nn.MSELoss()

The training steps:
# Train
for epoch in range(epochs):
    for data, labels in train_loader:
        inputs = data.view(-1, 84)

        # Forward
        codes, decoded = model(inputs)

        # Backward
        optimizer.zero_grad()
        loss = loss_function(decoded, inputs)
        loss.backward()
        optimizer.step()

    # Show progress
    print('[{}/{}] Loss:'.format(epoch+1, epochs), loss.item())

The Autoencoder model is saved as:
# Save
torch.save(model,'autoencoder.pth')

At this point, I would like to ask some help to understand how I could extract the features from the hidden layer. These features extracted from the hidden layer will be used in another classification algorithm.

Output:
You need to place an hook to your model. And you can use this hook to extract features from any layer. However it is a lot easier if you don't use nn.Sequential because it combines the layer together and they act as one. I run your code using this function:
There is a function for Feature Extraction which basically takes model as an input and place a hook using index of layer.
class FE(nn.Module):
  def __init__(self,model_instance, output_layers, *args):
    super().__init__(*args)
    self.output_layers = output_layers

    self.selected_out = OrderedDict()

    self.pretrained = model_instance

    self.fhooks = []
    print("model_instance._modules.keys():",model_instance._modules.keys())

    for i,l in enumerate(list(self.pretrained._modules.keys())):
        print("index:",i, ", keys:",l )
        if i in self.output_layers:

            print("------------------------ > Hook is placed output of :" , l )

            self.fhooks.append(getattr(self.pretrained,l).register_forward_hook(self.forward_hook(l)))

  def forward_hook(self,layer_name):
    def hook(module, input, output):
        self.selected_out[layer_name] = output
    return hook

  def forward(self, x):
    out = self.pretrained(x,None)
    return out, self.selected_out

And to use:
model_hooked=FE(model ,output_layers = [0])

model_instance._modules.keys(): odict_keys(['encoder', 'decoder'])

index: 0 , keys: encoder

------------------------ > Hook is placed output of : encoder

index: 1 , keys: decoder

After placing the hook you can simply put data to new hooked model and it will output 2 values.First one is original output from last layer and second output will be the output from hooked layer
out, layerout = model_hooked(data_sample)

If you want to extract features from a loaders you can use this function:
def extract_features(FE ,layer_name, train_loader, test_loader):
  extracted_features=[]
  lbls=[]

  extracted_features_test=[]
  lbls_test=[]

  for data , target in train_loader:

    out, layerout = FE(data)

    a=layerout[layer_name]

    extracted_features.extend(a)
    lbls.extend(target)

  for data , target in test_loader:

        out, layerout = FE(data)

        a=layerout[layer_name]
        extracted_features_test.extend(a)
        lbls_test.extend(target)

  extracted_features = torch.stack(extracted_features)
  extracted_features_test = torch.stack(extracted_features_test)
  lbls = torch.stack(lbls)
  lbls_test = torch.stack(lbls_test)

  return extracted_features, lbls  ,extracted_features_test, lbls_test

And usage is like this :
Features_TRAINLOADER , lbls ,  Features_TESTLOADER, lbls_test  =extract_features(model_hooked, "encoder",  train_loader, test_loader)


Instruction: Pytorch - skip calculating features of pretrained models for every epoch

Input:
I am used to work with tenserflow - keras but now I am forced to start working with Pytorch for flexibility issues. However, I don't seem to find a pytorch code that is focused on training only the classifciation layer of a model. Is that not a common practice ? Now I have to wait out the calculation of the feature extraction of the same data for every epoch. Is there a way to avoid that ?
# in tensorflow - keras :
from tensorflow.keras.applications import vgg16, MobileNetV2, mobilenet_v2

# Load a pre-trained
pretrained_nn = MobileNetV2(weights='imagenet', include_top=False, input_shape=(Image_size, Image_size, 3))

# Extract features of the training data only once
X = mobilenet_v2.preprocess_input(X)
features_x = pretrained_nn.predict(X)

# Save features for later use
joblib.dump(features_x, "features_x.dat")

# Create a model and add layers
model = Sequential()
model.add(Flatten(input_shape=features_x.shape[1:]))
model.add(Dense(100, activation='relu', use_bias=True))
model.add(Dense(Y.shape[1], activation='softmax', use_bias=False))

# Compile & train only the fully connected model

model.compile( loss="categorical_crossentropy", optimizer=keras.optimizers.Adam(learning_rate=0.001))
history = model.fit(  features_x, Y_train, batch_size=16, epochs=Epochs)

Output:
Assuming you already have the features ìn features_x, you can do something like this to create and train the model:
# create a loader for the data
dataset = torch.utils.data.TensorDataset(features_x, Y_train)
loader = torch.utils.data.DataLoader(dataset, batch_size=16, shuffle=True)

# define the classification model
in_features = features_x.flatten(1).size(1)
model = torch.nn.Sequential(
    torch.nn.Flatten(),
    torch.nn.Linear(in_features=in_features, out_features=100, bias=True),
    torch.nn.ReLU(),
    torch.nn.Linear(in_features=100, out_features=Y.shape[1], bias=False) # Softmax is handled by CrossEntropyLoss below
)
model.train()

# define the optimizer and loss function
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
loss_function = torch.nn.CrossEntropyLoss()

# training loop
for e in range(Epochs):
    for batch_x, batch_y in enumerate(loader):
        optimizer.zero_grad() # clear gradients from previous batch
        out = model(batch_x)  # forward pass
        loss = loss_function(out, batch_y) # compute loss
        loss.backward() # backpropagate, get gradients
        optimizer.step() # update model weights


Instruction: How to make one-hot data compatible with non one-hot?

Input:
I'm making a machine learning model to calculate game win rate on different character combination.
I got error at last line using loss function. I think it's because the input is one-hot vector.
The output of the model doesn't compatile with target data. Because target data is just boolean value, win or lose. Please give me advice to get through this problem. How to make one-hot input compatible with non one-hot?
'''for example, when the number of character is 4 and eahc team member is 2.
   x_data is [ [[0,0,1,0], [0,1,0,0], [1,0,0,0,],[0,1,0,0]],  [game2]...]
                team A1,    temaA2,     temaB1     teamB2
'''

y_data = [[0], [0], [0], [1], [1], [1]] # team blue win: 1, lose : 0
x_train = torch.FloatTensor(x_data)
y_train = torch.FloatTensor(y_data)

class BinaryClassifier(nn.Module):
    def __init__(self):
        super(BinaryClassifier, self).__init__()

        self.layer1 = nn.Sequential(
            nn.Linear(in_features=num_characters, out_features=10, bias=True),
            nn.ReLU(),
            )

        self.layer2 = nn.Sequential(
            nn.Linear(in_features=10, out_features=1, bias=True),
            nn.Sigmoid(),
            )

    def forward(self, x):
        x = self.layer1(x)
        x = self.layer2(x)
        return torch.sigmoid(x)

model = BinaryClassifier()
optimizer = optim.SGD(model.parameters(), lr=1)

nb_epochs = 1000
for epoch in range(nb_epochs + 1):

    hypothesis = model(x_train)
    cost = nn.BCELoss(hypothesis, y_train)

 # RuntimeError: bool value of Tensor with more than one value is ambiguous

Output:
First, your issue is not about One-hot encoding, because the output of your model is a number and Y_data is 0-1, so they're compatible. Your problem is about instantiating the loss.
Therefore, you have to instantiate the loss and then pass arguments:
...
model = BinaryClassifier()
optimizer = torch.optim.SGD(model.parameters(), lr=1)
loss = nn.BCELoss()

nb_epochs = 1000
for epoch in range(nb_epochs + 1):

    hypothesis = model(x_train)
    cost = loss(hypothesis, y_train)

About your x_data, if your data is like:
[[0,0,1,0], [0,1,0,0], [1,0,0,0,], [0,1,0,0],...]

in self.layer1 you should specify in_features with 4.
If x_data is like:
[ [[0,0,1,0], [0,1,0,0], [1,0,0,0,], [0,1,0,0]], [[0,0,1,0], [0,1,0,0], [1,0,0,0,], [0,1,0,0]], ...]

and you want to use a Linear layer, you have to flatten each sample because a linear layer accepts 1-dim input.
For example, the above would be:
[[0,0,1,0,0,1,0,0,1,0,0,0,0,1,0,0], [0,0,1,0,0,1,0,0,1,0,0,0,0,1,0,0], ...]

and in_features=16.
For your information, you can use CNN (Convolutional Neural Net) for 2 and more dimensions inputs, and for series inputs, you can use RNN (Recurrent neural network).
Hope it can be helpful.


Instruction: How does backpropagation work when some layers are frozen in the network?

Input:
Assume there are two models to be used: X and Y. Data is sequentially passed through X and Y. Only the parameters of model Y need to be optimized with respect to a loss computed over output of model Y. Is the following snippet a correct implementation of this requirement. Few specific queries I need answers to:

What does the with torch.no_grad() exactly do?
As only the parameters of model Y are registered with the optimizer do we still need to freeze model X to be correct or is it only required to reduce the computational load?
More generally I want an explanation on how the computation graph and backpropagation behaves in the presence of with torch.no_grad() or when some layers are freezed by setting the corresponding requires_grad parameter to False.
Also comment on whether we can have non-consecutive layers in the network frozen at once.

optimizer = AdamW(model_Y.parameters(), lr= , eps= , ...)

optimizer.zero_grad()
with torch.no_grad():
    A = model_X(data)
B = model_Y(A)
loss = some_function(B)
loss.backward()
optimizer.step()

Output:
torch.no_grad serves as a context manager that disables gradient computation. This is very useful for, e.g.,  inference where no use of .backward() call will be done. It saves both memory and computations.
In your example, you can treat A, the output of model_X as your "inputs": you will not modify anything related to model_X. In this case, you do not care about gradients w.r.t model_X: not w.r.t its parameters nor w.r.t its inputs. It is safe to wrap this call A = model_X(data) with the context of torch.no_grad().
However, in other cases, you might not want to modify the weights of model_X ("freeze" them), but you might still need to propagate gradients through them - if you want to modify elements feeding model_X.
It's all in the chain rule.


Instruction: Get input of fully connected layer of ResNet model during runtime

Input:
Found a Solution, left it as an answer to this question down below :)
Info about the project: Classification task with 2 classes.
I am trying to get the output of the fully connected layer of my model for each image I put into the model during runtime. I plan to use them after the model is done training or testing all images to visualize with UMAP.
The model:
#Load resnet
def get_model():
    model = torchvision.models.resnet50(pretrained=True)
    num_ftrs = model.fc.in_features
    model.fc = nn.Linear(num_ftrs, 2)
    return model

The relevant part of pl module:
class classifierModel(pl.LightningModule):
   def __init__(self, model):
     super().__init__()
     self.model = model
     self.learning_rate = 0.0001

def training_step(self, batch, batch_idx):
        x= batch['image']
        y = batch['targets']
        x_hat = self.model(x)
        output = nn.CrossEntropyLoss()
        loss= output(x_hat,y)
        return loss

def test_step(self, batch, batch_idx):
        x= batch['image']
        y = batch['targets']
        x_hat = self.model(x)

Is it possible to do this by adding a empty list to the init of the pl module and then add the output after x_hat = model(x) is executed?
How would i know if after x_hat = model(x) is executed, the out_features aren't immediatly deleted/discarded ?

Output:
I was able to do this using a forward hook on the avgpool layer and saving the output on each test_step like described here :
#Define Hook:
def get_features(name):
    def hook(model, input, output):
        features[name] = output.detach()
    return hook

Now when I load my model, I register the hook:
#Load resnet model:
def get_model():
    model = models.resnet50(pretrained=True)
    num_ftrs = model.fc.in_features
    model.fc = nn.Linear(num_ftrs, 2)
    model.avgpool.register_forward_hook(get_features('feats')) #register the hook
    return model

I did not need to change the init of the pytorch lightning model but the test step function:
FEATS = []
# placeholder for batch features
features = {}

class classifierModel(pl.LightningModule):
   def __init__(self, model):
     super().__init__()
     self.model = model
     self.learning_rate = 0.0001

def test_step(self, batch,batch_idx):
        x= batch['image']
        y = batch['targets']
        x_hat = self.model(x)
        FEATS.append(features['feats'].cpu().numpy()) #added this line to save output

now we have the output FEATS[0].shape --> (16, 2048, 1, 1) which is what i wanted to get(16 is the batchsize is use).


Instruction: variable out of scope but still works

Input:
I am following the pytorch tutorial here and got to the word embeddings tutorial but there is some code I do not understand.
When constructing n-grams they use the following:
ngrams = [
    (
        [test_sentence[i - j - 1] for j in range(CONTEXT_SIZE)],
        test_sentence[i]
    )
    for i in range(CONTEXT_SIZE, len(test_sentence))
]

This to me does not seem syntactically correct as i is referenced before it is initialized, there is nothing inside the for loop, and the for loop is missing the : at the end. What is going on with this block of code? It does not seem like it should work but it does.

Output:
This is a syntax of python like you can define a list like this:
a_list = [i*2 for i in range(100)]

in your example it is defining a 2-D list, so there are two for loop
like:
b_list = [(a_list) for j in range(2)]

if we combine, we can write them like the example you posted.