Instruction: Gradient of Image in PyTorch - for Gradient Penalty calculation in WGAN

Input:
I am following this Github Repo for the WGAN implementation with Gradient Penalty.
And I am trying to understand the following method, which does the job of unit-testing the gradient-penalty calulations.
def test_gradient_penalty(image_shape):
    bad_gradient = torch.zeros(*image_shape)
    bad_gradient_penalty = gradient_penalty(bad_gradient)
    assert torch.isclose(bad_gradient_penalty, torch.tensor(1.))

    image_size = torch.prod(torch.Tensor(image_shape[1:]))
    good_gradient = torch.ones(*image_shape) / torch.sqrt(image_size)
    good_gradient_penalty = gradient_penalty(good_gradient)
    assert torch.isclose(good_gradient_penalty, torch.tensor(0.))

    random_gradient = test_get_gradient(image_shape)
    random_gradient_penalty = gradient_penalty(random_gradient)
    assert torch.abs(random_gradient_penalty - 1) < 0.1

# Now pass tuple argument for image dimenstion of
# (batch_size, channel, height, width)
test_gradient_penalty((256, 1, 28, 28))

I don't understand the below line
good_gradient = torch.ones(*image_shape) / torch.sqrt(image_size)
In above the torch.ones(*image_shape) is just filling a 4-D Tensor filled up with 1 and then
torch.sqrt(image_size) is just representing the value of tensor(28.)
So, what I am trying to understand why I need to divide the 4-D Tensor by tensor(28.) to get the good_gradient
If I print bad_gradient, it will be a 4-D Tensor as below
tensor([[[[0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          ...,
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.]]],

          ---
          ---

If I print good_gradient, the output will be
tensor([[[[0.0357, 0.0357, 0.0357,  ..., 0.0357, 0.0357, 0.0357],
          [0.0357, 0.0357, 0.0357,  ..., 0.0357, 0.0357, 0.0357],
          [0.0357, 0.0357, 0.0357,  ..., 0.0357, 0.0357, 0.0357],
          ...,
          [0.0357, 0.0357, 0.0357,  ..., 0.0357, 0.0357, 0.0357],
          [0.0357, 0.0357, 0.0357,  ..., 0.0357, 0.0357, 0.0357],
          [0.0357, 0.0357, 0.0357,  ..., 0.0357, 0.0357, 0.0357]]],

          ---
          ---

Output:
For the line
good_gradient = torch.ones(*image_shape) / torch.sqrt(image_size)
First, note the Gradient Penalty term in WGAN is =>
(norm(gradient(interpolated)) - 1)^2
And for the Ideal Gradient (i.e. a Good Gradient), this Penalty term would be 0. i.e.  A Good gradient is one which has its gradient_penalty is as close to 0 as possible
This means the following should satisfy, after considering the L2-Norm of the Gradient
(norm(gradient(x')) -1)^2 = 0
i.e  norm(gradient(x')) = 1
i.e. sqrt(Sum(gradient_i^2) ) = 1
Now if you just continue simplifying the above (considering how norm is calculated, see my note below) math expression, you will end up with
good_gradient = torch.ones(*image_shape) / torch.sqrt(image_size)
Since you are passing the image_shape as (256, 1, 28, 28) - so  torch.sqrt(image_size) in your case  is tensor(28.)
Effectively the above line is dividing each element of A 4-D Tensor like [[[[1., 1. ... ]]]] with a scaler tensor(28.)

Separately, note how norm is calculated

torch.norm without extra arguments performs, what is called a Frobenius norm which is effectively reshaping the matrix into one long vector and returning the 2-norm of that.
Given an M * N matrix, The Frobenius Norm of a matrix is defined as the square root of the sum of the squares of the elements of the matrix.
Input: mat[][] = [[1, 2], [3, 4]]
Output: 5.47723
sqrt(1^2 + 2^2 + 3^2 + 4^2) = sqrt(30) = 5.47723


Instruction: How to solve the pytorch RuntimeError: Numpy is not available without upgrading numpy to the latest version because of other dependencies

Input:
I am running a simple CNN using Pytorch for some audio classification on my Raspberry Pi 4 on Python 3.9.2 (64-bit). For the audio manipulation needed I am using librosa. librosa depends on the numba package which is only compatible with numpy version <= 1.20.
When running my code, the line
spect_tensor = torch.from_numpy(spect).double()

throws the RuntimeError:
RuntimeError: Numpy is not available

Searching the internet for solutions I found upgrading Numpy to the latest version to resolve that specific error, but throwing another error, because Numba only works with Numpy <= 1.20.
Is there a solution to this problem which does not include searching for an alternative to using librosa?

Output: Just wanted to give an update on my situation. I downgraded torch to version 0.9.1 which solved the original issue. Now OpenBLAS is throwing a warning because of an open MPLoop. But for now my code is up and running.


Instruction: speeding up 1d convolution in PyTorch

Input:
For my project I am using pytorch as a linear algebra backend. For the performance part of my code, I need to do 1D convolutions of 2 small (length between 2 and 9) vectors (1D tensors) a very large number of times. My code allows for batch-processing of inputs and thus I can stack a couple of input vectors to create matrices that can then be convolved all at the same time. Since torch.conv1d does not allow for convolving along a single dimension for 2D inputs, I had to write my own convolution function called convolve. This new function however consists of a double for-loop and is therefore very very slow.
Question: how can I make the convolve function perform faster through better code-design and let it be able to deal with batched inputs (=2D tensors)?
Partial answer: somehow avoid the double for-loop
Below are three jupyter notebook cells that recreate a minimal example. Note that the you need line_profiler and the %%writefile magic command to make this work!
%%writefile SO_CONVOLVE_QUESTION.py
import torch

def conv1d(a, v):
    padding = v.shape[-1] - 1
    return torch.conv1d(
        input=a.view(1, 1, -1), weight=v.flip(0).view(1, 1, -1), padding=padding, stride=1
    ).squeeze()

def convolve(a, v):
    if a.ndim == 1:
        a = a.view(1, -1)
        v = v.view(1, -1)

    nrows, vcols = v.shape
    acols = a.shape[1]

    expanded = a.view((nrows, acols, 1)) * v.view((nrows, 1, vcols))
    noutdim = vcols + acols - 1
    out = torch.zeros((nrows, noutdim))
    for i in range(acols):
        for j in range(vcols):
            out[:, i+j] += expanded[:, i, j]
    return out.squeeze()

x = torch.randn(5)
y = torch.randn(7)

I write the code to the SO_CONVOLVE_QUESTION.py because that is necessary for line_profiler and to use as a setup for timeit.timeit.
Now we can evaluate the output and performance of the code above on non-batch input (x, y) and batched input (x_batch, y_batch):
from SO_CONVOLVE_QUESTION import *
# Without batch processing
res1 = conv1d(x, y)
res = convolve(x, y)
print(torch.allclose(res1, res)) # True

# With batch processing, NB first dimension!
x_batch = torch.randn(5, 5)
y_batch = torch.randn(5, 7)

results = []
for i in range(5):
    results.append(conv1d(x_batch[i, :], y_batch[i, :]))
res1 = torch.stack(results)
res = convolve(x_batch, y_batch)
print(torch.allclose(res1, res))  # True

print(timeit.timeit('convolve(x, y)', setup=setup, number=10000)) # 4.83391789999996
print(timeit.timeit('conv1d(x, y)', setup=setup, number=10000))   # 0.2799923000000035

In the block above you can see that performing convolution 5 times using the conv1d function produces the same result as convolve on batched inputs. We can also see that convolve (= 4.8s) is much slower than the conv1d (= 0.28s). Below we assess the slow part of the convolve function WITHOUT batch processing using line_profiler:
%load_ext line_profiler
%lprun -f convolve convolve(x, y)  # evaluated without batch-processing!

Output:
Timer unit: 1e-07 s

Total time: 0.0010383 s
File: C:\python_projects\pysumo\SO_CONVOLVE_QUESTION.py
Function: convolve at line 9

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
     9                                           def convolve(a, v):
    10         1         68.0     68.0      0.7      if a.ndim == 1:
    11         1        271.0    271.0      2.6          a = a.view(1, -1)
    12         1         44.0     44.0      0.4          v = v.view(1, -1)
    13
    14         1         28.0     28.0      0.3      nrows, vcols = v.shape
    15         1         12.0     12.0      0.1      acols = a.shape[1]
    16
    17         1       4337.0   4337.0     41.8      expanded = a.view((nrows, acols, 1)) * v.view((nrows, 1, vcols))
    18         1         12.0     12.0      0.1      noutdim = vcols + acols - 1
    19         1        127.0    127.0      1.2      out = torch.zeros((nrows, noutdim))
    20         6         32.0      5.3      0.3      for i in range(acols):
    21        40        209.0      5.2      2.0          for j in range(vcols):
    22        35       5194.0    148.4     50.0              out[:, i+j] += expanded[:, i, j]
    23         1         49.0     49.0      0.5      return out.squeeze()

Obviously a double for-loop and the line creating the expanded tensor are the slowest. Are these parts avoidable with better code-design?

Output:
Turns out that there is a way to do it without for-loops via grouping of the inputs along a dimension:
out = torch.conv1d(x_batch.unsqueeze(0), y_batch.unsqueeze(1).flip(2), padding=y_batch.size(1)-1, groups=x_batch.size(0))
print(torch.allclose(out, res1))  # True


Instruction: Stack expects each tensor to be equal size

Input:
I am following PyTorch tutorial on speech command recogniton   and trying to implement my own recognition of 22 sentences in german language. In the tutorial they use padding for audio tensors, but for labels they use only torch.stack. Because of that, I have an error, as I start training the network:
RuntimeError: stack expects each tensor to be equal size, but got [456] at entry 0 and [470] at entry 1.
I do understand what this says, but since I am new to PyTorch can't unfortunately implement padding function for sentences from scratch. Therefore I would be happy if you could give me some hints and tipps for this.
Here is the code for collate_fn and pad_sequence functions:
def pad_sequence(batch):
    # Make all tensor in a batch the same length by padding with zeros
    batch = [item.t() for item in batch]
    batch = torch.nn.utils.rnn.pad_sequence(batch, batch_first=True, padding_value=0.)
    return batch.permute(0, 2, 1)

def collate_fn(batch):
    # A data tuple has the form:
    # waveform,  label
    tensors, targets = [], []

    # Gather in lists, and encode labels as indices
    for waveform, label in batch:
        tensors += [waveform]
        targets += [label]

    # Group the list of tensors into a batched tensor
    tensors = pad_sequence(tensors)
    targets = torch.stack(targets)

    return tensors, targets

Output:
As I started working directly with pad_sequence, I understood how simple it works. So, in my case I needed only bunch of strings (batch), which were automatically compared by PyTorch and extended to the maximal length of the one of the several strings in the batch.
My code looks now like this:

def pad_AudioSequence(batch):
  # Make all tensor in a batch the same length by padding with zeros
  batch = [item.t() for item in batch]
  batch = torch.nn.utils.rnn.pad_sequence(batch, batch_first=True, padding_value=0.)
  return batch.permute(0, 2, 1)

def pad_TextSequence(batch):
  return torch.nn.utils.rnn.pad_sequence(batch,batch_first=True, padding_value=0)

def collate_fn(batch):
  # A data tuple has the form:
  # waveform,  label
  tensors, targets = [], []
  # Gather in lists, and encode labels as indices
  for waveform, label in batch:
      tensors += [waveform]
      targets += [label]
  # Group the list of tensors into a batched tensor
  tensors = pad_AudioSequence(tensors)
  targets = pad_TextSequence(targets)
  return tensors, targets

For those, who still don't understand how that works, here is little example:
encDecClass2 = dummyEncoderDecoder()
sent1 = audioWorkerClass.sentences[4] # wie viel Prozent hat der Akku noch?
sent2 = audioWorkerClass.sentences[5] # Wie spät ist es?
sent3 = audioWorkerClass.sentences[6] # Mach einen Timer für 5 Sekunden.

# encode sentences into tensor of numbers, representing words, using my own enc-dec class
sent1 = encDecClass2.encode(sent1) # tensor([11, 94, 21, 94, 22, 94, 23, 94, 24, 94, 25, 94, 26, 94, 15, 94])
sent2 = encDecClass2.encode(sent2) # tensor([27, 94, 28, 94, 12, 94, 29, 94, 15, 94])
sent3 = encDecClass2.encode(sent3) # tensor([30, 94, 31, 94, 32, 94, 33, 94, 34, 94, 35, 94, 19, 94])

print(sent1.shape) # torch.Size([16])
print(sent2.shape) # torch.Size([10])
print(sent3.shape) # torch.Size([14])

batch = []
# add sentences to the batch as separate arrays
batch +=[sent1]
batch +=[sent2]
batch +=[sent3]

output = pad_sequence(batch,batch_first=True, padding_value=0)

print(f"{output}\n{output.shape}")

#############################################################################
# output:
# tensor([[11, 94, 21, 94, 22, 94, 23, 94, 24, 94, 25, 94, 26, 94, 15, 94],
#         [27, 94, 28, 94, 12, 94, 29, 94, 15, 94,  0,  0,  0,  0,  0,  0],
#         [30, 94, 31, 94, 32, 94, 33, 94, 34, 94, 35, 94, 19, 94,  0,  0]])
# torch.Size([3, 16])
#############################################################################

As you may see all arrays were equalized to the maximum length of those three arrays and padded with zeros. Shape of the output is 3x16, because we had three sentences and longest array had sequence of 16 in the batch.


Instruction: RuntimeError: DataLoader worker (pid(s) 15876, 2756) exited unexpectedly

Input:
I am compiling some existing examples from the PyTorch tutorial website. I am working especially on the CPU device no GPU.
When running a program the type of error below is shown. Does it become I'm working on the CPU device or setup issue? raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) from e
RuntimeError: DataLoader worker (pid(s) 15876, 2756) exited unexpectedly`. How can I solve it?
import torch
import torch.functional as F
import torch.nn as nn
import torch.optim as optim

import torchvision
import torchvision.transforms as transforms

import matplotlib.pyplot as plt
import numpy as np

from torch.utils.tensorboard import SummaryWriter
from torch.utils.data import DataLoader
from torchvision import datasets

device = 'cpu' if torch.cuda.is_available() else 'cuda'
print(device)

transform = transforms.Compose(
[transforms.ToTensor(),
 transforms.Normalize((0.5,), (0.5,))]
)
#Store separate training and validations splits in data
training_set = datasets.FashionMNIST(
 root='data',
 train=True,
 download=True,
 transform=transform
)
validation_set = datasets.FashionMNIST(
root='data',
train=False,
download=True,
transform=transform
)
training_loader = DataLoader(training_set, batch_size=4, shuffle=True, num_workers=2)
validation_loader = DataLoader(validation_set, batch_size=4, shuffle=False, num_workers=2)
classes = ('T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat',
    'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle Boot')

def matplotlib_imshow(img, one_channel=False):
  if one_channel:
     img = img.mean(dim=0)
img = img/2+0.5 #unnormalize
npimg = img.numpy()
if one_channel:
    plt.imshow(npimg, cmap="Greys")
else:
    plt.imshow(np.transpose(npimg, (1, 2, 0)))

dataiter = iter(training_loader)
images, labels = dataiter.next()

img_grid = torchvision.utils.make_grid(images)
matplotlib_imshow(img_grid, one_channel=True)

Output: You need to first figure out why the dataLoader worker crashed. A common reason is out of memory. You can check this by running dmesg -T after your script crashes and see if the system killed any python process.


Instruction: How do I more efficiently convert a string of slices to slice objects that can then be used to slice arrays & tensors in PyTorch / NumPy?

Input:
How can I simplify this function that converts strings of slices for PyTorch / NumPy to slice list objects that can then be used to slice arrays & tensors?
The code below works, but it seems rather inefficient in terms of how many lines it takes.
def str_to_slice_indices(slicing_str: str):
    # Convert indices to lists
    indices = [
        [i if i else None for i in indice_set.strip().split(":")]
        for indice_set in slicing_str.strip("[]").split(",")
    ]

    # Handle Ellipsis "..."
    indices = [
        ... if index_slice == ["..."] else index_slice for index_slice in indices
    ]
    # Handle "None" values
    indices = [
        None if index_slice == ["None"] else index_slice for index_slice in indices
    ]
    # Handle single number values
    indices = [
        int(index_slice[0])
        if isinstance(index_slice, list)
        and len(index_slice) == 1
        and index_slice[0].lstrip("-").isdigit()
        else index_slice
        for index_slice in indices
    ]

    # Create indice slicing list
    indices = [
        slice(*[int(i) if i and i.lstrip("-").isdigit() else None for i in index_slice])
        if isinstance(index_slice, list)
        else index_slice
        for index_slice in indices
    ]
    return indices

Running the above function with an example covering the various types of inputs, give this:
out = str_to_slice_indices("[None, :1, 3:4, 2, :, 2:, ...]")
print(out)

# out:
# [None, slice(None, 1, None), slice(3, 4, None), 2, slice(None, None, None), slice(2, None, None), Ellipsis]

Output:
Iterating multiple times is not necessary. The sample string has been slightly expanded to test more cases.
def str2slices(s):
    d = {True: lambda e: slice(*[int(i) if i else None for i in e.split(':')]),
        'None': lambda e: None,
        '...': lambda e: ...}
    return [d.get(':' in e or e.strip(), lambda e: int(e))(e.strip()) for e in s[1:-1].split(',')]

str2slices('[None, :1, 3:4, 2, :, -10: ,::,:4:2, 1:10:2, -32,...]')

Output
[None,
 slice(None, 1, None),
 slice(3, 4, None),
 2,
 slice(None, None, None),
 slice(-10, None, None),
 slice(None, None, None),
 slice(None, 4, 2),
 slice(1, 10, 2),
 -32,
 Ellipsis]

The same errors as in OP's solution are caught. They don't silently change the result, but throw a ValueError for unsupported input.

Breakdown of the solution
Assuming string slicing and the split function are known.
With example
s = '[None, :1, 3:4, 2, :, -10: ,::,:4:2, 1:10:2, -32,...]'

we can find slices with
[':' in e for e in s[1:-1].split(',')]
#[False, True, True, False, True, True, True, True, True, False, False]

Using or short-circutting we can distinguish other cases
[':' in e or e.strip() for e in s[1:-1].split(',')]
#['None', True, True, '2', True, True, True, True, True, '-32', '...']

This values can be used as keys of a dictionary
d = {True: 'slice', 'None': None, '...': ...}
[d[':' in e or e.strip()] for e in s[1:-1].split(',')]
#KeyError: '2'

To prevent the KeyError we can use the get method with a default value.
d = {True: 'slice', 'None': None, '...': ...}
[d.get(':' in e or e.strip(), 'number') for e in s[1:-1].split(',')]
#[None, 'slice', 'slice', 'number', 'slice', 'slice', 'slice', 'slice', 'slice', 'number', Ellipsis]

In order to process slices, we need to parse additional values at runtime. So we use lambdas as dictionary values to be able to call them with (e.strip()). Finally, we convert values to int if necessary.
d = {True: lambda e: slice(*[int(i) if i else None for i in e.split(':')]),
    'None': lambda e: None,
    '...': lambda e: ...}
[d.get(':' in e or e.strip(), lambda e: int(e))(e.strip()) for e in s[1:-1].split(',')]

Output
[None,
 slice(None, 1, None),
 slice(3, 4, None),
 2,
 slice(None, None, None),
 slice(-10, None, None),
 slice(None, None, None),
 slice(None, 4, 2),
 slice(1, 10, 2),
 -32,
 Ellipsis]


Instruction: Output of the model depends on the shape of the weights tensor

Input:
I want to train the model to sum the three inputs. So it is as simple as possible.
Firstly the weights are initialized randomly. It produces bad error estimate (approx. 0.5)
Then I initialize the weights with zeros. There are two options:

the shape of the weights tensor is [1, 3]
the shape of the weights tensor is [3]

When I choose the 1st option the model still works bad and can't learn this simple formula.
When I choose the 2nd option it works perfect with the error of 10e-12.
Why the result depends on the shape of the weights? Why do I need to initialize the model with zeros to solve this simple problem?
    import torch
    from torch.nn import Sequential as Seq, Linear as Lin
    from torch.optim.lr_scheduler import ReduceLROnPlateau

    X = torch.rand((1024, 3))
    y = (X[:,0] + X[:,1] + X[:,2])
    m = Seq(Lin(3, 1, bias=False))

    # 1 option
    m[0].weight = torch.nn.parameter.Parameter(torch.tensor([[0, 0, 0]], dtype=torch.float))

    # 2 option
    #m[0].weight = torch.nn.parameter.Parameter(torch.tensor([0, 0, 0], dtype=torch.float))

    optim = torch.optim.SGD(m.parameters(), lr=10e-2)
    scheduler = ReduceLROnPlateau(optim, 'min', factor=0.5, patience=20, verbose=True)
    mse = torch.nn.MSELoss()
    for epoch in range(500):
        optim.zero_grad()
        out = m(X)
        loss = mse(out, y)
        loss.backward()
        optim.step()
        if epoch % 20 == 0:
            print(loss.item())
        scheduler.step(loss)

Output: First option doesn't learning because it fails with broadcasting: while out.shape == (1024, 1) corresponding targets y  has shape of (1024, ). MSELoss, as expected, computes mean of tensor (out - y)^2, which in this case has shape (1024, 1024), clearly wrong objective for this task. At the same time, after applying 2-nd option tensor (out - y)^2 has size (1024, ) and mean of it corresponds to actual mse. Default approach, without explicit changing weights shape (through option 1 and 2), would work if set target shape to (1024, 1) for example by y = y.unsqueeze(-1) after definition of y.


Instruction: Does inplace matter when we return ReLU(x)

Input:
Is there a difference between the following two classes?
I know what inplace is (you don't need to do x = function(x) but only function(x) to modify x if inplace is True). But here because we return self.conv(x), it should not matter, right?
class ConvBlock(nn.Module):

    def __init__(
        self,
        in_channels,
        out_channels,
        down=True,
        use_act=True,
        **kwargs
        ):
        super().__init__()
        self.conv = nn.Sequential((nn.Conv2d(in_channels, out_channels,
                                  padding_mode='reflect',
                                  **kwargs) if down else nn.ConvTranspose2d(in_channels,
                                  out_channels, **kwargs)),
                                  nn.InstanceNorm2d(out_channels),
                                  (nn.ReLU() if use_act else nn.Identity()))

    def forward(self, x):
        return self.conv(x)

class ConvBlockInplace(nn.Module):

    def __init__(
        self,
        in_channels,
        out_channels,
        down=True,
        use_act=True,
        **kwargs
        ):
        super().__init__()
        self.conv = nn.Sequential((nn.Conv2d(in_channels, out_channels,
                                  padding_mode='reflect',
                                  **kwargs) if down else nn.ConvTranspose2d(in_channels,
                                  out_channels, **kwargs)),
                                  nn.InstanceNorm2d(out_channels),
                                  (nn.ReLU(inplace=True) if use_act else nn.Identity()))

    def forward(self, x):
        return self.conv(x)

Output:
The inplace operations do the exact amount of computations. However, there are less memory accesses, if your task is memory bound. Then, it would "matter".

I used the ptflops flops counter to generate the following statistics
ConvBlock(
  0.0 M, 100.000% Params, 0.015 GMac, 100.000% MACs,
  (conv): Sequential(
    0.0 M, 100.000% Params, 0.015 GMac, 100.000% MACs,
    (0): Conv2d(0.0 M, 100.000% Params, 0.014 GMac, 93.333% MACs, 3, 10, kernel_size=(3, 3), stride=(1, 1), padding_mode=reflect)
    (1): InstanceNorm2d(0.0 M, 0.000% Params, 0.0 GMac, 3.333% MACs, 10, eps=1e-05, momentum=0.1, affine=False, track_running_stats=False)
    (2): ReLU(0.0 M, 0.000% Params, 0.0 GMac, 3.333% MACs, )
  )
)
Computational complexity:       0.01 GMac
Number of parameters:           280
Warning: module ConvBlockInplace is treated as a zero-op.
ConvBlockInplace(
  0.0 M, 100.000% Params, 0.015 GMac, 100.000% MACs,
  (conv): Sequential(
    0.0 M, 100.000% Params, 0.015 GMac, 100.000% MACs,
    (0): Conv2d(0.0 M, 100.000% Params, 0.014 GMac, 93.333% MACs, 3, 10, kernel_size=(3, 3), stride=(1, 1), padding_mode=reflect)
    (1): InstanceNorm2d(0.0 M, 0.000% Params, 0.0 GMac, 3.333% MACs, 10, eps=1e-05, momentum=0.1, affine=False, track_running_stats=False)
    (2): ReLU(0.0 M, 0.000% Params, 0.0 GMac, 3.333% MACs, inplace=True)
  )
)
Computational complexity:       0.01 GMac
Number of parameters:           280


Instruction: RuntimeError: Boolean value of Tensor with more than one value is ambiguous in python

Input:
I'm facing the following error, and I don't know why:
This code is on GitHub, I ran it correctly on Collab, but it gives me the following error here:
device="cpu"
lr=3e-5#1e-3
num_training_steps=int(len(dataset) / TRAIN_BATCH_SIZE * EPOCH)

model=Bert_Classification_Model().to(device)
optimizer=AdamW(model.parameters(), lr=lr)
scheduler = get_linear_schedule_with_warmup(optimizer,
                                        num_warmup_steps = 0,
                                        num_training_steps = num_training_steps)
val_losses=[]
batches_losses=[]
val_acc=[]
for epoch in range(EPOCH):
    t0 = time.time()
    print(f"\n=============== EPOCH {epoch+1} / {EPOCH} ===============\n")
    batches_losses_tmp=train_loop_fun1(train_data_loader, model, optimizer, device)
    epoch_loss=np.mean(batches_losses_tmp)
    print(f"\n*** avg_loss : {epoch_loss:.2f}, time : ~{(time.time()-t0)//60} min ({time.time()-t0:.2f} sec) ***\n")
    t1=time.time()
    output, target, val_losses_tmp=eval_loop_fun1(valid_data_loader, model, device)
    print(f"==> evaluation : avg_loss = {np.mean(val_losses_tmp):.2f}, time : {time.time()-t1:.2f} sec\n")
    tmp_evaluate=evaluate(target.reshape(-1), output)
    print(f"=====>\t{tmp_evaluate}")
    val_acc.append(tmp_evaluate['accuracy'])
    val_losses.append(val_losses_tmp)
    batches_losses.append(batches_losses_tmp)
    print("\t§§ model has been saved §§")
    torch.save(model, f"model1/model_epoch{epoch+1}.pt")

Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertModel: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.bias', 'cls.seq_relationship.weight']
- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).

=============== EPOCH 1 / 3 ===============

---------------------------------------------------------------------------

RuntimeError                              Traceback (most recent call last)

<ipython-input-33-aa98faac385e> in <module>()
     14     t0 = time.time()
     15     print(f"\n=============== EPOCH {epoch+1} / {EPOCH} ===============\n")
---> 16     batches_losses_tmp=train_loop_fun1(train_data_loader, model, optimizer, device)
     17     epoch_loss=np.mean(batches_losses_tmp)
     18     print(f"\n*** avg_loss : {epoch_loss:.2f}, time : ~{(time.time()-t0)//60} min ({time.time()-t0:.2f} sec) ***\n")

6 frames

/content/RoBERT_Recurrence_over_BERT/Custom_Dataset_Class.py in long_terms_tokenizer(self, data_tokenize, targets)
    158         targets_list.append(targets)
    159
--> 160         if remain and self.approach != 'head':
    161             remain = torch.tensor(remain, dtype=torch.long)
    162             idxs = range(len(remain)+self.chunk_len)

RuntimeError: Boolean value of Tensor with more than one value is ambiguous

This is the file link:
https://github.com/helmy-elrais/RoBERT_Recurrence_over_BERT/blob/master/train.ipynb

Output: Your tensor remain (in your Dataset class) is a boolean tensor and not a boolean variable. Therefore, the condition if remain is not well-defined.


Instruction: RuntimeError: mat1 and mat2 shapes cannot be multiplied (5400x64 and 5400x64)

Input:
I'm working on an image classification network and got a problem with the right values of inputs and outputs in the forward() function. I don't have an idea to solve this, because they seem the same to me. The error comes from this line:
 x = F.relu(self.fc1(x)), but I can't figure it out.
Can anyone please help me with this problem?
That's my code:
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 8, kernel_size=2)
        self.conv2 = nn.Conv2d(8, 12, kernel_size=2)
        self.conv3 = nn.Conv2d(12, 18, kernel_size=2)
        self.conv4 = nn.Conv2d(18, 24, kernel_size=2)
        self.fc1 = nn.Linear(5400, 64)
        self.fc2 = nn.Linear(64, 2)

    def forward(self, x):
        print(f'1. {x.size()}')
        x = self.conv1(x)
        x = F.max_pool2d(x, 2)
        x = F.relu(x)
        print(f'2. {x.size()}')
        x = self.conv2(x)
        x = F.max_pool2d(x, 2)
        x = F.relu(x)
        print(f'3. {x.size()}')
        x = self.conv3(x)
        x = F.max_pool2d(x, 2)
        x = F.relu(x)
        print(f'4. {x.size()}')
        x = self.conv4(x)
        x = F.max_pool2d(x, 2)
        x = F.relu(x)
        print(f'5. {x.size()}')
        x = x.view(-1, x.size(0))
        print(f'6. {x.size()}')
        x = F.relu(self.fc1(x))
        print(f'7. {x.size()}')
        x = self.fc2(x)
        print(f'8. {x.size()}')

        return torch.sigmoid(x)

That's the print output:
1. torch.Size([64, 3, 256, 256])
2. torch.Size([64, 8, 127, 127])
3. torch.Size([64, 12, 63, 63])
4. torch.Size([64, 18, 31, 31])
5. torch.Size([64, 24, 15, 15])
6. torch.Size([5400, 64])

Output:
I think changing
x = x.view(-1, x.size(0))

to
x = x.view([-1, 5400], x.size(0))

Will solve your problem, You see that in print 6:
6. torch.Size([5400, 64])

the batch size 64 is in the 1 axes and not in the 0 axes. The fully connected layer expects an input of size 5400 therefore changing this will likely solve since you do not know that batch size but you know that the input to the fully-connected is 5400.


Instruction: Explain (T,) tensor shape

Input:
In the following d2l tutorial:
import torch

T = 1000

time = torch.arange(1, T + 1, dtype=torch.float32)
print(f"time shape: {time.shape}")

x = torch.sin(0.01 * time) + torch.normal(0.0, 0.2, size=(T,))

Given that torch.sin(0.01 * time) shape is torch.Size([1000]) why the size attribute provided to normal function is (T,) and not (T)?

Output: Because (T) is equal to T having type of int, while torch.normal requires a tuple. (T,) is a Python way to pass one-element tuple.


Instruction: What does \* mean in the function signature of PyTorch?

Input:
For example in the randint signature there is a \* as 4th argument. What does it mean ?
torch.randint(low=0, high, size, \*, generator=None, out=None, dtype=None, layout=torch.strided, device=None, requires_grad=False) → Tensor

I am aware of position only args and keyword only args introduced in Python 3.8 which use \  and *. But here I see \*.

Output:
I was pretty sure user2357112 supports Monica is right. To confirm it, I've tried looking for the source of the documentation, which is autogenerated. I've looked at the implementation, which is in C++ where functions signatures work differently. I found the type annotation generation code:
        'randint': ['def randint(low: _int, high: _int, size: _size, *,'
                    ' generator: Optional[Generator]=None, {}) -> Tensor: ...'
                    .format(FACTORY_PARAMS),
                    'def randint(high: _int, size: _size, *,'
                    ' generator: Optional[Generator]=None, {}) -> Tensor: ...'
                    .format(FACTORY_PARAMS)],

This suggests that the commenter was indeed right, and it should have been * rather than \*.
Positional arguments in Python are denoted by /, not by \. In Python's syntax \ is only used in two places: as an escape character in string literals, and at the end of a line as an explicit line continuation marker.

It looks like this has been an issue since at least May 2020, with no activity since.


Instruction: YOLOv5 get boxes, scores, classes, nums

Input:
im trying to bind the Object Tracking with Deep Sort in my Project and i need to get the boxes, scores, classes, nums.
Loading Pretrained Yolov5 model:
model = torch.hub.load('ultralytics/yolov5', 'yolov5s', pretrained=True)
model.eval()

Getting the Prediction:
result = model(img)
print(result.shape)
print(result)
torch.Size([8, 6])
tensor([[277.50000, 379.25000, 410.50000, 478.75000,   0.90625,   2.00000],
        [404.00000, 205.12500, 498.50000, 296.00000,   0.88623,   2.00000],
        [262.50000, 247.75000, 359.50000, 350.25000,   0.88281,   2.00000],
        [210.50000, 177.75000, 295.00000, 261.75000,   0.83154,   2.00000],
        [195.50000, 152.50000, 257.75000, 226.00000,   0.78223,   2.00000],
        [137.00000, 146.75000, 168.00000, 162.00000,   0.55713,   2.00000],
        [ 96.00000, 130.12500, 132.50000, 161.12500,   0.54199,   2.00000],
        [ 43.56250,  89.56250,  87.68750, 161.50000,   0.50146,   5.00000]], device='cuda:0')
tensor([[277.50000, 379.25000, 410.50000, 478.75000,   0.90625,   2.00000],
        [404.00000, 205.12500, 498.50000, 296.00000,   0.88623,   2.00000],
        [262.50000, 247.75000, 359.50000, 350.25000,   0.88281,   2.00000],
        [210.50000, 177.75000, 295.00000, 261.75000,   0.83154,   2.00000],
        [195.50000, 152.50000, 257.75000, 226.00000,   0.78223,   2.00000],
        [137.00000, 146.75000, 168.00000, 162.00000,   0.55713,   2.00000],
        [ 96.00000, 130.12500, 132.50000, 161.12500,   0.54199,   2.00000],
        [ 43.56250,  89.56250,  87.68750, 161.50000,   0.50146,   5.00000]], device='cuda:0')

so now my question is how do i get the boxes, scores, classes, nums in each variables?
I need that for the Object Tracking
I tried it once with the example on Pytorch Documentation:
result.xyxy[0]
but in my Case I get an Error:
Tensor has no attribute xyxy

Output:
The output from the model is a torch tensor and has no xyxy method. You need to extract the values manually. Either you can go through each detection one by one:
import torch

det = torch.rand(8, 6)

for *xyxy, conf, cls in det:
    print(*xyxy)
    print(conf)
    print(cls)

or you can slice the detections tensor by:
xyxy = det[:, 0:4]
conf = det[:, 4]
cls = det[:, 5]

print(xyxy)
print(conf)
print(cls)


Instruction: Filter torch tensor based on another tensor without loops

Input:
Suppose I have the following two torch.Tensors:
x = torch.tensor([0,0,0,1,1,2,2,2,2], dtype=torch.int64)
y = torch.tensor([0,2], dtype=torch.int64)

I want to somehow filter x such that only the values that are in y remain:
x_filtered = torch.tensor([0,0,0,2,2,2,2])

For another example, if y = torch.tensor([0,1]), then x_filtered = torch.tensor([0,0,0,1,1]). Both x,y are always 1D and int64. y is always sorted, if it makes it simpler, we can assume that x is always sorted as well.
I tried to think of various ways to do it without using loops, but failed. I cannot really use loops because my use case involves x in the millions and y in tens of thousands. Any help is appreciated.

Just realised what I need is the torch equivalent of numpy.in1d

Output:
For filtering tensor as you want in you task, you need to use isin function available in torch. The way it is used is given below:-
import torch
x = torch.tensor([0,0,0,1,1,2,2,2,2,3], dtype=torch.int64)
y = torch.tensor([0,2], dtype=torch.int64)
# torch.isin(x, y)
c=x[torch.isin(x,y)]
print(c)

After running this code you will get your preferred answer.


Instruction: Trying to recreate BiLSTM model from Adhikari et al. 2019 (LSTM_reg) in Tensorflow

Input:
Im trying to recreate this model LSTM_reg from this paper in TensorFlow to use in my problem. I've come up with the following code:
def get_model(lr=0.001):
    model = tf.keras.models.Sequential()
    model.add(tf.keras.layers.Embedding(nb_words, output_dim=embed_size, weights=[embedding_matrix], input_length = maxlen, trainable=False))
    model.add(tf.keras.layers.Dropout(0.2)) # embedding dropouts
    model.add(tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(256, return_sequences=True, recurrent_dropout=0.2, activation = 'tanh'))) # weight drop on recurrent layers using recurrent_dropout
    model.add(tf.keras.layers.MaxPooling1D(pool_size=2, padding = 'valid'))
    model.add(tf.keras.layers.Flatten())
    model.add(tf.keras.layers.Dense(512, activation='relu'))
    model.add(tf.keras.layers.Dropout(0.5))
    model.add(tf.keras.layers.Dense(20))
    model.add(tf.keras.layers.Activation('sigmoid'))

    model.compile(loss = 'categorical_crossentropy' , optimizer = 'adam', metrics = ['accuracy', tfa.metrics.F1Score(num_classes = 20)])
    return model

Have I gone about this the right way? Got some pretty weird values while training my dataset, hence was wondering about my implementation.. There is a pytorch implementation for this model here. But I'm not sure if I have reproduced this correctly.

Output:
One major difference I can see is that the paper uses a global max pooling, whereas you've only used max pooling with a kernel size of 2:
def get_model(lr=0.001):
    model = tf.keras.models.Sequential()
    model.add(tf.keras.layers.Embedding(nb_words, output_dim=embed_size, weights=[embedding_matrix], input_length = maxlen, trainable=False))
    model.add(tf.keras.layers.Dropout(0.2)) # embedding dropouts
    model.add(tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(256, return_sequences=True, recurrent_dropout=0.2, activation = 'tanh'))) # weight drop on recurrent layers using recurrent_dropout
    model.add(tf.keras.layers.GlobalMaxPooling1D())
    model.add(tf.keras.layers.Dense(512, activation='relu'))
    model.add(tf.keras.layers.Dropout(0.5))
    model.add(tf.keras.layers.Dense(20))
    model.add(tf.keras.layers.Activation('sigmoid'))

    model.compile(loss = 'categorical_crossentropy' , optimizer = 'adam', metrics = ['accuracy', tfa.metrics.F1Score(num_classes = 20)])
    return model

I obviously don't have your data so make sure that the arguments are set correctly: https://keras.io/api/layers/pooling_layers/global_max_pooling1d/.
Another change is that the pytorch repo you shared has a ReLU after the LSTM (I don't know why). You could try adding that in and see if it helps.


Instruction: Error during traning my model with pytorch, stack expects each tensor to be equal size

Input:
I am using MMSegmentainon library to train my model for instance image segmentation, during the traingin, I craete the model(Vision Transformer) and when I try to train  the model using this:
I get this error:

RuntimeError:CaughtRuntimeErrorinDataLoaderworkerprocess0.OriginalTraceback(mostrecentcalllast):
File"/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/worker.py",line287,in
_worker_loop
data=fetcher.fetch(index)
File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/fetch.py", line 47, infetch
returnself.collate_fn(data)
File "/usr/local/lib/python3.7/dist-packages/mmcv/parallel/collate.py", line 81, in collateforkeyinbatch[0]
File"/usr/local/lib/python3.7/dist-packages/mmcv/parallel/collate.py",line81,in
<dictcomp>
forkey in batch[0]
File"/usr/local/lib/python3.7/dist-packages/mmcv/parallel/collate.py",line59,incollatestacked.append(default_collate(padded_samples))
File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/collate.py", line 56, indefault_collate
returntorch.stack(batch,0,out=out)
RuntimeError: stack expects each tensor to be equal size, but got [1, 256, 256, 256] at entry0 and[1,256,256] at entry3

** I must also mention that I have tested my own dataset with other models available in their library but all of them works properly.
tried :

model=build_segmentor(cfg.model,train_cfg=cfg.get('train_cfg'),test_cfg=cfg.get('test_cfg'))train_segmentor(model,datasets,cfg,distributed=False,validate=True,
meta=dict())

Output:
It seems that images in your dataset might not have the same size, as in the VIT model https://arxiv.org/abs/2010.11929, you are using an MLP model,
if it was not the case, it is worth checking if your labels are all in the expected dimension.
presumably, MMsegmentattion expects the output to be just the annotation map (a 2D array).
It is recommended that you revise your dataset and prepare the annotation map.


Instruction: How can generator get input noise z?

Input:
Hi I'm looking this GAN implementation code. code here
My question is generator class has no input parameter when defining class Generator(#38 from the link)
But when training, generator gets input z(#141 from the link).
I looked into the nn.Module class which is parent of class Generator but I can't find input parameter for noise z. Can anyone help?
class Generator(nn.Module):    #38
    def __init__(self):
        super(Generator, self).__init__()

        def block(in_feat, out_feat, normalize=True):
            layers = [nn.Linear(in_feat, out_feat)]
            if normalize:
                layers.append(nn.BatchNorm1d(out_feat, 0.8))
            layers.append(nn.LeakyReLU(0.2, inplace=True))
            return layers

        self.model = nn.Sequential(
            *block(opt.latent_dim, 128, normalize=False),
            *block(128, 256),
            *block(256, 512),
            *block(512, 1024),
            nn.Linear(1024, int(np.prod(img_shape))),
            nn.Tanh()
        )

    def forward(self, z):
        img = self.model(z)
        img = img.view(img.size(0), *img_shape)
        return img

generator = Generator()    #88

gen_imgs = generator(z)    #141

I tried looking for nn.Module, variable() in pytorch docs and still can't get what I wanted.

Output:
Consider every quoted line (38, 88 and 141):

On line 38 is a definition of class, by putting nn.Module in brackets it's declaring inheritance of class Generator from class nn.Module (which is common way to define your own neural network).
On line 88 instance of class Generator is created -- for parameters it needs all what's inside brackets of __init__ (line 39), besides self,  that's why brackets on line 88 is empty.
And on line 141 there is a calling of genearator, behavior here defined by method forward (line 58) and there is one parameter to be passed --  z.

Again, line 88 creates an instance, line 141 calls forward method of an instance.


Instruction: Pytorch: Assign values from one mask to another, masked by itself

Input:
I have a mask active that tracks batches that still have not terminated in a recurrent process. It's dimension is [batch_full,], and it's true entries show which elements need to still be used in current step. The recurrent process generates another mask, terminated, which has as many elements as true values in active mask. Now, I want to take values from ~terminated and put them back into active, but at the right indices. Basically I want to do:
import torch

active = torch.ones([4,], dtype=torch.bool)
active[:2] = torch.tensor(False)

terminated = torch.tensor([True, False])

active[active] = ~terminated

print(active)  # expected [F, F, F, T]

However, I get error:

RuntimeError: unsupported operation: some elements of the input tensor and the written-to tensor refer to a single memory location. Please clone() the tensor before performing the operation.

How can I do the described above operation in an effective way?

Output:
There are a few solutions, I will also give their speed as measured by timeit, 10k repetitions, on 2021 macbook pro.
The simplest solution, taking 0.260s:
active[active.clone()] = ~terminated

We can use masked_scatter_ inplace operation for abt. 2x speedup (0.136s):
active.masked_scatter_(
        active,
        ~terminated,
    )

Out of place operation, taking 0.161s, would be:
active = torch.masked_scatter(
        active,
        active,
        ~terminated,
    )


Instruction: Why does my variational autoencoder only produce positive values?

Input: I copied this example to build a variational autoencoder (VAE). The example uses images, but I use it for a signal that contains negative values. After training, the autoencoder only reconstructs the positive part of the signal, it does not produce negative values. Can anyone spot where the problem is or explain why this is the case?

Output:
If you used the exact code as the one shown in the example you put the link in, then at the end of the decoder you have x = torch.sigmoid(self.decConv2(x)) which take the real number line and outputs numbers between [0, 1]. This is why the network is unable to output negative numbers.
If you want to change the model to output negative numbers as well, remove the sigmoid function.
This means of course that you also have to change the loss function with which you train your model since the BCE loss is only good for outputs in the range of [0, 1].
As a recommendation I would suggest anyone to use the BCE with logits loss and avoid using the sigmoid in the decoder since this method incorporates the sigmoid and the BCE loss in a more numerically stable manner.


Instruction: How do you implement SVoice?

Input:
I'm trying to use Facebook's SVoice to split out different speakers in my audio file using python. I found a library that implemented it here:
https://github.com/facebookresearch/svoice
However, I'm having trouble running it. The readme discusses how to train my own dataset which I can't really do since I don't have the noises parsed out in my own audio files. It also talks about how I can separate my own file using one of the models in the models folder but I get the following error when I try to follow the readme and create a model from the toy dataset:
File "/mnt/c/Users/imrea/PycharmProjects/svoice/svoice/data/audio.py", line 34, in find_audio_files
    siginfo, _ = torchaudio.info(file)
TypeError: cannot unpack non-iterable AudioMetaData object

How do I run this to test the output on an audio file of my own? Has anyone used this before? Any guidance would be greatly appreciated!

Output:
You need to have torchaudio version 0.6.0
Try:
pip install torch==1.6.0+cu101 torchvision==0.7.0+cu101 torchaudio==0.6.0 -f https://download.pytorch.org/whl/torch_stable.html
This worked for me.


Instruction: Viewing Pytorch weights from a *.pth file

Input:
I have a .pth file created with Pytorch with weights. How would I be able to view the weights from this file?
I tried this code to load and view but it was not working (as a newbie, I might be entirely wrong)-
import torch
import torchvision.models as models

torch.save('weights\kharif_crops_final.pth')

models.load_state_dict(torch.load('weights\kharif_crops_final.pth'))
models.eval()
print(models)

Output:
import torch

model = torch.load('path')
print(model)

(Verify and confirm)


Instruction: How do I load a fine-tuned AllenNLP BERT-SRL model using BertPreTrainedModel.from_pretrained()?

Input:
I have fine-tuned a BERT model for semantic role labeling, using AllenNLP. This produces a model directory (serialization directory, if I recall?) that contains the following:
best.th
config.json
meta.json
metrics_epoch_0.json
metrics_epoch_10.json
metrics_epoch_11.json
metrics_epoch_12.json
metrics_epoch_13.json
metrics_epoch_14.json
metrics_epoch_1.json
metrics_epoch_2.json
metrics_epoch_3.json
metrics_epoch_4.json
metrics_epoch_5.json
metrics_epoch_6.json
metrics_epoch_7.json
metrics_epoch_8.json
metrics_epoch_9.json
metrics.json
model_state_e14_b0.th
model_state_e15_b0.th
model.tar.gz
out.log
training_state_e14_b0.th
training_state_e15_b0.th
vocabulary

Where vocabulary is a folder with labels.txt and non_padded_namespaces.txt.
I'd now like to use this fine-tuned model BERT model as the initialization when learning a related task, event extraction, using this library: https://github.com/wilsonlau-uw/BERT-EE (ie I want to exploit some transfer learning). The config.ini file has a line for fine_tuned_path, where I can specify an already-fine-tuned model that I want to use here. I provided the path to the AllenNLP serialization directory, and I got the following error:
2022-04-05 13:07:28,112 -  INFO - setting seed 23
2022-04-05 13:07:28,113 -  INFO - loading fine tuned model in /data/projects/SRL/ser_pure_clinical_bert-large_thyme_and_ontonotes/
Traceback (most recent call last):
  File "main.py", line 65, in <module>
    model = BERT_EE()
  File "/data/projects/SRL/BERT-EE/model.py", line 88, in __init__
    self.__build(self.use_fine_tuned)
  File "/data/projects/SRL/BERT-EE/model.py", line 118, in __build
    self.__get_pretrained(self.fine_tuned_path)
  File "/data/projects/SRL/BERT-EE/model.py", line 110, in __get_pretrained
    self.__model = BERT_EE_model.from_pretrained(path)
  File "/home/richier/anaconda3/envs/allennlp/lib/python3.7/site-packages/transformers/modeling_utils.py", line 1109, in from_pretrained
    f"Error no file named {[WEIGHTS_NAME, TF2_WEIGHTS_NAME, TF_WEIGHTS_NAME + '.index', FLAX_WEIGHTS_NAME]} found in "
OSError: Error no file named ['pytorch_model.bin', 'tf_model.h5', 'model.ckpt.index', 'flax_model.msgpack'] found in directory /data/projects/SRL/ser_pure_clinical_bert-large_thyme_and_ontonotes/ or `from_tf` and `from_flax` set to False.

Of course, the serialization directory doesn't have any of those files, hence the error. I tried unzipping model.tar.gz but it only has:
config.json
weights.th
vocabulary/
vocabulary/.lock
vocabulary/labels.txt
vocabulary/non_padded_namespaces.txt
meta.json

Digging into the codebase of the GitHub repo I linked above, I can see that BERT_EE_model inherits from BertPreTrainedModel from the transformers library, so the trick would seem to be getting the AllenNLP model into a format that BertPreTrainedModel.from_pretrained() can load...?
Any help would be greatly appreciated!

Output:
I believe I have figured this out. Basically, I had to re-load my model archive, access the underlying model and tokenizer, and then save those:
from allennlp.models.archival import load_archive
from allennlp_models.structured_prediction import SemanticRoleLabeler, srl, srl_bert

archive = load_archive('ser_pure_clinical_bert-large_thyme_and_ontonotes/model.tar.gz')

bert_model = archive.model.bert_model #type is transformers.models.bert.modeling_bert.BertModel
bert_model.save_pretrained('ser_pure_clinical_bert-large_thyme_and_ontonotes_save_pretrained/')

bert_tokenizer = archive.dataset_reader.bert_tokenizer
bert_tokenizer.save_pretrained('ser_pure_clinical_bert-large_thyme_and_ontonotes_save_pretrained/')

(This last part is probably less interesting to most folks, but also, in the config.ini I mentioned, the directory 'ser_pure_clinical_bert-large_thyme_and_ontonotes_save_pretrained' needed to be passed to the line pretrained_model_name_or_path not to fine_tuned_path.)


Instruction: ANN not training accurately as i am not getting a better loss reduction

Input:
Just starting up with regression and it seems i'm not getting something right please what am i doing wrong here as my loss is not reducing.
import torch
from torch import nn
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split

df = pd.read_excel('Folds5x2_pp.xlsx')

x = df.iloc[:,:-1].values
y = df.iloc[:,-1].values

x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2, random_state=0)

class ANN(nn.Module):
  def __init__(self, input, output):
    super(ANN, self).__init__()
    self.fc1 = nn.Linear(input, 6)
    self.r1 = nn.ReLU()
    self.fc2 = nn.Linear(6, output)

  def forward(self, x):
    return self.fc2(self.r1(self.fc1(x)))

f, s = x.shape
ann = ANN(s, 1)

criterion = nn.MSELoss()
optimizer = torch.optim.Adam(ann.parameters(), lr=0.01)

x = torch.from_numpy(x_train.astype(np.float32))
y = torch.from_numpy(y_train.astype(np.float32))
for i in range(100):
  y_pred = ann(x)
  loss = criterion(y_pred, y)
  print(f"i: {i}, loss: {loss.item()}")
  loss.backward()
  optimizer.step()
  optimizer.zero_grad()

Output:
You should put optimizer.zero_grad() first, because the gradient will be relative to the previous batch of data if you don't zero it out.
Like this:
for i in range(100):
  y_pred = ann(x)
  loss = criterion(y_pred, y)
  print(f"i: {i}, loss: {loss.item()}")
  optimizer.zero_grad()
  loss.backward()
  optimizer.step()


Instruction: Get Model Summary with `torchsummary` pip Package

Input:
I am trying to get a good summary of my deep learning model like Keras summary function (can be found in here).
For that, what I have found is torch-summary pip package (details can be found here) is the best package I have found from this question.
My pytorch model is like this-
class DeepLearningModel(Module):
    # define model elements
    def __init__(self, n_inputs=34):
        super(DeepLearningModel, self).__init__()
        # input to first hidden layer
        self.hidden1 = Linear(n_inputs, 10)
        kaiming_uniform_(self.hidden1.weight, nonlinearity='relu')
        self.act1 = ReLU()
        # second hidden layer
        self.hidden2 = Linear(10, 8)
        kaiming_uniform_(self.hidden2.weight, nonlinearity='relu')
        self.act2 = ReLU()
        # third hidden layer and output
        self.hidden3 = Linear(8, 1)
        xavier_uniform_(self.hidden3.weight)
        self.act3 = Sigmoid()

    # forward propagate input
    def forward(self, X):
        # input to the first hidden layer
        X = self.hidden1(X)
        X = self.act1(X)
         # second hidden layer
        X = self.hidden2(X)
        X = self.act2(X)
        # third hidden layer and output
        X = self.hidden3(X)
        X = self.act3(X)
        return X

And to view the model's summary with the package, I am using this-
model_stats = summary(my_model, input_size=(1, 34, 8))

But for that, I am finding this error-
  Message=mat1 and mat2 shapes cannot be multiplied (68x8 and 34x10)
  Source=D:\Education\0. Research\1. Computer Science Knowledge Graph\Code\Terms Extractor\TermExtractor\BinaryClassifier\DeepLearningModel.py
  StackTrace:
  File "D:\Education\0. Research\1. Computer Science Knowledge Graph\Code\Terms Extractor\TermExtractor\BinaryClassifier\DeepLearningModel.py", line 25, in forward (Current frame)
    X = self.hidden1(X)
  File "D:\Education\0. Research\1. Computer Science Knowledge Graph\Code\Terms Extractor\TermExtractor\BinaryClassifier\DeepLearningClassifier.py", line 170, in printModel
    model_stats = summary(self.model, (1, 34, 8))
  File "D:\Education\0. Research\1. Computer Science Knowledge Graph\Code\Terms Extractor\TermExtractor\BinaryClassifier\main-caller.py", line 11, in <module>
    model.printModel()

So, I am not sure what values should I put in the input_size parameter. Can anyone please help me find the issue or find my actual input shape for the model for getting a summary?

Output:
Linear expects the number of channels on the last axis of the input to be in_features.
model_stats = summary(my_model, input_size=(1, 8, 34))


Instruction: Parametric estimation of a Gaussian Mixture Model

Input:
I am trying to train a model to estimate a GMM. However, the means of the GMM are calculated each time based on a mean_placement parameter. I am following the solution provided here, I'll copy and paste the original code:
import numpy as np
import matplotlib.pyplot as plt
import sklearn.datasets as datasets

import torch
from torch import nn
from torch import optim
import torch.distributions as D

num_layers = 8
weights = torch.ones(8,requires_grad=True)
means = torch.tensor(np.random.randn(8,2),requires_grad=True)
stdevs = torch.tensor(np.abs(np.random.randn(8,2)),requires_grad=True)

parameters = [weights, means, stdevs]
optimizer1 = optim.SGD(parameters, lr=0.001, momentum=0.9)

num_iter = 10001
for i in range(num_iter):
    mix = D.Categorical(weights)
    comp = D.Independent(D.Normal(means,stdevs), 1)
    gmm = D.MixtureSameFamily(mix, comp)

    optimizer1.zero_grad()
    x = torch.randn(5000,2)#this can be an arbitrary x samples
    loss2 = -gmm.log_prob(x).mean()#-densityflow.log_prob(inputs=x).mean()
    loss2.backward()
    optimizer1.step()

    print(i, loss2)

What I would like to do is this:
num_layers = 8
weights = torch.ones(8,requires_grad=True)
means_coef = torch.tensor(10.,requires_grad=True)
means = torch.tensor(torch.dstack([torch.linspace(1,means_coef.detach().item(),8)]*2).squeeze(),requires_grad=True)
stdevs = torch.tensor(np.abs(np.random.randn(8,2)),requires_grad=True)
parameters = [means_coef]
optimizer1 = optim.SGD(parameters, lr=0.001, momentum=0.9)

num_iter = 10001
for i in range(num_iter):
    means = torch.tensor(torch.dstack([torch.linspace(1,means_coef.detach().item(),8)]*2).squeeze(),requires_grad=True)

    mix = D.Categorical(weights)
    comp = D.Independent(D.Normal(means,stdevs), 1)
    gmm = D.MixtureSameFamily(mix, comp)

    optimizer1.zero_grad()
    x = torch.randn(5000,2)#this can be an arbitrary x samples
    loss2 = -gmm.log_prob(x).mean()#-densityflow.log_prob(inputs=x).mean()
    loss2.backward()
    optimizer1.step()

    print(i, means_coef)
    print(means_coef)

However in this case the parameter is not updated and the grad value is always None. Any ideas how to fix this?

Output:
According to your instructions I have re-written your model.
If you run it you can see that all the parameters are changing after the model is optimized. I also have provided the graph  of the model at the end. You can simply modify the GMM class as you need if you want to make a new one.
import numpy as np
import matplotlib.pyplot as plt
import sklearn.datasets as datasets

import torch
from torch import nn
from torch import optim
import torch.distributions as D

class GMM(nn.Module):

    def __init__(self, weights, base, scale, n_cell=8, shift=0, dim=2):
        super(GMM, self).__init__()
        self.weight = nn.Parameter(weights)
        self.base = nn.Parameter(base)
        self.scale = nn.Parameter(scale)
        self.grid = torch.arange(1, n_cell+1)
        self.shift = shift
        self.n_cell = n_cell
        self.dim = dim

    def trsf_grid(self):
        trsf = (
            torch.log(self.scale * self.grid + self.shift)
            / torch.log(self.base)
            ).reshape(-1, 1)
        return trsf.expand(self.n_cell, self.dim)

    def forward(self, x, std):
        means = self.trsf_grid()
        mix = D.Categorical(self.weight)
        comp = D.Independent(D.Normal(means, std), 1)
        gmm = D.MixtureSameFamily(mix, comp)
        return -gmm.log_prob(x).mean()

if __name__ == "__main__":
    weight = torch.ones(8)
    base = torch.tensor(3.)
    scale = torch.tensor(1.)
    stds = torch.tensor(np.abs(np.random.randn(8,2)),requires_grad=False)
    model = GMM(weight, base, scale)
    print(list(model.parameters()))

    optimizer = optim.SGD(model.parameters(), lr=0.001, momentum=0.9)
    for i in range(1000):
        optimizer.zero_grad()
        x = torch.randn(5000,2)
        loss = model(x, stds)
        loss.backward()
        optimizer.step()

    print(list(model.parameters()))

In my case It returned the following parameters:
[Parameter containing:
tensor([1., 1., 1., 1., 1., 1., 1., 1.], requires_grad=True), Parameter containing:
tensor(3., requires_grad=True), Parameter containing:
tensor(1., requires_grad=True)]

[Parameter containing:
tensor([0.7872, 1.1010, 1.3390, 1.3757, 0.5122, 0.2884, 1.2597, 0.7597],
       requires_grad=True), Parameter containing:
tensor(3.3207, requires_grad=True), Parameter containing:
tensor(0.2814, requires_grad=True)]

which indeed shows that the parameters are updating.
Also you can see the computation graph below:


Instruction: Accuracy 0% for binary classification

Input:
I am using the OpenFL framework for doing Federated Learning experiments. I run their tutorial notebooks without problems, so for example I am able to run classification on MNIST and everything is ok.
Now I am using 2 clients with 2 different datasets. However, my accuracy is around 0% for a binary classification problem.
So, I have 2 classes, "neg" and "pos" for both datasets. Images of the first dataset are 3000x2951 while images of the second are 4892x4020. I resize both to 256x256. My network is a ResNet9 without any sigmoid at the end, because I am using BCEWithLogitsLoss(). Here a bit of code, to check if everything is ok:
optimizer_adam = optim.Adam(params_to_update, lr=1e-4)

def cross_entropy(output, target):
    """Binary cross-entropy metric
    """
    target = target.unsqueeze(1)
    criterion = nn.BCEWithLogitsLoss()
    loss = criterion(output, target.float())
    return loss

def train(net_model, train_loader, optimizer, device, loss_fn=cross_entropy, some_parameter=None):
    torch.manual_seed(0)
    device='cpu'
    function_defined_in_notebook(some_parameter)

    train_loader = tqdm.tqdm(train_loader, desc="train")
    net_model.train()
    net_model.to(device)

    losses = []

    for data, target in train_loader:
        data, target = torch.tensor(data).to(device), torch.tensor(
            target).to(device, dtype=torch.int64)
        optimizer.zero_grad()
        #data = data.type(torch.LongTensor)
        #target = target.type(torch.LongTensor)
        output = net_model(data)
        loss = loss_fn(output=output, target=target)
        loss.backward()
        optimizer.step()
        losses.append(loss.detach().cpu().numpy())

    return {'train_loss': np.mean(losses),}

@task_interface.register_fl_task(model='net_model', data_loader='val_loader', device='device')
def validate(net_model, val_loader, device):
    torch.manual_seed(0)
    device = torch.device('cpu')
    net_model.eval()
    net_model.to(device)

    val_loader = tqdm.tqdm(val_loader, desc="validate")
    val_score = 0
    total_samples = 0

    with torch.no_grad():
        for data, target in val_loader:
            samples = target.shape[0]
            total_samples += samples
            data, target = torch.tensor(data).to(device), \
                torch.tensor(target).to(device, dtype=torch.int64)
            output = net_model(data)

            pred = (output >= 0.5).long() # Binarize predictions to 0 and 1
            val_score = (pred == target).sum().cpu().item()/data.size(0)

            #val_score += pred.eq(target).sum().cpu().numpy()

    return {'acc': val_score / total_samples,}

I think that all this is correct. So the only part that can be wrong is when I import the data because in this federated learning framework is a bit tricky. Basically my datasets are organized both in this way: /Dataset1(2)/Train(Test)/neg(pos)/images.png. I want to extract x_train, y_train, x_test and y_test because I am following exactly the structure of a tutorial that works. So this is my proposed solution:
def download_data(self):
        """Download prepared dataset."""
        image_list_train = []
        image_list_test = []
        x_train = []
        y_train = []
        x_test = []
        y_test = []
        base_dir_train = 'Montgomery_real_splitted/TRAIN/'
        base_dir_test = 'Montgomery_real_splitted/TEST/'
        for f in sorted(os.listdir(base_dir_train)):
            if os.path.isdir(base_dir_train+f):
                print(f"{f} is a target class")
                for i in sorted(os.listdir(base_dir_train+f)):
                    y_train.append(f)
                    im = Image.open(base_dir_train+f+'/'+i)
                    x_train.append(im)
        for f in sorted(os.listdir(base_dir_test)):
            if os.path.isdir(base_dir_test+f):
                print(f"{f} is a target class")
                for i in sorted(os.listdir(base_dir_test+f)):
                    y_test.append(f)
                    imt=Image.open(base_dir_test+f+'/'+i)
                    x_test.append(imt)
        y_train = np.array(y_train)
        y_test = np.array(y_test)

        for i in range(len(y_train)):
            if y_train[i]=="neg":
                y_train[i]=0
            else:
                y_train[i]=1
        y_train = y_train.astype(np.uint8)

        for i in range(len(y_test)):
            if y_test[i]=="neg":
                y_test[i]=0
            else:
                y_test[i]=1
        y_test = y_test.astype(np.uint8)

        print('Mont-china data was loaded!')
        return (x_train, y_train), (x_test, y_test)

This code above is in a python script needed to load the data. Then, inside the Jupyter notebook I have these cells in order to import the dataset:
normalize = T.Normalize(
    mean=[0.1307],
    std=[0.3081]
)

augmentation = T.RandomApply(
    [T.RandomHorizontalFlip(),
     T.RandomRotation(10)],
    p=.8
)

training_transform = T.Compose(
    [T.Resize((256,256)),
     augmentation,
     T.ToTensor()]
)

valid_transform = T.Compose(
    [T.Resize((256,256)),
     T.ToTensor()]
)

class TransformedDataset(Dataset):

    def __init__(self, dataset, transform=None, target_transform=None):
        """Initialize Dataset."""
        self.dataset = dataset
        self.transform = transform
        self.target_transform = target_transform

    def __len__(self):
        """Length of dataset."""
        return len(self.dataset)

    def __getitem__(self, index):
        img, label = self.dataset[index]
        label = self.target_transform(label) if self.target_transform else label
        img = self.transform(img) if self.transform else img
        return img, label

class MontChinaDataset(DataInterface):
    def __init__(self, **kwargs):
        self.kwargs = kwargs

    @property
    def shard_descriptor(self):
        return self._shard_descriptor

    @shard_descriptor.setter
    def shard_descriptor(self, shard_descriptor):
        """
        Describe per-collaborator procedures or sharding.

        This method will be called during a collaborator initialization.
        Local shard_descriptor  will be set by Envoy.
        """
        self._shard_descriptor = shard_descriptor

        self.train_set = TransformedDataset(
            self._shard_descriptor.get_dataset('train'),
            transform=training_transform
        )
        self.valid_set = TransformedDataset(
            self._shard_descriptor.get_dataset('val'),
            transform=valid_transform
        )

    def get_train_loader(self, **kwargs):
        """
        Output of this method will be provided to tasks with optimizer in contract
        """
        generator=torch.Generator()
        generator.manual_seed(0)
        return DataLoader(
            self.train_set, batch_size=self.kwargs['train_bs'], shuffle=True, generator=generator
            )

    def get_valid_loader(self, **kwargs):
        """
        Output of this method will be provided to tasks without optimizer in contract
        """
        return DataLoader(self.valid_set, batch_size=self.kwargs['valid_bs'])

    def get_train_data_size(self):
        """
        Information for aggregation
        """
        return len(self.train_set)

    def get_valid_data_size(self):
        """
        Information for aggregation
        """
        return len(self.valid_set)

fed_dataset = MontChinaDataset(train_bs=16, valid_bs=16)

The strange thing is that the loss decreases, while the accuracy remains 0 or around 0.
[12:29:44] METRIC   Round 0, collaborator env_one train result train_loss:  0.673127                                                           experiment.py:116
[12:29:53] METRIC   Round 0, collaborator env_one locally_tuned_model_validate result acc:  0.000000                                           experiment.py:116
[12:29:56] METRIC   Round 0, collaborator env_one aggregated_model_validate result acc:     0.000000                                           experiment.py:116
[12:30:49] METRIC   Round 0, collaborator env_two train result train_loss:  0.562856                                                           experiment.py:116
[12:31:14] METRIC   Round 0, collaborator env_two locally_tuned_model_validate result acc:  0.000000                                           experiment.py:116
[12:31:19] METRIC   Round 0, collaborator env_two aggregated_model_validate result acc:     0.000000                                           experiment.py:116
[12:31:21] METRIC   Round 0, collaborator Aggregator train result train_loss:       0.581464                                                   experiment.py:116
           METRIC   Round 0, collaborator Aggregator locally_tuned_model_validate result acc:       0.000000                                   experiment.py:116
[12:31:22] METRIC   Round 0, collaborator Aggregator aggregated_model_validate result acc:  0.000000                                           experiment.py:116
[12:31:39] METRIC   Round 1, collaborator env_one train result train_loss:  0.637785                                                           experiment.py:116
[12:31:41] METRIC   Round 1, collaborator env_one locally_tuned_model_validate result acc:  0.000000                                           experiment.py:116
[12:31:44] METRIC   Round 1, collaborator env_one aggregated_model_validate result acc:     0.000000                                           experiment.py:116
[12:31:55] METRIC   Round 1, collaborator env_two train result train_loss:  0.432979                                                           experiment.py:116
[12:32:00] METRIC   Round 1, collaborator env_two locally_tuned_model_validate result acc:  0.000000                                           experiment.py:116
[12:32:05] METRIC   Round 1, collaborator env_two aggregated_model_validate result acc:     0.000000                                           experiment.py:116
[12:32:08] METRIC   Round 1, collaborator Aggregator train result train_loss:       0.467540                                                   experiment.py:116
           METRIC   Round 1, collaborator Aggregator locally_tuned_model_validate result acc:       0.000000                                   experiment.py:116
           METRIC   Round 1, collaborator Aggregator aggregated_model_validate result acc:  0.000000

And this goes on for several rounds

Output:
I'm not sure if this will solve your problem, but your validation code has some bugs (two new lines annotated below):
@task_interface.register_fl_task(model='net_model', data_loader='val_loader', device='device')
def validate(net_model, val_loader, device):
    torch.manual_seed(0)
    device = torch.device('cpu')
    net_model.eval()
    net_model.to(device)

    val_loader = tqdm.tqdm(val_loader, desc="validate")
    val_score = 0
    total_samples = 0

    with torch.no_grad():
        for data, target in val_loader:
            samples = target.shape[0]
            total_samples += samples
            data, target = torch.tensor(data).to(device), \
                torch.tensor(target).to(device, dtype=torch.int64)
            output = net_model(data)

            ##new line vvv
            output = torch.sigmoid(output) #compress output into prob distribution
            pred = (output >= 0.5).long() # Binarize predictions to 0 and 1

            ##changed line below
            val_score += (pred == target).sum().cpu().item() ###/data.size(0)

            #val_score += pred.eq(target).sum().cpu().numpy()

    return {'acc': val_score / total_samples,}

Essentially there are two issues:

You're comparing to 0.5 when you haven't put your output through a sigmoid. I know you said that you didn't do this because of your loss function, and that is correct, however you must use a sigmoid/softmax in eval mode.
You're dividing your val_score by both data.size(0) (batch size?) and then also total_samples which is NOT the number of batches, but the count of all of your data.
You weren't increasing val_score every iteration, you were resetting it. If you have a lot of batches this would explain why it was 0 or close to 0.

Hopefully these fixes should get you closer to your goal!


Instruction: numpy how to speed up tanh?

Input:
numpy tanh seems much slower than its pytorch equivalence:
import torch
import numpy as np

data=np.random.randn(128,64,32).astype(np.float32)
%timeit torch.tanh(torch.tensor(data))
%timeit np.tanh(data)

820 µs ± 24.6 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)
3.89 ms ± 95.4 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

is there a way to speed up tanh in numpy? Thanks!

Output:
You could try with numexpr as follows:
pip install numexpr

Then:
import numexpr as ne
import numpy as np

data=np.random.randn(128,64,32).astype(np.float32)

resne = ne.evaluate("tanh(data)")
resnp = np.tanh(data)

Then check all close:
In [16]: np.allclose(resne,resnp)
Out[16]: True

And check timings:
In [14]: %timeit res = ne.evaluate("tanh(data)")
311 µs ± 1.26 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

In [15]: %timeit np.tanh(data)
1.85 ms ± 7.43 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)


Instruction: How to fix the error where the target batch size does not match when I use CrossEntropyLoss function?

Input:
I am working on a trainning task with CNN. When I created the loss function with CrossEntropyLoss and trained the dataset, the error reminded me that the batch size is not matched.
This is the main code for trainning:
net = SimpleConvolutionalNetwork()

train_history, val_history = train(net, batch_size=32, n_epochs=10, learning_rate=0.001)

plot_losses(train_history, val_history)

This is the neuron network code:
class SimpleConvolutionalNetwork(nn.Module):

  # Q: why the scope of input not changed after relu??

  def __init__(self) -> None:
      super(SimpleConvolutionalNetwork, self).__init__()

      # define convolutional filting layer(3 grids) and output size(18 channels)
      self.conv1 = nn.Conv2d(3, 18, kernel_size=3, stride=1, padding=1)

      # define pooling layer with max-pooling function
      self.pool = nn.MaxPool2d(kernel_size=2, stride=2, padding=0)

      # define FCL and output layer by Linear function
      self.fc1 = nn.Linear(18*16*16, 64)
      self.fc2 = nn.Linear(64, 10)

  # Q: where the pooling layer??

  def forward(self, x):
    # input shape: 3(grids) * 32 * 32(32*32 is the scope of each grid)
    # filted by conv1 defined in the construction function
    # then relu the filted x
    x = F.relu(self.conv1(x))

    # now let 18*32*32 -> 18*16*16
    x = x.view(-1, 18*16*16)

    # two step for 18*16*16(totally 4608) -> 64
    # output by FC firstly, then relu again the output
    x = F.relu(self.fc1(x))

    # 64 -> 10 finally
    x = self.fc2(x)
    return x

In the train function, the error place is at the construction of loss function. Because it is a very long context, the main part is showed below:
def train(net, batch_size, n_epochs, learning_rate):
...
    # load the training dataset
  train_loader = get_train_loader(batch_size)

  # get validation dataset
  val_loader = get_val_loader(batch_size)

  # set batch size
  n_minibatches = len(train_loader)

  # set loss function and validation test checking
  criterion, optimizer = createLossAndOptimizer(net, learning_rate)

  train_history = []
  val_history = []

  training_start_time = time.time()
  best_error = np.inf
  best_model_path = "best_model_path"

  # GPU if possible
  net = net.to(device)

  for epoch in range(n_epochs):

    running_loss = 0.0
    print_every = n_minibatches
    start_time = time.time()
    total_train_loss = 0.0

    # step1: training the datasets
    for i, (inputs, labels) in enumerate(train_loader):
      inputs, labels = inputs.to(device), labels.to(device)

      optimizer.zero_grad()

      # forward + backward + optimize
      outputs = net(inputs)
      loss = criterion(outputs, labels)
      loss.backward()
      optimizer.step()

      #print statistics
      running_loss += loss.item()
      total_train_loss += loss.item()

      # print every 10th of epoch
      if (i + 1) % (print_every + 1) == 0:
        print("Epoch {}, {:d}% \t train_loss: {:.2f} took: {:.2f}s".format(
          epoch + 1, int(100 * (i + 1) / n_minibatches), running_loss / print_every,
          time.time() - start_time))
        running_loss = 0.0
        start_time = time.time()

    train_history.append(total_train_loss / len(train_loader))
...

the loss construction funciton and dataset loading are like this:
def createLossAndOptimizer(net, learning_rate=0.001):

  # define a cross-entropy loss function:
  criterion = nn.CrossEntropyLoss()

  # optimizer include three parameters: net, learning rate, and
  # momentum rate for validate the dataset from over-fitting(default
  # value is 0.9)

  optimizer = opt.Adam(net.parameters(), lr=learning_rate)
  return criterion, optimizer

def get_train_loader(batch_size):
  return th.utils.data.DataLoader(train_set,batch_size=batch_size,sampler=train_sampler, num_workers=num_workers)

def get_val_loader(batch_size):
  return th.utils.data.DataLoader(train_set,batch_size=batch_size,sampler=train_sampler, num_workers=num_workers)

However, the error reminded me that the input batch size is more than the target batch size:
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-19-07b692e7a2bb> in <module>()
    173 net = SimpleConvolutionalNetwork()
    174
--> 175 train_history, val_history = train(net, batch_size=32, n_epochs=10, learning_rate=0.001)
    176
    177 plot_losses(train_history, val_history)

3 frames
/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction, label_smoothing)
   2844     if size_average is not None or reduce is not None:
   2845         reduction = _Reduction.legacy_get_string(size_average, reduce)
-> 2846     return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)
   2847
   2848

ValueError: Expected input batch_size (128) to match target batch_size (32).

I primarily thought that I mistakely set the incorrect parameters because of the 'labels' which is size 4. But I don't know how to fix it. Thanks for answering.

Output:
In forward method of SimpleConvolutionalNetwork after applying conv1, tensor x has shape of (batch_size, 18, 32, 32). So when doing x = x.view(-1, 18 * 16 * 16) shape of x turns to (batch_size * 4, 18 * 16 * 16) and because fully-connected layers applyed further don't change this new batch size, output has shape (batch_size * 4, 10). My suggestion would be using pooling right after convolution, like:
 x = F.relu(self.conv1(x))  # after that x will have shape (batch_size, 18, 32, 32)
 x = self.pool(x)           # after that x will have shape (batch_size, 18, 16, 16)

That way forward will return tensor with shape (batch_size, 10) and batch size mismatch error won't occur.


Instruction: Network stops learning once batchsize is set to > 1

Input:
I started switching from Keras to Pytorch and played around with some simple feedforward network today. It is supposed to learn the squaring operation, i.e. f(x) = x^2. However, my network only learns reasonably if I set the batchsize to 1. Any other batchsize yields very poor results. I tried also different learning rates between 1 and 0.0001 to see if this somehow fixed it and also tested a few changes to the network but to no avail. Could anyone tell me what I am doing wrong, i.e. why does my network not learn once I set the batchsize to any value above 1? Find a minimal working example below. Thank you for your help!
import numpy as np
from random import randint
import random
import time
from multiprocessing import Pool
import torch
from torch import nn
from torch.utils.data import Dataset, DataLoader
from torchvision import datasets, transforms

class SquareDataset(Dataset):
     def __init__(self, num_samples):
         super(Dataset, self).__init__()
         self.num_samples = num_samples
         self.train  = [None] * num_samples
         self.target = [None] * num_samples

         for i in range(0, num_samples):
             self.train[i]  = random.random() * randint(1, 10)
             self.target[i] =  self.train[i] ** 2

     def __len__(self):
         return self.num_samples

     def __getitem__(self, index):
        return self.train[index], self.target[index]

def trainNetwork(epochs=50):
    data_train = SquareDataset(num_samples=1000)
    data_train_loader = DataLoader(data_train, batch_size=1, shuffle=False)

    model = nn.Sequential(nn.Linear(1, 32),
                      nn.ReLU(),
                      nn.Linear(32, 32),
                      nn.ReLU(),
                      nn.Linear(32, 1))
    # Define the loss
    criterion = nn.MSELoss()
    # Optimizers require the parameters to optimize and a learning rate
    optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
    for e in range(epochs):
        running_loss = 0
        for number, labels in data_train_loader:
            optimizer.zero_grad()
            number = number.view(number.size(0), -1)
            output = model(number.float())
            loss = criterion(output, labels.float())
            loss.backward()
            optimizer.step()

            running_loss += loss.item()
        else:
            print(f"Training loss: {running_loss/len(data_train_loader)}")
    # some test outputs
    sample = torch.tensor([0.2])
    out = model(sample.float())
    print("Out:")
    print(out.item())
    sample = torch.tensor([1])
    out = model(sample.float())
    print("Out:")
    print(out.item())

trainNetwork()

Output:
On line loss = criterion(output, labels.float()) first tensor has shape (batch_size, 1) while labels has shape (batch_size, ). Hence when batch_size > 1 broadcasting occurs and this lead to wrong objective, case similar to this. To overcome issue rewrite loss line but with equal shapes, like:
loss = criterion(output.squeeze(-1), labels.float())


Instruction: How to use detectron2's augmentation with datasets loaded using register_coco_instances

Input:
I've trained a detectron2 model on custom data I labeled and exported in the coco format, but I now want to apply augmentation and train using the augmented data. How can I do that if I'm not using a custom DataLoader, but the register_coco_instances function.
cfg = get_cfg()
cfg.merge_from_file(model_zoo.get_config_file("COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml"))
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.5
cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url("COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml")
predictor = DefaultPredictor(cfg)
outputs = predictor(im)

train_annotations_path = "./data/cvat-corn-train-coco-1.0/annotations/instances_default.json"
train_images_path = "./data/cvat-corn-train-coco-1.0/images"
validation_annotations_path = "./data/cvat-corn-validation-coco-1.0/annotations/instances_default.json"
validation_images_path = "./data/cvat-corn-validation-coco-1.0/images"

register_coco_instances(
    "train-corn",
    {},
    train_annotations_path,
    train_images_path
)
register_coco_instances(
    "validation-corn",
    {},
    validation_annotations_path,
    validation_images_path
)
metadata_train = MetadataCatalog.get("train-corn")
dataset_dicts = DatasetCatalog.get("train-corn")

cfg = get_cfg()
cfg.merge_from_file(model_zoo.get_config_file("COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml"))
cfg.DATASETS.TRAIN = ("train-corn",)
cfg.DATASETS.TEST = ("validation-corn",)
cfg.DATALOADER.NUM_WORKERS = 2
cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url("COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml")  # Let training initialize from model zoo
cfg.SOLVER.IMS_PER_BATCH = 2
cfg.SOLVER.BASE_LR = 0.00025
cfg.SOLVER.MAX_ITER = 10000
cfg.SOLVER.STEPS = []
cfg.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 128
cfg.MODEL.ROI_HEADS.NUM_CLASSES = 4
os.makedirs(cfg.OUTPUT_DIR, exist_ok=True)
trainer = DefaultTrainer(cfg)
trainer.resume_or_load(resume=False)
trainer.train()

I saw in the documentation you can load a dataset and apply augmentation like this:
dataloader = build_detection_train_loader(cfg,
   mapper=DatasetMapper(cfg, is_train=True, augmentations=[
      T.Resize((800, 800))
   ]))

But I'm not using a custom dataloader, what is the best approach to do this?

Output:
From my experience, how you register your datasets (i.e., tell Detectron2 how to obtain a dataset named "my_dataset") has no bearing on what dataloader to use during training (i.e., how to load information from a registered dataset and process it into a format needed by the model).
So, you can register your dataset however you want - either by using the register_coco_instances function or by using the dataset APIs (DatasetCatalog, MetadataCatalog) directly; it doesn't matter. What matters is that you want to apply some transformation(s) during the data loading part.
Basically, you want to customise the data loading part which can only be achieved by using a custom dataloader (unless you perform offline augmentation which is likely not what you want).
Now, you don't need to define and use a custom dataloader directly in your top-level code. You can just create your own trainer deriving from DefaultTrainer, and override its build_train_loader method. This is as simple as the following.
class MyTrainer(DefaultTrainer):

    @classmethod
    def build_train_loader(cls, cfg):
        mapper = DatasetMapper(cfg, is_train=True, augmentations=[T.Resize((800, 800))])
        return build_detection_train_loader(cfg, mapper=mapper)

In your top-level code then, the only change required would be to use MyTrainer instead of DefaultTrainer.
trainer = MyTrainer(cfg)
trainer.resume_or_load(resume=False)
trainer.train()


Instruction: VGG parameters not updating

Input:
I am trying to train the VGG network from PyTorch to build a predictive model for the FashionMNIST dataset. But when I print the gradients out, it seems that the parameters are not updating and the gradient is always zero. Here is my implementation
## Specify Batch Size
train_batch_size = 32
test_batch_size = 32

## Specify Image Transforms
img_transform = transforms.Compose([
    transforms.Resize((64,64)),
    transforms.ToTensor(),
    transforms.Normalize((0.5,), (0.5,))
])

## Download Datasets
train_data = FashionMNIST('./data', transform=img_transform, download=True, train=True)
test_data = FashionMNIST('./data', transform=img_transform, download=True, train=False)

## Initialize Dataloaders
training_dataloader = DataLoader(train_data, batch_size=train_batch_size, shuffle=True)
test_dataloader = DataLoader(test_data, batch_size=test_batch_size, shuffle=True)

vgg16 = models.vgg16()
model_a = vgg16
model_a.classifier[6] = nn.Linear(4096, 10) # to match the output dimension FashionMNIST
model_a.cuda()

# Hyperparameters and weights init
num_epochs = 50
batch_size = 196 #64
learning_rate = 1e-3
def init_weights(m):
    if isinstance(m, nn.Linear):
        torch.nn.init.xavier_uniform(m.weight)
        m.bias.data.fill_(0.01)
        print("Weights initialized using xavier_uniform")
model_a.apply(init_weights)

optimizer = torch.optim.Adam(model_a.parameters(), lr=learning_rate, weight_decay=1e-5)
scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=10, gamma=0.1)
criterion = nn.CrossEntropyLoss()

# Training
for epoch in tqdm(range(num_epochs)):
    for i, (images, labels) in enumerate(training_dataloader):
        images = torch.cat((images, images, images),1)
        optimizer.zero_grad()
        outputs = model_a(images.cuda())
        loss = criterion(outputs, labels.cuda())
        loss.backward()
        optimizer.step()

Output:
Changing the training loop to this will resolve the issue:
for i, (images, labels) in enumerate(training_dataloader):
        images = Variable(images.to(device))
        labels = Variable(labels.to(device))

        images = torch.cat((images, images, images),1)
        optimizer.zero_grad()
        outputs = model_b(images)

        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()


Instruction: When would I use model.to("cuda:1") as opposed to model.to("cuda:0")?

Input:
I have a user with two GPU's; the first one is AMD which can't run CUDA, and the second one is a cuda-capable NVIDIA GPU. I am using the code model.half().to("cuda:0"). I'm not sure if the invocation successfully used the GPU, nor am I able to test it because I don't have any spare computer with more than 1 GPU lying around.
In this case, does "cuda:0" mean the first device which can run CUDA, so it would've worked even if their first device was AMD? Or would I need to say "cuda:1" instead? How would I detect which number is the first CUDA-capable device?

Output:
I tested it and as I suspected the model.half().to("cuda:0") will put your model in the first available GPU with CUDA support i.e. NVIDIA GPU in your case, the AMD GPU isn't visible as a cuda device, feel safe to assume cuda:0 is only a CUDA enabled GPU, and AMD GPU won't be seen by your program.
Have a good day.


Instruction: How does the finetune on transformer (t5) work?

Input:
I am using pytorch lightning to finetune t5 transformer on a specific task. However, I was not able to understand how the finetuning works. I always see this code :
tokenizer = AutoTokenizer.from_pretrained(hparams.model_name_or_path)                                     model = AutoModelForSeq2SeqLM.from_pretrained(hparams.model_name_or_path)
I don't get how the finetuning is done, are they freezing the whole model and training the head only, (if so how can I change the head) or are they using the pre-trained model as a weight initializing? I have been looking for an answer for couple days already. Any links or help are appreciated.

Output:
If you are using PyTorch Lightning, then it won't freeze the head until you specify it do so. Lightning has a callback which you can use to freeze your backbone and training only the head module. See Backbone Finetuning
Also checkout Ligthning-Flash, it allows you to quickly build model for various text tasks and uses Transformers library for backbone. You can use the Trainer to specify which kind of finetuning you want to apply for your training.
Thanks


Instruction: Pytorch: How to transform image patches into matrix of feature vectors?

Input:
For use as input in a neural network, I want to obtain a matrix of feature vectors from image patches. I'm using the Fashion-MNIST dataset (28x28 images) and have used Tensor.unfold to obtain patches (16 7x7 patches) by doing:
#example on one image
mnist_train = torchvision.datasets.FashionMNIST(
        root="../data", train=True, transform=transforms.Compose([transforms.ToTensor()]), download=True)
x = mnist_train[0][0][-1, :, :]
x = x.unfold(0, 7, 7).unfold(1, 7, 7)
x.shape
>>> torch.Size([4, 4, 7, 7])

Here I end up with a 4x4 tensor of 7x7 patches, however I want to vectorize each patch to obtain a matrix X with dimensions (16: number of patches x d: dimensions of feature vector). I'm unsure whether flatten() can be used here and how I would go about using it.

Output:
To close this out, moving the content of the comments to here:
#example on one image
mnist_train = torchvision.datasets.FashionMNIST(
    root="../data", train=True,
transform=transforms.Compose([transforms.ToTensor()]), download=True)
x = mnist_train[0][0][-1, :, :]
x = x.unfold(0, 7, 7).unfold(1, 7, 7)
x.shape

Output:
>>> torch.Size([4, 4, 7, 7])

And then:
x.reshape(-1,7,7)
x.shape

Output:
torch.Size([16,7,7])


Instruction: Pytorch-> LSTM-> RuntimeError: input must have 3 dimensions, got 2

Input:
I'm facing error in LSTM input dimensions with following model code:
class Model(nn.Module):
def __init__(self, input_size, hidden_size, num_layers, num_keys):
    super(Model, self).__init__()
    self.hidden_size = hidden_size
    self.num_layers = num_layers
    self.lstm = nn.LSTM(input_size, hidden_size, num_layers, batch_first=True)
    self.fc = nn.Linear(hidden_size, num_keys)

def forward(self, x):
    h0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size).to(device)
    c0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size).to(device)
    out, _ = self.lstm(x, (h0, c0))
    out = self.fc(out[:, -1, :])
    return out

Input sequence shape: 2048, 10,1  -> (#batch, window_size, #input)
Input label shape: 2048

Following code I have for training model:
for epoch in range(num_epochs):  # Loop over the dataset multiple times
    train_loss = 0
    for step, (seq, label) in enumerate(dataloader):
        # Forward pass
        seq = seq.clone().detach().view(-1, window_size, input_size).to(device)

        output = model(seq)
        loss = criterion(output, label.to(device))
        print('step: ',step,'sequence: ', seq.shape, 'Label: ', label.shape, 'model output: ', output.shape)

        # Backward and optimize
        optimizer.zero_grad()
        loss.backward()
        train_loss += loss.item()
        optimizer.step()
        writer.add_graph(model, seq)

I get following error:
RuntimeError: input must have 3 dimensions, got 2

Can anyone tell what is the problem should I fixed, I used seq.unsequeeze(-1) but its not working.

Output:
I got it, the input data containing -1 values. I have used map function to convert negative integers into positive as following:

line = tuple(map(int, line.strip().split()))

Please make it sure for classification your data should contain positive number!


Instruction: How to get the ROC curve of a neural network?

Input:
I'm trying to get the ROC curve for my Neural Network. My network uses pytorch and im using sklearn to get the ROC curve. My model outputs the binary right and wrong and also the probability of the output.
output = model(batch_X)
_, pred = torch.max(output, dim=1)

I give the model both samples of input data (Am I doing this part right or should it only be 1 sample of the input data not both?)
I take the probability ( the _ ) and the labels of what both inputs should be and feed it to sklearn like so
nn_fpr, nn_tpr, nn_thresholds = roc_curve( "labels go here" , "probability go here" )

Next I plot it with.
plt.plot(nn_fpr,nn_tpr,marker='.')
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate' )
plt.show()

It comes out very accurately which my model is (0.0167% wrong out of 108,000), but I have a concave graph and I have been told it's normally not supposed to be concave. (attached pictures)

I have been using Neural Nets for a while but I have never been asked to plot the ROC curve. So my question is, am I doing this right? Also should it be both labels or just one? All the examples I have seen for neural networks use Keras which if I remember right has a probability function. Therefore I don't know if PyTorch outputs the probability in the way sklearn want's it. For all the other examples I can find aren't for Neural Networks and they have a probability function built in.

Output:
Function roc_curve expects array with true labels y_true and array with probabilities for positive class y_score (which usually means class 1). Therefore what you need is not
_, pred = torch.max(output, dim=1)

but simply (if your model outputs probabities, which is not default in pytorch)
probabilities = output[:, 1]

or (if your model output logits, which is common case in pytorch)
import torch.nn.functional as F

probabilities = F.softmax(output, dim=1)[:, 1]

After that, assuming that array with true labels called labels, and has shape (N,), you call roc_curve as:
y_score = probabilities.detach().numpy()
nn_fpr, nn_tpr, nn_thresholds = roc_curve(labels, y_score)

That way you'll get correct results (which wasn't the case with torch.max)
As recommendation -- for binary classification I would suggest using model with sigmoid on the end and one output (probability of positive class), like:
model = nn.Sequential(nn.Linear(input_dim, hidden_dim),
                      nn.ReLU(),
                      nn.Linear(hidden_dim, 1),
                      nn.Sigmoid())

That way you'll train model with nn.BCELoss, which expects probabilites (unlike nn.CrossEntropyLoss which expects logits). Also code to get roc curve gets simpler:
probabilites = model(batch_X)
y_score = probabilites.squeeze(-1).detach().numpy()
fpr, tpr, threshold = roc_curve(labels, y_score)

Take a look at gist where ROC curve created for neural network classificator.


Instruction: How can I find multiple maximum indices of a torch tensor?

Input:
If I have a tensor which has multiple maximum values how can I get all the indices of maximum value. I have tried torch.argmax(tensor) but it only gives me the first index.
>>> a_list = [3,23,53,32,53]
>>> a_tensor = torch.Tensor(a_list)
>>> a_tensor
tensor([ 3., 23., 53., 32., 53.])
>>> torch.max(a_tensor)
tensor(53.)
>>> torch.argmax(a_tensor)
tensor(2)

I have following function to do it but was wondering if there are more efficient approaches:
def max_tensor_indices(tensor_t,max_value):
  tensor_list=tensor_t[0]
  indices_list=[]
  for i in range(len(tensor_list)):
     if tensor_list[i]==max_value:
         indices_list.append(i)
  return indices_list

Output:
Find the maximum value, then find all elements with that value.
(x == torch.max(x)).nonzero()

Note: nonzero may also be called with as_tuple=True, which may be helpful.