Instruction: memmap arrays to pytorch and gradient accumulation

Input:
I have A Large dataset (> 62 GiB) after processing saved as two NumPy.memmap arrays one of the data and the other for the labels the dataset has these shapes (7390,60,224,224,3) , and (7390) and is NOT shuffled so i need to shuffle it first.
now i use tensorflow2 and used this code with my generator to manage memmap arrays before
def my_generator():
    for i in range(len(numpy_array)):
          yield numpy_array[i,:,:,:,:],np.array(labels[i]).reshape(1)

full_dataset = tf.data.Dataset.from_generator(
    generator=my_generator,
    output_types=(np.uint8,np.int32),
    output_shapes=((60,224,224,3),(1))
)

full_dataset = full_dataset.shuffle(SHUFFLE_BUFFER_SIZE, reshuffle_each_iteration=False)
train_dataset = full_dataset.take(train_size)
test_dataset = full_dataset.skip(train_size)
val_dataset = test_dataset.skip(test_size)
test_dataset = test_dataset.take(test_size)

That way i can train without loading to memory the entire dataset with shuffling and batching.
Now with this current model and dataset the vram is not enogh for more than 2 batches to be loaded as tensors.
and i can't train with batchsize of 2.
i thought of gradient accumulation but i couldn't do it with TF2 and i found it easy with pytorch but i can't find how to deal with the memmap arrays with shuffle and split as in tensorflow with generators.
so i need to know how to load the datset from pytorch with the same shuffling and batching in pytorch.
Or if someone has a readymade code for GA on TF2

Output:
I will just address the shuffle question.
Instead of shuffling with tf.data.Dataset, do it at the generator level. This should work:
class Generator(object):
    def __init__(self, images, labels, batch_size):
        self.images = images
        self.labels = labels
        self.batch_size  = batch_size
        self.idxs = np.arange(len(self.images))
        self.on_epoch_end()

    def on_epoch_end(self):
        # Shuffle the indices
        np.random.shuffle(self.idxs)

    def generator(self):
        i = 0
        while i < len(self.idxs):
            idx = self.idxs[i]
            yield (self.images[idx], self.labels[i])
            i += 1
        self.on_epoch_end()

    def batch_generator(self):
        it = iter(self.generator)
        while True:
            vals = [next(it) for i in range(self.batch_size)]
            images, labels = zip(*vals)
            yield images, labels

Then you can use it by
gen = Generator(...)
it = iter(gen)

batch = next(it)  # Call this every time you want a new batch

I'm sure pytorch has build in methods for this kind of stuff though


Instruction: Pytorch Set value for Tensor with index tensor

Input:
Suppose i have a 2d index Array of shape [B,1,N,2] i.e N points holding indexes on a target tensor of size [B,1,H,W].
what is the best way to assign value to all the indices in the tensor?
for example:
for b in batchsize:
  for i in N:
    target[b,0,ind[i,0],ind[i,1]] = 1

but not in loop form
thanks

Output:
If we look at this setup, you have a tensor target shaped (b, 1, h, w) and a tensor containing indices ind, shaped (b, 1, N, 2). You want to assign 1 to the N points per batch given by the two coordinates in ind.
The way I see it you could use torch.scatter_. We will stick with a 3D tensor since axis=1 is unused. Given a 3D tensor and argument value=1 and dim=1, .scatter_ operates on the input tensor as so:
input[i][index[i][j]] = 1

This does not exactly fit your setting since what would wish for is rather
input[i][index1[i][j]][index2[i][j]] = 1

In order to use scatter you could flatten target and unfold the values in ind accordingly.

Let's take an example:
>>> target = torch.zeros(2, 1, 10, 10)
>>> ind = torch.tensor([[[[0, 0], [1, 1], [2, 2]]],
                        [[[1, 2], [3, 4], [7, 8]]]])
tensor([[[[0, 0],
          [1, 1],
          [2, 2]]],

        [[[1, 2],
          [3, 0],
          [4, 2]]]])

We will start by splitting ind into xs and ys coordinates:
>>> x, y = ind[..., 0], ind[..., 1]

Unfold and reshape them:
>>> z = x*target.size(-1) + y
tensor([[[ 0,  4,  8]],

        [[ 5,  9, 14]]])

Flatten target:
>>> t = target.flatten(2)
tensor([[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]])

Then scatter the 1s:
>>> t.scatter_(dim=2, index=z, value=1)
tensor([[[1., 0., 0., 0., 1., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0.]],

        [[0., 0., 0., 0., 0., 1., 0., 0., 0., 1., 0., 0., 0., 0., 1.]]])

And finally reshape back to the original shape:
>>> t.reshape_as(target)
tensor([[[[1., 0., 0.],
          [0., 1., 0.],
          [0., 0., 1.],
          [0., 0., 0.],
          [0., 0., 0.]]],

        [[[0., 0., 0.],
          [0., 0., 1.],
          [0., 0., 0.],
          [1., 0., 0.],
          [0., 0., 1.]]]])

In summary:
>>> x, y = ind[..., 0], ind[..., 1]
>>> z = x*target.size(-1) + y
>>> target.flatten(2).scatter_(dim=2, index=z, value=1).reshape_as(target)

This last line will mutate target.


Instruction: Pytorch: mat1 and mat2 shapes cannot be multiplied

Input:
I have set up a toy example for my first pytorch model:
x = torch.from_numpy(np.linspace(1,100,num=100))
y = torch.from_numpy(np.dot(2,x))

I have built the model as follows:
class Net(nn.Module):
    def __init__(self):
        super(Net,self).__init__()
        self.fc1 = nn.Linear(1,10)
        self.fc2 = nn.Linear(10,20)
        self.fc3 = nn.Linear(16,1)

    def forward(self,inputs):
        x = F.relu(self.fc1(inputs))
        x = F.relu(self.fc2(x))
        x = F.linear(self.fc3(x))
        return x

However, I have run into this error when I try to train:
RuntimeError: mat1 and mat2 shapes cannot be multiplied (1x20 and 1x10)

Here is the full code for reference:
import numpy as np # linear algebra
import torch
from torch.utils.data import Dataset
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

x = torch.from_numpy(np.linspace(1,100,num=100))
y = torch.from_numpy(np.dot(2,x))

class MyDataset(Dataset):
    def __init__(self):
        self.sequences = x
        self.target = y

    def __getitem__(self,i):
        return self.sequences[i], self.target[i]

    def __len__(self):
        return len(self.sequences)

class Net(nn.Module):
    def __init__(self):
        super(Net,self).__init__()
        self.fc1 = nn.Linear(1,10)
        self.fc2 = nn.Linear(10,20)
        self.fc3 = nn.Linear(16,1)

    def forward(self,inputs):
        x = F.relu(self.fc1(inputs))
        x = F.relu(self.fc2(x))
        x = F.linear(self.fc3(x))
        return x

model = Net().to('cpu')

# Generators
training_set = MyDataset()
loader = torch.utils.data.DataLoader(training_set, batch_size=20)

#criterion and optimizer
criterion = nn.MSELoss()
optimizer = optim.Adam(model.parameters(), lr=0.0001)

### Training
n_epochs = 12

for epoch in range(n_epochs):
    for inputs,target in loader:
        print(target)

        optimizer.zero_grad()

        output = model(inputs)

        loss = criterion(output,target)

        loss.backward()

        optimizer.step()

And the full error message:
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-107-d32fd01d3b41> in <module>
      9         optimizer.zero_grad()
     10
---> 11         output = model(inputs)
     12
     13         loss = criterion(output,target)

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

<ipython-input-103-aefe4823d2e8> in forward(self, inputs)
      7
      8     def forward(self,inputs):
----> 9         x = F.relu(self.fc1(inputs))
     10         x = F.relu(self.fc2(x))
     11         x = F.linear(self.fc3(x))

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/linear.py in forward(self, input)
     91
     92     def forward(self, input: Tensor) -> Tensor:
---> 93         return F.linear(input, self.weight, self.bias)
     94
     95     def extra_repr(self) -> str:

/opt/conda/lib/python3.7/site-packages/torch/nn/functional.py in linear(input, weight, bias)
   1690         ret = torch.addmm(bias, input, weight.t())
   1691     else:
-> 1692         output = input.matmul(weight.t())
   1693         if bias is not None:
   1694             output += bias

RuntimeError: mat1 and mat2 shapes cannot be multiplied (1x20 and 1x10)

Any advice would be very much appreciated.

Output:
There are four issues here:

Looking at the model's first layer, I assume your batch size is 100. In that case, the correct input shape should be (100, 1), not (100,). To fix this you could use unsqueeze(-1).

The input should be dtype float: x.float().

Layer self.fc3 has an incorrect sizing. The following is valid for self.fc2 with respect to self.fc2: nn.Linear(20,1).

Lastly F.linear is not a linear function (i.e. the identity function). It's an actually linear transformation (i.e. x @ A.T + b). Take a look at the documentation for further details. I don't believe this is what you were looking to do in your case.

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(1, 10)
        self.fc2 = nn.Linear(10, 20)
        self.fc3 = nn.Linear(20, 1)

    def forward(self,inputs):
        x = F.relu(self.fc1(inputs))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

Inference:
>>> x = torch.linspace(1, 100, 100).float().unsqueeze(-1)
>>> y_hat = Net()(x)
>>> y_hat.shape
torch.Size([100, 1])


Instruction: Memory Efficient Nearest Neighbour Algorithm

Input:
I have 10,00,000 agents, each associated with (x,y) coordinates. I am trying to find agents close to each other (radius=1.5). I tried to implement this using PyTorch:
X = torch.DoubleTensor(1000000,2).uniform_(0,10000)
torch.cdist(X,X,p=2)

However, with this the session crashes. I am running this on google colab. The same happened when I tried constructing the graph using, radius_neighbors_graph of scikit-learn package. It would be of great help if someone suggested a memory efficient way to implement the same.

Output:
I found three solutions,
Solution 1
import torch
x = torch.randn(3000000, 2).cuda()
y = x

# Turn our Tensors into KeOps symbolic variables:
from pykeops.torch import LazyTensor
x_i = LazyTensor( x[:,None,:] )
y_j = LazyTensor( y[None,:,:] )

# We can now perform large-scale computations, without memory overflows:
D_ij = ((x_i - y_j)**2).sum(dim=2)
D_ij.argKmin(20,dim=1)

Solution 2
M = 3000000
import numpy as np
from pykeops.numpy import LazyTensor as LazyTensor_np
x = np.random.rand(M, 2)
y = x
x_i = LazyTensor_np(
    x[:, None, :]
)  # (M, 1, 2) KeOps LazyTensor, wrapped around the numpy array x
y_j = LazyTensor_np(
    y[None, :, :]
)  # (1, N, 2) KeOps LazyTensor, wrapped around the numpy array y

D_ij = ((x_i - y_j) ** 2).sum(-1)  # **Symbolic** (M, N) matrix of squared distances
s_i = D_ij.argKmin(20,dim=1).ravel()  # genuine (M,) array of integer indices

Solution 3
from sklearn.neighbors import NearestNeighbors
import numpy as np
M = 3000000
x = np.random.rand(M, 2)
nbrs = NearestNeighbors(n_neighbors=20, algorithm='ball_tree').fit(x)
distances, indices = nbrs.kneighbors(x)

Although the execution time of all the three solutions is the same, a minute, the memory requirements are approximately 2GB, 1GB and 1.3GB, respectively. It would be great to hear ideas to lower the execution time.


Instruction: UNet loss is NaN + UserWarning: Warning: converting a masked element to nan

Input:
I'm training a UNet, which class looks like this:
class UNet(nn.Module):
def __init__(self):
    super().__init__()

    # encoder (downsampling)
    # Each enc_conv/dec_conv block should look like this:
    # nn.Sequential(
    #     nn.Conv2d(...),
    #     ... (2 or 3 conv layers with relu and batchnorm),
    # )
    self.enc_conv0 = nn.Sequential(
        nn.Conv2d(in_channels=3, out_channels=64, kernel_size=3, stride=1, padding=1),
        nn.BatchNorm2d(64),
        nn.ReLU(),
        nn.Conv2d(in_channels=64, out_channels=64, kernel_size=3, padding=1, stride=1),
        nn.BatchNorm2d(64),
        nn.ReLU()
        )
    self.pool0 = nn.MaxPool2d(kernel_size=2, stride=2, return_indices=False)  # 256 -> 128
    self.enc_conv1 = nn.Sequential(
        nn.Conv2d(in_channels=64, out_channels=128, kernel_size=3, padding=1),
        nn.BatchNorm2d(128),
        nn.ReLU(),
        nn.Conv2d(in_channels=128, out_channels=128, kernel_size=3, padding=1),
        nn.BatchNorm2d(128),
        nn.ReLU()
        )
    self.pool1 = nn.MaxPool2d(kernel_size=2, stride=2, return_indices=False) # 128 -> 64
    self.enc_conv2 = nn.Sequential(
        nn.Conv2d(in_channels=128, out_channels=256, kernel_size=3, padding=1),
        nn.BatchNorm2d(256),
        nn.ReLU(),
        nn.Conv2d(in_channels=256, out_channels=256, kernel_size=3, padding=1),
        nn.BatchNorm2d(256),
        nn.ReLU(),
        nn.Conv2d(in_channels=256, out_channels=256, kernel_size=3, padding=1),
        nn.BatchNorm2d(256),
        nn.ReLU()
        )
    self.pool2 = nn.MaxPool2d(kernel_size=2, stride=2) # 64 -> 32
    self.enc_conv3 = nn.Sequential(
        nn.Conv2d(in_channels=256, out_channels=512, kernel_size=3, stride=1, padding=1),
        nn.BatchNorm2d(512),
        nn.ReLU(),

        nn.Conv2d(in_channels=512, out_channels=512, kernel_size=3, stride=1, padding=1),
        nn.BatchNorm2d(512),
        nn.ReLU(),

        nn.Conv2d(in_channels=512, out_channels=512, kernel_size=3, stride=1, padding=1),
        nn.BatchNorm2d(512),
        nn.ReLU()
    )
    self.pool3 = nn.MaxPool2d(kernel_size=2, stride=2) # 32 -> 16

    # bottleneck
    self.bottleneck_conv = nn.Sequential(
        nn.Conv2d(in_channels=512, out_channels=1024, kernel_size=1, stride=1, padding=0),
        nn.BatchNorm2d(1024),
        nn.ReLU(),

        nn.Conv2d(in_channels=1024, out_channels=512, kernel_size=1, stride=1, padding=0),
        nn.BatchNorm2d(512),
        nn.ReLU()
    )

    # decoder (upsampling)
    self.upsample0 = nn.UpsamplingBilinear2d(scale_factor=2) # 16 -> 32
    self.dec_conv0 = nn.Sequential(
        nn.Conv2d(in_channels=512*2, out_channels=256, kernel_size=3, stride=1, padding=1),
        nn.BatchNorm2d(256),
        nn.ReLU(),

        nn.Conv2d(in_channels=256, out_channels=256, kernel_size=3, stride=1, padding=1),
        nn.BatchNorm2d(256),
        nn.ReLU(),

        nn.Conv2d(in_channels=256, out_channels=256, kernel_size=3, stride=1, padding=1),
        nn.BatchNorm2d(256),
        nn.ReLU()
    )
    self.upsample1 = nn.UpsamplingBilinear2d(scale_factor=2) # 32 -> 64
    self.dec_conv1 = nn.Sequential(
        nn.Conv2d(in_channels=256*2, out_channels=128, kernel_size=3, stride=1, padding=1),
        nn.BatchNorm2d(128),
        nn.ReLU(),

        nn.Conv2d(in_channels=128, out_channels=128, kernel_size=3, stride=1, padding=1),
        nn.BatchNorm2d(128),
        nn.ReLU(),

        nn.Conv2d(in_channels=128, out_channels=128, kernel_size=3, stride=1, padding=1),
        nn.BatchNorm2d(128),
        nn.ReLU()
    )
    self.upsample2 = nn.UpsamplingBilinear2d(scale_factor=2) # 64 -> 128
    self.dec_conv2 = nn.Sequential(
        nn.Conv2d(in_channels=128*2, out_channels=64, kernel_size=3, stride=1, padding=1),
        nn.BatchNorm2d(64),
        nn.ReLU(),

        nn.Conv2d(in_channels=64, out_channels=64, kernel_size=3, stride=1, padding=1),
        nn.BatchNorm2d(64),
        nn.ReLU()
    )
    self.upsample3 = nn.UpsamplingBilinear2d(scale_factor=2) # 128 -> 256
    self.dec_conv3 = nn.Sequential(
        nn.Conv2d(in_channels=64*2, out_channels=1, kernel_size=3, stride=1, padding=1),
        nn.BatchNorm2d(1),
        nn.ReLU(),

        nn.Conv2d(in_channels=1, out_channels=1, kernel_size=3, stride=1, padding=1),
        nn.BatchNorm2d(1),
        nn.ReLU(),

        nn.Conv2d(in_channels=1, out_channels=1, kernel_size=3, stride=1, padding=1),
        nn.BatchNorm2d(1)
    )

def forward(self, x):
    # encoder
    e0 = self.enc_conv0(x)
    pool0 = self.pool0(e0)
    e1 = self.enc_conv1(pool0)
    pool1 = self.pool1(e1)
    e2 = self.enc_conv2(pool1)
    pool2 = self.pool2(e2)
    e3 = self.enc_conv3(pool2)
    pool3 = self.pool3(e3)

    # bottleneck
    b = self.bottleneck_conv(pool3)

    # decoder
    d0 = self.dec_conv0(torch.cat([self.upsample0(b), e3], 1))
    d1 = self.dec_conv1(torch.cat([self.upsample1(d0), e2], 1))
    d2 = self.dec_conv2(torch.cat([self.upsample2(d1), e1], 1))
    d3 = self.dec_conv3(torch.cat([self.upsample3(d2), e0], 1))  # no activation
    return d3

Train method:
def train(model, opt, loss_fn, score_fn, epochs, data_tr, data_val):

torch.cuda.empty_cache()

losses_train = []
losses_val = []
scores_train = []
scores_val = []

for epoch in range(epochs):
    tic = time()
    print('* Epoch %d/%d' % (epoch+1, epochs))

    avg_loss = 0
    model.train()  # train mode
    for X_batch, Y_batch in data_tr:
        # data to device
        X_batch = X_batch.to(device)
        Y_batch = Y_batch.to(device)

        # set parameter gradients to zero
        opt.zero_grad()

        # forward
        Y_pred = model(X_batch)
        loss = loss_fn(Y_pred, Y_batch) # forward-pass
        loss.backward()  # backward-pass
        opt.step()  # update weights

        # calculate loss to show the user
        avg_loss += loss / len(data_tr)
    toc = time()
    print('loss: %f' % avg_loss)
    losses_train.append(avg_loss)

    avg_score_train = score_fn(model, iou_pytorch, data_tr)
    scores_train.append(avg_score_train)

    # show intermediate results
    model.eval()  # testing mode
    avg_loss_val = 0
    #Y_hat = # detach and put into cpu

    for X_val, Y_val in data_val:
      with torch.no_grad():
        Y_hat = model(X_val.to(device)).detach().cpu()

        loss = loss_fn(Y_hat, Y_val)
        avg_loss_val += loss / len(data_val)

    toc = time()
    print('loss_val: %f' % avg_loss_val)
    losses_val.append(avg_loss_val)

    avg_score_val = score_fn(model, iou_pytorch, data_val)
    scores_val.append(avg_score_val)

    torch.cuda.empty_cache()

    # Visualize tools
    clear_output(wait=True)
    for k in range(5):
        plt.subplot(2, 6, k+1)
        plt.imshow(np.rollaxis(X_val[k].numpy(), 0, 3), cmap='gray')
        plt.title('Real')
        plt.axis('off')

        plt.subplot(2, 6, k+7)
        plt.imshow(Y_hat[k, 0], cmap='gray')
        plt.title('Output')
        plt.axis('off')
    plt.suptitle('%d / %d - loss: %f' % (epoch+1, epochs, avg_loss))
    plt.show()

return (losses_train, losses_val, scores_train, scores_val)

However, when executing I get train_loss and val_loss both equal nan and also a warning. In addition, when plotting the segmented picture and the target one, the output picture is not shown. I tried to execute with different loss function, but still the same. There is probably something wrong with my class.
Could you please help me? Thanks in advance.

Output:
I am not sure if this is your error, but your last Convolution layer (self.dec_conv3) has looks odd. I would only reduce to 1 channel at the very last convolution and do not perform 2 Convolutions with 1 In and 1 Out channel. Also ending with a batchnorm can only produce normalized outputs, which could be far from what you really want:
self.dec_conv3 = nn.Sequential(
    nn.Conv2d(in_channels=64*2, out_channels=32, kernel_size=3, stride=1, padding=1),
    nn.BatchNorm2d(32),
    nn.ReLU(),

    nn.Conv2d(in_channels=32, out_channels=32, kernel_size=3, stride=1, padding=1),
    nn.BatchNorm2d(32),
    nn.ReLU(),
    nn.Conv2d(in_channels=32, out_channels=1, kernel_size=3, stride=1, padding=1)
)

It would be interesting if your loss is Nan already at the first iteration or only after a few iterations. Maybe, you use a loss function, that devides by zero?


Instruction: How can I make torch tensor?

Input:
I want to make a torch tensor composed of only 1 or -1.
I just try to use torch.empty() and torch.randn().
tmp = torch.randint(low=0, high=2, size=(4, 4))
tmp[tmp==0] = -1
tmp
>> tensor([[ 1, -1, -1,  1],
           [-1,  1,  1, -1],
           [-1, -1,  1,  1],
           [-1,  1, -1,  1]])

However, I don`t know what method is most efficient in time.
And I want to make this code to one line as possible because this code is positioned at __init__ ()

Output:
Why not:
tmp = -1 + 2 * torch.randint(low=0, high=2, size=(4, 4))


Instruction: How to convert Bounding Box coordinates to COCO format?

Input:
I need to convert the coordinates. I have this format: Horizontal and
Vertical coordinates of the top left and lower right of the element ((x1, y1) and (x2, y2)). And I need x_center y_center width height in this format. How can I do it?

Output:
The center and the size are simply
x_center = 0.5 * (x1 + x2)
y_center = 0.5 * (y1 + y2)
width = np.abs(x2 - x1)
height = np.abs(y2 - y1)

Note that by using np.abs when computing the width and height we do need to assume anything on the "order" of the first and second corners.
If you further want to normalize the center and size by the image size (img_w, img_h):
n_x_center = x_center / img_w
n_y_center = y_center / img_h
n_width = width / img_w
n_height = height / img_h


Instruction: RuntimeError: Error(s) in loading state_dict for DataParallel: Unexpected key(s) in state_dict: “module.scibert_layer.embeddings.position_ids”

Input:
I am getting the following error while trying to load a saved model checkpoint (.pth file).
RuntimeError: Error(s) in loading state_dict for DataParallel: Unexpected key(s) in state_dict: "module.scibert_layer.embeddings.position_ids"

I trained my sequence labeling model in nn.DataParallel (torch version 1.7.0) but am trying to load it without the nn.DataParallel (torch version 1.9.0). Currently, I understand that not using nn.DataParallel caused the issue of RuntimeError: Error(s) in loading state_dict for DataParallel:, but could it also be because I am using different versions of torch or training and loading the model checkpoint?
The model is wrapped in nn.DataParallel using the following chunk of code.
    if exp_args.parallel == 'true':
        if torch.cuda.device_count() > 1:
            model = nn.DataParallel(model, device_ids = [0, 1, 2, 3])
            print("Using", len(model.device_ids), " GPUs!")
            print("Using", str(model.device_ids), " GPUs!")
        model.to(f'cuda:{model.device_ids[0]}')
    elif exp_args.parallel == 'false':
        model = nn.DataParallel(model, device_ids = [0])

This is my model.
DataParallel(
  (module): SCIBERTPOSAttenCRF(
    (scibert_layer): BertModel(
      (embeddings): BertEmbeddings(
        (word_embeddings): Embedding(31090, 768, padding_idx=0)
        (position_embeddings): Embedding(512, 768)
        (token_type_embeddings): Embedding(2, 768)
        (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
        (dropout): Dropout(p=0.1, inplace=False)
      )
      (encoder): BertEncoder(
        (layer): ModuleList(
          (0): BertLayer(
            (attention): BertAttention(
              (self): BertSelfAttention(
                (query): Linear(in_features=768, out_features=768, bias=True)
                (key): Linear(in_features=768, out_features=768, bias=True)
                (value): Linear(in_features=768, out_features=768, bias=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
              (output): BertSelfOutput(
                (dense): Linear(in_features=768, out_features=768, bias=True)
                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
            )
            (intermediate): BertIntermediate(
              (dense): Linear(in_features=768, out_features=3072, bias=True)
            )
            (output): BertOutput(
              (dense): Linear(in_features=3072, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (1): BertLayer(
            (attention): BertAttention(
              (self): BertSelfAttention(
                (query): Linear(in_features=768, out_features=768, bias=True)
                (key): Linear(in_features=768, out_features=768, bias=True)
                (value): Linear(in_features=768, out_features=768, bias=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
              (output): BertSelfOutput(
                (dense): Linear(in_features=768, out_features=768, bias=True)
                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
            )
            (intermediate): BertIntermediate(
              (dense): Linear(in_features=768, out_features=3072, bias=True)
            )
            (output): BertOutput(
              (dense): Linear(in_features=3072, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (2): BertLayer(
            (attention): BertAttention(
              (self): BertSelfAttention(
                (query): Linear(in_features=768, out_features=768, bias=True)
                (key): Linear(in_features=768, out_features=768, bias=True)
                (value): Linear(in_features=768, out_features=768, bias=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
              (output): BertSelfOutput(
                (dense): Linear(in_features=768, out_features=768, bias=True)
                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
            )
            (intermediate): BertIntermediate(
              (dense): Linear(in_features=768, out_features=3072, bias=True)
            )
            (output): BertOutput(
              (dense): Linear(in_features=3072, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (3): BertLayer(
            (attention): BertAttention(
              (self): BertSelfAttention(
                (query): Linear(in_features=768, out_features=768, bias=True)
                (key): Linear(in_features=768, out_features=768, bias=True)
                (value): Linear(in_features=768, out_features=768, bias=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
              (output): BertSelfOutput(
                (dense): Linear(in_features=768, out_features=768, bias=True)
                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
            )
            (intermediate): BertIntermediate(
              (dense): Linear(in_features=768, out_features=3072, bias=True)
            )
            (output): BertOutput(
              (dense): Linear(in_features=3072, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (4): BertLayer(
            (attention): BertAttention(
              (self): BertSelfAttention(
                (query): Linear(in_features=768, out_features=768, bias=True)
                (key): Linear(in_features=768, out_features=768, bias=True)
                (value): Linear(in_features=768, out_features=768, bias=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
              (output): BertSelfOutput(
                (dense): Linear(in_features=768, out_features=768, bias=True)
                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
            )
            (intermediate): BertIntermediate(
              (dense): Linear(in_features=768, out_features=3072, bias=True)
            )
            (output): BertOutput(
              (dense): Linear(in_features=3072, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (5): BertLayer(
            (attention): BertAttention(
              (self): BertSelfAttention(
                (query): Linear(in_features=768, out_features=768, bias=True)
                (key): Linear(in_features=768, out_features=768, bias=True)
                (value): Linear(in_features=768, out_features=768, bias=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
              (output): BertSelfOutput(
                (dense): Linear(in_features=768, out_features=768, bias=True)
                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
            )
            (intermediate): BertIntermediate(
              (dense): Linear(in_features=768, out_features=3072, bias=True)
            )
            (output): BertOutput(
              (dense): Linear(in_features=3072, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (6): BertLayer(
            (attention): BertAttention(
              (self): BertSelfAttention(
                (query): Linear(in_features=768, out_features=768, bias=True)
                (key): Linear(in_features=768, out_features=768, bias=True)
                (value): Linear(in_features=768, out_features=768, bias=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
              (output): BertSelfOutput(
                (dense): Linear(in_features=768, out_features=768, bias=True)
                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
            )
            (intermediate): BertIntermediate(
              (dense): Linear(in_features=768, out_features=3072, bias=True)
            )
            (output): BertOutput(
              (dense): Linear(in_features=3072, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (7): BertLayer(
            (attention): BertAttention(
              (self): BertSelfAttention(
                (query): Linear(in_features=768, out_features=768, bias=True)
                (key): Linear(in_features=768, out_features=768, bias=True)
                (value): Linear(in_features=768, out_features=768, bias=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
              (output): BertSelfOutput(
                (dense): Linear(in_features=768, out_features=768, bias=True)
                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
            )
            (intermediate): BertIntermediate(
              (dense): Linear(in_features=768, out_features=3072, bias=True)
            )
            (output): BertOutput(
              (dense): Linear(in_features=3072, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (8): BertLayer(
            (attention): BertAttention(
              (self): BertSelfAttention(
                (query): Linear(in_features=768, out_features=768, bias=True)
                (key): Linear(in_features=768, out_features=768, bias=True)
                (value): Linear(in_features=768, out_features=768, bias=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
              (output): BertSelfOutput(
                (dense): Linear(in_features=768, out_features=768, bias=True)
                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
            )
            (intermediate): BertIntermediate(
              (dense): Linear(in_features=768, out_features=3072, bias=True)
            )
            (output): BertOutput(
              (dense): Linear(in_features=3072, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (9): BertLayer(
            (attention): BertAttention(
              (self): BertSelfAttention(
                (query): Linear(in_features=768, out_features=768, bias=True)
                (key): Linear(in_features=768, out_features=768, bias=True)
                (value): Linear(in_features=768, out_features=768, bias=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
              (output): BertSelfOutput(
                (dense): Linear(in_features=768, out_features=768, bias=True)
                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
            )
            (intermediate): BertIntermediate(
              (dense): Linear(in_features=768, out_features=3072, bias=True)
            )
            (output): BertOutput(
              (dense): Linear(in_features=3072, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (10): BertLayer(
            (attention): BertAttention(
              (self): BertSelfAttention(
                (query): Linear(in_features=768, out_features=768, bias=True)
                (key): Linear(in_features=768, out_features=768, bias=True)
                (value): Linear(in_features=768, out_features=768, bias=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
              (output): BertSelfOutput(
                (dense): Linear(in_features=768, out_features=768, bias=True)
                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
            )
            (intermediate): BertIntermediate(
              (dense): Linear(in_features=768, out_features=3072, bias=True)
            )
            (output): BertOutput(
              (dense): Linear(in_features=3072, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (11): BertLayer(
            (attention): BertAttention(
              (self): BertSelfAttention(
                (query): Linear(in_features=768, out_features=768, bias=True)
                (key): Linear(in_features=768, out_features=768, bias=True)
                (value): Linear(in_features=768, out_features=768, bias=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
              (output): BertSelfOutput(
                (dense): Linear(in_features=768, out_features=768, bias=True)
                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
            )
            (intermediate): BertIntermediate(
              (dense): Linear(in_features=768, out_features=3072, bias=True)
            )
            (output): BertOutput(
              (dense): Linear(in_features=3072, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
        )
      )
      (pooler): BertPooler(
        (dense): Linear(in_features=768, out_features=768, bias=True)
        (activation): Tanh()
      )
    )
    (lstmpos_layer): LSTM(44, 20, batch_first=True, bidirectional=True)
    (self_attention): MultiheadAttention(
      (out_proj): NonDynamicallyQuantizableLinear(in_features=40, out_features=40, bias=True)
    )
    (lstm_layer): LSTM(808, 512, batch_first=True, bidirectional=True)
    (hidden2tag): Linear(in_features=1024, out_features=2, bias=True)
    (crf_layer): CRF(num_tags=2)
  )
)

How should I go ahead with correctly wrapping the checkpoint in nn.DataParallel or should I use the correct version of the torch that could fix this problem?
I will be grateful for any help or hint.

Output:
The problem was not caused by the DataParallel library, but rather the transformers library (notice that I used the SciBERT model from huggingface transformers library).
The error was caused by using older versions of transformers and tokeniezrs. Upgrading the transformers version from 3.0.2 to 4.8.2 and tokeniezrs to 0.10.3


Instruction: How can I measure activation level of each nodes when predict using trained model?

Input:
I'm interesting in which nodes are activated and how much strongly activated when some input is entered to trained model.
The picture as bellow shows what I want to get from model.
(I want to know the degree of activation in every nodes)
As I know there are some techniques to visualize what the nodes(or filter) are paying attention to. (especially in CNN)
Is there any good way to measure how active each nodes are?
I usually using Keras. But pyTorch is ok too.

Output:
You are looking for activation map / grad cam ..., you can get a look at : https://keras.io/examples/vision/grad_cam/ and https://keras.io/examples/vision/visualizing_what_convnets_learn/
You can also try keract : https://github.com/philipperemy/keract
There are others github repository on the subject on explainability in the AI field, such as : https://github.com/XAI-ANITI/ethik
Alternatively, you can do it yourself it both keras and pytorch, for example by registering a hook in pytorch :
def get_feat_vector(self, img, model):
        with torch.no_grad():
            my_output = None

            def my_hook(module_, input_, output_):
                nonlocal my_output
                my_output = output_

            a_hook = model.layers[0].register_forward_hook(my_hook)
            model(img)
            a_hook.remove()
            return my_output

...

for element in val_dataloader:
    model.eval()
    feature_vect = get_feat_vector(element[0].float().cuda(), model)


Instruction: torch Tensor add dimension

Input:
I have a tensor with this size
torch.Size([128, 64])

how do I add one "dummy" dimension such as
torch.Size([1, 128, 64])

Output:
There are several ways:
torch.unsqueeze:
torch.unsqueeze(x, 0)

Using None (or np.newaxis):
x[None, ...]
# or
x[np.newaxis, ...]

reshape or view:
x.reshape(1, *x.shape)
# or
x.view(1, *x.shape)


Instruction: Pytorch giving cuda error on mac when creating a neural net

Input:
I have Python 3.8 and an installation of torch from using
pip3 install torch
When I try to create a model with 3 input dimensions, I get an error relating to cuda support.
I'm on a mac which doesn't have a GPU with cuda support. How can I prevent this error?
to reproduce:
import torch as T
import torch.nn as nn

def create(input_dims):
    return nn.Sequential(
                nn.Linear(*input_dims, 256),
                nn.ReLU(),
                nn.Linear(256, 256),
                nn.ReLU(),
                nn.Linear(256, 50),
                nn.Softmax(dim=-1)
    )

create(input_dims=(100,100))    #this works fine
create(input_dims=(100,100,3))  #this produces the error:

"""
Traceback (most recent call last):
  File "example.py", line 15, in <module>
    create(input_dims=(100,100,3))
  File "example.py", line 6, in create
    nn.Linear(*input_dims, 256),
  File "/Users/darrinwiley/Library/Python/3.8/lib/python/site-packages/torch/nn/modules/linear.py", line 81, in __init__
    self.weight = Parameter(torch.empty((out_features, in_features), **factory_kwargs))
  File "/Users/darrinwiley/Library/Python/3.8/lib/python/site-packages/torch/cuda/__init__.py", line 166, in _lazy_init
    raise AssertionError("Torch not compiled with CUDA enabled")
AssertionError: Torch not compiled with CUDA enabled
"""

Output:
As shown in the Pytorch docs, If you are on a mac, then Pytorch does not offer any way to install the cuda toolkit as shown in this image:

A way to solve this error is by either using some shared server that is on linux or windows or by running virtual machines.


Instruction: arguments and function call of LSTM in pytorch

Input:
Could anyone please explain me the below code:
import torch
import torch.nn as nn

input = torch.randn(5, 3, 10)
h0 = torch.randn(2, 3, 20)
c0 = torch.randn(2, 3, 20)

rnn = nn.LSTM(10,20,2)

output, (hn, cn) = rnn(input, (h0, c0))
print(input)

While calling rnn rnn(input, (h0, c0)) we gave arguments h0 and c0 in parenthesis. What is it supposed to mean? if (h0, c0) represents a single value then what is that value and what is the third argument passed here?
However, in the  line rnn = nn.LSTM(10,20,2)  we are passing arguments in LSTM function without paranthesis.
Can anyone explain me how this function call is working?

Output:
The assignment rnn = nn.LSTM(10, 20, 2) instanciates a new nn.Module using the nn.LSTM class. It's first three arguments are input_size (here 10), hidden_size (here 20) and num_layers (here 2).
On the other hand rnn(input, (h0, c0)) corresponds to actually calling the class instance, i.e. running __call__ which is roughly equivalent to the forward function of that module. The __call__ method of nn.LSTM takes in two parameters: input  (shaped (sequnce_length, batch_size, input_size), and a tuple of two tensors (h_0, c_0) (both shaped (num_layers, batch_size, hidden_size) in the basic use case of nn.LSTM)
Please refer to the PyTorch documentation whenever using builtins, you will find the exact definition of the parameters list (the arguments used to initialize the class instance) as well as the input/outputs specifications (whenever inferring with that said module).

You might be confused with the notation, here's a small example that could help:

tuple as input:
def fn1(x, p):
    a, b = p # unpack input
    return a*x + b

>>> fn1(2, (3, 1))
>>> 7

tuple as output
def fn2(x):
    return x, (3*x, x**2) # actually output is a tuple of int and tuple

>>> x, (a, b) = fn2(2) # unpacking
(2, (6, 4))

>>> x, a, b
(2, 6, 4)


Instruction: PyTorch new model function leaking gpu memory

Input:
I had a perfectly working code, until I had to do some modification to make it more flexible with some hyperparameters.
In the bellow snippet you can see my forward function which before worked using the 2 commented lines. As I needed more flexibility in the nn.Linear from self.classifier, I made the function classifierAdaptive() which does the same thing but allows some parameters to change in the nn.Linear.
Previous GPU Ram usage was around 7GB/11GB, now I am running out of GPU Ram and I can't figure out why. I can't think of any variable to cause memory issues.
Edit:
The dataset is of images of size 2200x2200, from which I make small cuts to train on. These cuts are currently hardcoded as in 6 * 6.
Desired behaviour will be to only have 1 nn.Linear which would work for all img.shapes. However, as @jsho said, the mode maybe is creating too many layers instead of just 1 and overwriting?
I cannot include the entire code for debugging. It would be too much. But if needed can be seen here without the bellow code modification.
    self.classifier = nn.Sequential(
        nn.Linear(512 * 6 * 6, 1024),
        nn.LeakyReLU(0.2, True),
        nn.Linear(1024, 1),
        nn.Sigmoid()
    )

def classifierAdaptive(self,img):
    out = torch.flatten(img, 1)
    outLinear = nn.Linear(512 * img.shape[2] * img.shape[3], 1024).cuda()
    out = outLinear(out).cuda()
    out = self.lrelu(out) # self.lrelu = nn.LeakyReLU(0.2, True)
    out = self.dens2(out) # self.dens2 = nn.Linear(1024, 1)
    out = self.sig(out) # self.sig = nn.Sigmoid()

    return out

def forward(self, x):
    out = self.features(x)

    # out = torch.flatten(out, 1)
    # out = self.classifier(out)

    out = self.classifierAdaptive(out)

    return out

Output: You are creating a new layer for every image with a different size, which not only will not work, but every new layer will have its set of parameters which it saves in GPU memory, and so you will quickly run out of memory space. This is a usual problem in image processing due to the fixed size of linear layers as opposed to the dynamic H,W dimensions of the convolution layers. The only real way to fix it is to reshape your images to the same size prior to putting them in the network (look at  torch.nn.functional.interpolate or cv2.resize) or to use a convolution with a kernel size of 1 (effectively a linear layer) and do some global average/max pooling to shrink the data to the same size.


Instruction: What do we mean by 'register' in PyTorch?

Input:
I am not asking about registers which are memory locations to store the content.
I am asking about the usage of word 'register' in PyTorch documentation.
While I am reading the documenation regarding MODULE in PyTorch, I encountered the usage of word registers, registered several times.
The context of usage is as follows
1. tensor (Tensor) – buffer to be registered.

2. Submodules assigned in this way will be registered, and will have their parameters converted too when you call to(), etc.

3. Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

4. Registers a backward hook on the module.

5. Registers a forward hook on the module.

.....

And the word 'register' has been used in the names of several methods
1. register_backward_hook(hook)

2. register_buffer(name, tensor, persistent=True)

3. register_forward_hook(hook)

4. register_forward_pre_hook(hook)

5. register_parameter(name, param)

......

What does it mean by the usage of word register programmatically?
Does it just mean the act of recording a name or information on an official list as in plain English or has any significance programmatically?

Output:
This "register" in pytorch doc and methods names means "act of recording a name or information on an official list".
For instance, register_backward_hook(hook) adds the function hook to a list of other functions that nn.Module executes during the execution of the forward pass.
Similarly, register_parameter(name, param) adds an nn.Parameter param with name to the list of trainable parameters of the nn.Module.
It is crucial to register trainable parameters so pytorch will know what tensors to pass to the optimizer and what tensors to store as part of the nn.Module's state_dict.


Instruction: does pytorch support complex numbers?

Input:
Minimum (not) working example

kernel = Conv2d(in_channels=1, out_channels=1, kernel_size=(3, 2))
data = torch.rand(1, 1, 100, 100).type(torch.complex64)
kernel(data)

yields RuntimeError: "unfolded2d_copy" not implemented for 'ComplexDouble' for 64 and 128 bit complex numbers, while for 32 bit, i get RuntimeError: "copy_" not implemented for 'ComplexHalf'.
Am I missing something, or is pytorch missing support for complex numbers??
note: I'm on macbook, using cpu only.

Output:
Currently (@ latest stable version - 1.9.0) Pytorch is missing support for such operations on complex tensors (which are a beta feature). See this feature request at Native implementation of convolution for complex numbers
Splitting into convolution on real & image separately, though not ideal, is the way to go for now.


Instruction: Positional Encoding for time series based data for Transformer DNN models

Input:
In several academic papers, researchers use the following positional encoding to denote the positioning of elements in a sequence, whether it be a time series-based sequence or words in a sentence for NLP purposes.
My question is how the positioning is actually applied to the data before it is fed to the deep neural network (in my case a transformer network):

Are the positional values added directly to the actual values of the elements in the sequence (or to the word representation values)? Or are they concatinated? Is the positional embedding part of the data preprocessing stage?
Does the Tensorflow/Keras MultiHeadAttention layer actually already contain an Embeeding layer that takes care of the positional encoding? Or not?
What about the normalization of data? Are only the actual element values normalized and then the positional encoding is added to that normalized value? Or is the positional encoding value added to the raw value of the element and the resulting values are normalized?

I am interested in actual implementation details not the conceptual part of positional encoding as I read most of the academic papers on positional encoding already. Unfortunately, most academic papers fall short of describing in detail at what stage and how precisely the positional encoding is applied to the data structure.
Thanks!!!

Output:
Positional encoding is just a way to let the model differentiates two elements (words) that're the same but which appear in different positions in a sequence.
After applying embeddings in a LM - language model for example, we add PE to add an information about position of each word.

Are the positional values added directly to the actual values of the elements in the sequence (or to the word representation values)? Or are they concatinated? Is the positional embedding part of the data preprocessing stage?

Yes PE values are just added directly to actual values (embeddings in a LM). This will results that the embedding vector of the word a that appears in the beginning of the sequence will be different of the embedding vector of the same word that appears in the middle of the sequence. And no, PE is not a part of data preprocessing stage.
Here's an example of code:
class PositionalEncodingLayer(nn.Module):

    def __init__(self, d_model, max_len=100):
        super(PositionalEncodingLayer, self).__init__()
        self.d_model = d_model
        self.max_len = max_len

    def get_angles(self, positions, indexes):
        d_model_tensor = torch.FloatTensor([[self.d_model]]).to(positions.device)
        angle_rates = torch.pow(10000, (2 * (indexes // 2)) / d_model_tensor)
        return positions / angle_rates

    def forward(self, input_sequences):
        """
        :param Tensor[batch_size, seq_len] input_sequences
        :return Tensor[batch_size, seq_len, d_model] position_encoding
        """
        positions = torch.arange(input_sequences.size(1)).unsqueeze(1).to(input_sequences.device) # [seq_len, 1]
        indexes = torch.arange(self.d_model).unsqueeze(0).to(input_sequences.device) # [1, d_model]
        angles = self.get_angles(positions, indexes) # [seq_len, d_model]
        angles[:, 0::2] = torch.sin(angles[:, 0::2]) # apply sin to even indices in the tensor; 2i
        angles[:, 1::2] = torch.cos(angles[:, 1::2]) # apply cos to odd indices in the tensor; 2i
        position_encoding = angles.unsqueeze(0).repeat(input_sequences.size(0), 1, 1) # [batch_size, seq_len, d_model]
        return position_encoding

class InputEmbeddingAndPositionalEncodingLayer(nn.Module):

    def __init__(self, vocab_size, max_len, d_model, dropout):
        super(InputEmbeddingAndPositionalEncodingLayer, self).__init__()
        self.vocab_size = vocab_size
        self.max_len = max_len
        self.d_model = d_model
        self.dropout = nn.Dropout(p=dropout)
        self.token_embedding = nn.Embedding(vocab_size, d_model)
        self.position_encoding = PositionalEncodingLayer(d_model=d_model, max_len=max_len)

    def forward(self, sequences):
        """
        :param Tensor[batch_size, seq_len] sequences
        :return Tensor[batch_size, seq_len, d_model]
        """
        token_embedded = self.token_embedding(sequences) # [batch_size, seq_len, d_model]
        position_encoded = self.position_encoding(sequences) # [batch_size, seq_len, d_model]
        return self.dropout(token_embedded) + position_encoded # [batch_size, seq_len, d_model]

Does the Tensorflow/Keras MultiHeadAttention layer actually already contain an Embeeding layer that takes care of the positional encoding? Or not?

Simply No. You have to build PE yourself.

What about the normalization of data? Are only the actual element values normalized and then the positional encoding is added to that normalized value? Or is the positional encoding value added to the raw value of the element and the resulting values are normalized?

The normalization part is at your discretion. You do what you want. But you should apply the normalization. Also, PE is added to normalized values not actual one.


Instruction: CPU only pytorch is crashing with error AssertionError: Torch not compiled with CUDA enabled

Input:
I'm trying to run the code from this repository and I need to use Pytorch 1.4.0. I've installed the CPU only version of pytorch with pip install torch==1.4.0+cpu torchvision==0.5.0+cpu -f https://download.pytorch.org/whl/torch_stable.html.
I ran the program by doing py -m train_Kfold_CV --device 0 --fold_id 10 --np_data_dir "C:\Users\username\OneDrive\Desktop\emadeldeen\AttnSleep\prepare_datasets\edf_20_npz" but I'm getting this error:
  File "C:\Users\username\AppData\Local\Programs\Python\Python37\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Users\username\AppData\Local\Programs\Python\Python37\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\username\OneDrive\Desktop\emadeldeen\AttnSleep\train_Kfold_CV.py", line 94, in <module>
    main(config, fold_id)
  File "C:\Users\username\OneDrive\Desktop\emadeldeen\AttnSleep\train_Kfold_CV.py", line 65, in main
    trainer.train()
  File "C:\Users\username\OneDrive\Desktop\emadeldeen\AttnSleep\base\base_trainer.py", line 66, in train
    result, epoch_outs, epoch_trgs = self._train_epoch(epoch, self.epochs)
  File "C:\Users\username\OneDrive\Desktop\emadeldeen\AttnSleep\trainer\trainer.py", line 49, in _train_epoch
    loss = self.criterion(output, target, self.class_weights)
  File "C:\Users\username\OneDrive\Desktop\emadeldeen\AttnSleep\model\loss.py", line 6, in weighted_CrossEntropyLoss
    cr = nn.CrossEntropyLoss(weight=torch.tensor(classes_weights).cuda())
  File "C:\Users\username\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\cuda\__init__.py", line 196, in _lazy_init
    _check_driver()
  File "C:\Users\username\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\cuda\__init__.py", line 94, in _check_driver
    raise AssertionError("Torch not compiled with CUDA enabled")
AssertionError: Torch not compiled with CUDA enabled

I've changed the number of GPU in the config to 0 and tried adding device = torch.device('cpu') at the begining of the program, but it's not doing anything. How can I fix this error? I'm using windows 10 with python 3.7.9 if it helps
Thanks

Output:
You are using CPU only pytorch, but your code has statement like cr = nn.CrossEntropyLoss(weight=torch.tensor(classes_weights).cuda()) which is trying to move the tensor to GPU.
To fix it,
remove all the .cuda() operations.


Instruction: Normalizing CNN network output to get a distance output between 0 and 1

Input:
I'm using PyTorch based CNNs to do feature extraction on images of humans in order to use it to re-identify that same person given a different picture. After the whole process I am left with a 1D vector, about 2048x1 which I then compare using L2 distance as a metric. I am currently trying to normalize that L2 distance output so I can represent the model's prediction as a confidence from 0-1.
I noticed that PyTorch recommends using the where images are loaded in as loaded in to a range of [0, 1] and then normalized using mean = [0.485, 0.456, 0.406] and std = [0.229, 0.224, 0.225]. When it does this, it seems to alter the range of the L2 distance output which changes the range of values for the L2 distance which makes it no longer 0-1. I am wondering is there a way to normalize the output vector back to the 0-1 range or even the L2 distance itself so I can represent it from 0-1
EDIT:
The following is the code that I am using to calculate the L2 distance
distmat = torch.pow(qf, 2).sum(dim=1, keepdim=True).expand(m, n) + \
              torch.pow(gf, 2).sum(dim=1, keepdim=True).expand(n, m).t()
distmat.addmm_(qf, gf.t(), beta=1, alpha=-2, )
distmat = distmat.cpu().numpy()

As was suggested by a commenter, I am attempting to change the L2 distance calculation to instead use cosine distance. This is my current code:
distmat = torch.mm(qf, gf.t())

However when I ran this, I was getting outputs that looked like this:
tensor([[ 0.3244,  0.2478,  0.1808, -0.0249,  0.2137,  0.2113]])
Wondering if this is the right way to do the cosine distance calculation?
EDIT 2:
Here's how cosine similarity's final implementation looked like for me:
qf_norm = qf / qf.norm(dim=1)[:, None]
gf_norm = gf / gf.norm(dim=1)[:, None]
distmat = torch.mm(qf_norm, gf_norm.transpose(0, 1)).cpu().numpy()

Output:
Using normalization vs not using normalization
Are you using a pretrained network?
If yes, and the pretrained model was trained with normalized input (mean/std transformation) then you should use those operations. If you use without those operations, the embeddings will not be useful or less useful. Simply, put you should input normalized images, because model was trained on normalized input.
Normalized distances
L2 distance is unbounded. Even if you compute the range over a sample of typical input, it is not guaranteed that a new input image would fall in that range. For this reason, people use cosine similarity which is akin to a normalized dot product and thus will always be between 0 and 1 -1 and 1.


Instruction: PyTorch 1.9 command equivalent for `zero_gradients`

Input:
This error comes after the upgrade of my PyTorch version from 1.8 to 1.9.0.
When using this line:
from torch.autograd.gradcheck import zero_gradients, I get this error message: ImportError: cannot import name 'zero_gradients' from 'torch.autograd.gradcheck'
The command:
zero_gradients(im)
is used.
What is the new command equivalent in PyTorch 1.9.0?

Output:
There is no function in pytorch named zero_gradients(). The nearest similar name is zero_grad(). It's instead a function defined in the repo you shared in auto-attack/autoattack/other_utils.py .
def zero_gradients(x):
    if isinstance(x, torch.Tensor):
        if x.grad is not None:
            x.grad.detach_()
            x.grad.zero_()
    elif isinstance(x, container_abcs.Iterable):
        for elem in x:
            zero_gradients(elem)


Instruction: What is the correct way to calculate the norm, 1-norm, and 2-norm of vectors in PyTorch?

Input:
I have a matrix:
t = torch.rand(2,3)
print(t)
>>>tensor([[0.5164, 0.3651, 0.0882],
        [0.4488, 0.9824, 0.4067]])

I'm following this introduction to norms and want to try it in PyTorch.
It seems like the:

norm of a vector is "the size or length of a vector is a nonnegative number that describes the extent of the vector in space, and is sometimes referred to as the vector’s magnitude or the norm"
1-Norm is "the sum of the absolute vector values, where the absolute value of a scalar uses the notation |a1|. In effect, the norm is a calculation of the Manhattan distance from the origin of the vector space."
2-Norm is "the distance of the vector coordinate from the origin of the vector space. The L2 norm is calculated as the square root of the sum of the squared vector values."

I currently only know of this:
print(torch.linalg.norm(t, dim=1))
>>>tensor([0.6385, 1.1541])

But I can't figure out which one of the three (norm, 1-norm, 2-norm) from here it is calculating, and how to calculate the rest

Output:
To compute the 0-, 1-, and 2-norm you can either use torch.linalg.norm, providing the ord argument (0, 1, and 2 respectively). Or directly on the tensor: Tensor.norm, with the p argument. Here are the three variants: manually computed, with torch.linalg.norm, and with Tensor.norm.

0-norm
>>> x.norm(dim=1, p=0)
>>> torch.linalg.norm(x, dim=1, ord=0)
>>> x.ne(0).sum(dim=1)

1-norm
>>> x.norm(dim=1, p=1)
>>> torch.linalg.norm(x, dim=1, ord=1)
>>> x.abs().sum(dim=1)

2-norm
>>> x.norm(dim=1, p=2)
>>> torch.linalg.norm(x, dim=1, ord=2)
>>> x.pow(2).sum(dim=1).sqrt()


Instruction: How to create a DataSet of 1000 graphs in python

Input:
I need to create a dataset of 1000 graphs. I used the following code:
data_list = []
ngraphs = 1000
for i in range(ngraphs):
      num_nodes = randint(10,500)
      num_edges = randint(10,num_nodes*(num_nodes - 1))
      f1 = np.random.randint(10, size=(num_nodes))
      f2 = np.random.randint(10,20, size=(num_nodes))
      f3 = np.random.randint(20,30, size=(num_nodes))
      f_final = np.stack((f1,f2,f3), axis=1)
      capital = 2*f1 + f2 - f3
      f1_t = torch.from_numpy(f1)
      f2_t = torch.from_numpy(f2)
      f3_t =  torch.from_numpy(f3)
      capital_t =  torch.from_numpy(capital)
      capital_t = capital_t.type(torch.LongTensor)
      x = torch.from_numpy(f_final)
      x = x.type(torch.LongTensor)
      edge_index = torch.randint(low=0, high=num_nodes, size=(num_edges,2), dtype=torch.long)
      edge_attr  = torch.randint(low=0, high=50, size=(num_edges,1), dtype=torch.long)
      data = Data(x = x, edge_index = edge_index.t().contiguous(), y = capital_t, edge_attr=edge_attr )
      data_list.append(data)

This works. But when I run my training function as follows:
for epoch in range(1, 500):
    loss = train()
    print(f'Loss: {loss:.4f}')

I keep getting the following error:

RuntimeError                              Traceback (most recent call
last)  in ()
1 for epoch in range(1, 500):
----> 2     loss = train()
3     print(f'Loss: {loss:.4f}')
5 frames /usr/local/lib/python3.7/dist-packages/torch/nn/functional.py
in linear(input, weight, bias)    1845     if
has_torch_function_variadic(input, weight):    1846         return
handle_torch_function(linear, (input, weight), input, weight,
bias=bias)
-> 1847     return torch._C._nn.linear(input, weight, bias)    1848     1849
RuntimeError: expected scalar type Float but found Long

Can someone help me troubleshoot this. Or make a 1000 graph dataset that doesn't throw this error.

Output: Change your x and y tensor into FloatTensor, since Linear layer in python only accept FloatTensor inputs


Instruction: Train a torch model using one GPU and the shared memory

Input:
I am new to training pytorch models
and on GPU
I have tried training it on windows, but was always use the dedicated memory (10GB) and does not utilise the shared memory
I have tried  enhancing its performance using multiprocessing, but I kept getting the error :
TypeError: cannot pickle 'module' object
The solution usually is to use num_wrokers =0 while loading the data
I actually use the multiprocessing after loading the data
And need only to utilise the shared memory
I am retraing meta-sr speaker verification code, specifically the training file:
https://github.com/seongmin-kye/meta-SR/blob/b4c1ea1728e33f7bbf7015c38f508f24594f3f88/train.py
I have edited the line 92 to use the shared GPU memory as the following
instead of:
train(train_generator, model, objective, optimizer, n_episode, log_dir, scheduler)
To:
    model.share_memory()
    p = mp.Process(target=train, args=(train_generator,model,objective, optimizer, n_episode, log_dir, scheduler))
    p.num_workers=0
    p.start()
    p.join()

Please let me know if more information shall be added
Thanks in advance

Output:
The shared memory (part of RAM) can be used only if there are two GPUs in PC
and this case there was only one GPU


Instruction: Trouble with minimal hvp on pytorch model

Input:
While autograd's hvp tool seems to work very well for functions, once a model becomes involved, Hessian-vector products seem to go to 0.  Some code.
First, I define the world's simplest model:
class SimpleMLP(nn.Module):
  def __init__(self, in_dim, out_dim):
      super().__init__()
      self.layers = nn.Sequential(
        nn.Linear(in_dim, out_dim),
      )

  def forward(self, x):
    '''Forward pass'''
    return self.layers(x)

Then, a loss function:
def objective(x):
  return torch.sum(0.25 * torch.sum(x)**4)

We instantiate it:
Arows = 2
Acols = 2

mlp = SimpleMLP(Arows, Acols)

Finally, I'm going to define a "forward" function (distinct from the model's forward function) that will serve as the the full model+loss that we want to analyze:
def forward(*params_list):
  for param_val, model_param in zip(params_list, mlp.parameters()):
    model_param.data = param_val

  x = torch.ones((Arows,))
  return objective(mlp(x))

This passes a ones vector into the single-layer "mlp," and passes it into our quadratic loss.
Now, I attempt to compute:
v = torch.ones((6,))
v_tensors = []
idx = 0
#this code "reshapes" the v vector as needed
for i, param in enumerate(mlp.parameters()):
  numel = param.numel()
  v_tensors.append(torch.reshape(torch.tensor(v[idx:idx+numel]), param.shape))
  idx += numel

And finally:
param_tensors = tuple(mlp.parameters())
reshaped_v = tuple(v_tensors)
soln =  torch.autograd.functional.hvp(forward, param_tensors, v=reshaped_v)

But, alas, the Hessian-Vector Product in soln is all 0's.  What is happening?

Output:
What's happening is that  strict is False by default in the hvp() function and a tensor of 0's is returned as the Hessian Vector Product instead of an error (source).
If you try with strict=True, an error RuntimeError: The output of the user-provided function is independent of input 0. This is not allowed in strict mode. is returned instead. And when I looked at the full error, I suspect that this error comes from _check_requires_grad(jac, "jacobian", strict=strict) which indicates that  the jacobian jac is None.
Update:
Following is a full working example:
import torch
from torch import nn

# your loss function
def objective(x):
    return torch.sum(0.25 * torch.sum(x)**4)

# Following are utilities to make nn.Module functional
# borrowed from the link I posted in comment
def del_attr(obj, names):
    if len(names) == 1:
        delattr(obj, names[0])
    else:
        del_attr(getattr(obj, names[0]), names[1:])

def set_attr(obj, names, val):
    if len(names) == 1:
        setattr(obj, names[0], val)
    else:
        set_attr(getattr(obj, names[0]), names[1:], val)

def make_functional(mod):
    orig_params = tuple(mod.parameters())
    # Remove all the parameters in the model
    names = []
    for name, p in list(mod.named_parameters()):
        del_attr(mod, name.split("."))
        names.append(name)
    return orig_params, names

def load_weights(mod, names, params):
    for name, p in zip(names, params):
        set_attr(mod, name.split("."), p)

# your forward function with update
def forward(*new_params):
    # this line replace your for loop
    load_weights(mlp, names, new_params)

    x = torch.ones((Arows,))
    out = mlp(x)
    loss = objective(out)
    return loss

# your simple MLP model
class SimpleMLP(nn.Module):
    def __init__(self, in_dim, out_dim):
        super().__init__()
        self.layers = nn.Sequential(
            nn.Linear(in_dim, out_dim),
        )

    def forward(self, x):
        '''Forward pass'''
        return self.layers(x)

if __name__ == '__main__':
    # your model instantiation
    Arows = 2
    Acols = 2
    mlp = SimpleMLP(Arows, Acols)

    # your vector computation
    v = torch.ones((6,))
    v_tensors = []
    idx = 0
    #this code "reshapes" the v vector as needed
    for i, param in enumerate(mlp.parameters()):
        numel = param.numel()
        v_tensors.append(torch.reshape(torch.tensor(v[idx:idx+numel]), param.shape))
        idx += numel
    reshaped_v = tuple(v_tensors)

    #make model's parameters functional
    params, names = make_functional(mlp)
    params = tuple(p.detach().requires_grad_() for p in params)

    #compute hvp
    soln = torch.autograd.functional.vhp(forward, params, reshaped_v, strict=True)
    print(soln)


Instruction: Direction of rotation of torch.rot90

Input:
In the documentation for torch.rot90 it is stated that

Rotation direction is from the first towards the second axis if k > 0, and from the second towards the first for k < 0.

However say that we are rotating from axis 0 to axis 1, is axis 0 rotating to axis 1 in the clockwise or anti-clockwise direction? (since they are both 90 degree rotations as per the image below)

Output:
axis=0 is the dimension that points downwards, while axis=1 points to the right. Visualize the axes like this:
---------> axis=1
|
|
|
\/
axis=0

Now, k>0 means counter-clockwise direction, k<0 is clockwise.
Thus,
>>> x = torch.arange(6).view(3, 2)
>>> x
tensor([[0, 1],
        [2, 3],
        [4, 5]])

>>> torch.rot90(x, 1, [0,1])
tensor([[1, 3, 5],
        [0, 2, 4]])

>>> torch.rot90(x, 1, [1,0])
tensor([[4, 2, 0],
        [5, 3, 1]])

The torch.rot90()  is similar to numpy.rot90()
e.g.
numpy.rot90(m, k=1, axes=(0, 1))
mean


Instruction: Randomly get index of one of the maximum values in a PyTorch tensor

Input:
I need to perform something similar to the built-in torch.argmax() function on a one-dimensional tensor, but instead of picking the index of the first of the maximum values, I want to be able to pick a random index of one of the maximum values. For example:
my_tensor = torch.tensor([0.1, 0.2, 0.2, 0.1, 0.1, 0.2, 0.1])
index_1 = random_max_val_index_fn(my_tensor)
index_2 = random_max_val_index_fn(my_tensor)

print(f"{index_1}, {index_2}")
> 5, 1

Output:
You can get the indexes of all the maximums first and then choose randomly from them:
def rand_argmax(tens):
    max_inds, = torch.where(tens == tens.max())
    return np.random.choice(max_inds)

sample runs:
>>> my_tensor = torch.tensor([0.1, 0.2, 0.2, 0.1, 0.1, 0.2, 0.1])
>>> rand_argmax(my_tensor)
2
>>> rand_argmax(my_tensor)
5
>>> rand_argmax(my_tensor)
2
>>> rand_argmax(my_tensor)
1


Instruction: PyTorch crashes when training: probable image decoding error, tensor value, corrupt image. (Runtime Error)

Input:
Premise
I am fairly new to using PyTorch, and more times than not I am getting a segfault when training my neural network with a small custom dataset (10 images of 90 classifications).
The output below is from these print statements that ran twice (with MNIST dataset at idx 0 and my custom dataset at idx 0). Both datasets were compiled using a csv file formatted the exact same (img_name, class) and with the image directory MNIST subet is of size 30, and my custom dataset is of size 10:
example, label = dataset[0]
print(dataset[0])
print(example.shape)
print(label)

The first tensor is an MNIST 28X28 png converted to a tensor using:
image = torchvision.io.read_image().type(torch.FloatTensor)

This was so I had a working dataset to compare to. It uses the same custom dataset class as the custom data I have.
The Neural Net class is the exact same as my custom data NN except it has 10 outputs as opposed to the 90 from my custom data.
The custom data is of varied sizes, that have all been resized to 28 X 28 using the transforms.Compose() listed below. In this 10 image subset of the data, there are images that are dimensions 800X170, 96X66, 64X34, 208X66, etc...
The second tensor output is from a png that was of size 800 X 170.
The transforms performed on both datasets are the exact same:
tf=transforms.Compose([
        transforms.Resize(size = (28,28)),
        transforms.Normalize(mean=[-0.5/0.5],std=[1/0.5])
        ])

There is no target transforms performed.
Output of tensor, tensor size, class, and train/test performed at end
(tensor([[[  0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000],
         [  0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000],
         [  0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000],
         [  0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000],
         [  0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,  19.5000,
          119.0000,  54.0000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000],
         [  0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,  32.5000,
          127.0000,  93.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000],
         [  0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,  32.5000,
          127.0000, 106.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000],
         [  0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,  32.5000,
          127.0000, 106.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000],
         [  0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,  32.5000,
          127.0000, 106.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000],
         [  0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,  85.5000,
          127.5000, 107.0000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000],
         [  0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,  63.5000,
          127.0000, 106.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000],
         [  0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,  59.0000,
          127.0000,  58.0000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000],
         [  0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,  32.5000,
          127.0000,  66.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000],
         [  0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,  32.5000,
          127.0000, 106.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000],
         [  0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,  33.0000,
          128.0000, 107.0000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000],
         [  0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,  32.5000,
          127.0000,  88.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000],
         [  0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,  59.5000,
          127.0000,  54.0000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000],
         [  0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,  85.0000,
          127.0000,  54.0000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000],
         [  0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,  85.0000,
          127.0000,  54.0000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000],
         [  0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,  85.5000,
          128.0000,  54.0000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000],
         [  0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,  85.0000,
          127.0000,  54.0000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000],
         [  0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,  85.0000,
          127.0000,  60.0000,   8.0000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000],
         [  0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,  85.0000,
          127.0000, 127.5000,  84.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000],
         [  0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,  28.0000,
          118.5000,  65.5000,  14.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000],
         [  0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000],
         [  0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000],
         [  0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000],
         [  0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,
            0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000,   0.5000]]]), 1)
torch.Size([1, 28, 28])
1

Train Epoch: 1 [0/25 (0%)]  Loss: -1.234500

Test set: Average loss: -1.6776, Accuracy: 1/5 (20%)

(tensor([[[68.1301, 67.3571, 68.4286, 67.9375, 69.5536, 69.2143, 69.0026,
          69.2283, 70.4464, 70.2857, 68.8839, 68.6071, 71.3214, 70.5102,
          71.0753, 71.9107, 71.5179, 71.5625, 73.6071, 71.9464, 73.2513,
          72.5804, 73.5000, 74.1429, 72.7768, 72.9107, 73.1786, 74.9069],
         [68.2028, 70.0714, 68.4821, 69.3661, 70.8750, 69.6607, 70.6569,
          70.2551, 70.9464, 70.3393, 70.3929, 71.3571, 71.1250, 72.1901,
          70.6850, 71.9464, 72.1071, 72.8304, 72.3036, 72.3214, 73.4528,
          73.4898, 72.4286, 73.0179, 73.1071, 73.5179, 73.0357, 74.0280],
         [71.3457, 70.4643, 70.4464, 70.7857, 70.6071, 71.9821, 71.6786,
          72.7564, 72.4107, 72.2321, 72.8571, 72.7321, 70.0357, 72.2640,
          73.8214, 72.8750, 73.0000, 73.0089, 74.8393, 74.1964, 74.9872,
          73.4248, 72.0179, 74.5357, 74.9018, 74.9821, 75.0357, 72.9286],
         [70.1429, 70.3750, 69.8750, 70.6250, 69.8750, 72.8750, 71.4107,
          71.5089, 73.3750, 73.2500, 74.4375, 73.8750, 73.0000, 74.4375,
          72.2768, 72.7500, 72.6250, 72.6250, 73.1250, 73.2500, 72.3571,
          73.0625, 72.5000, 74.8750, 73.6875, 74.2500, 75.2500, 73.7411],
         [53.1428, 56.1607, 57.4286, 58.3393, 60.6607, 59.3393, 62.2589,
          62.8380, 64.1250, 66.6429, 66.9821, 67.8750, 74.7679, 70.5192,
          68.7411, 69.3036, 66.0001, 67.9733, 67.4822, 68.3393, 68.3534,
          69.5740, 69.4465, 70.9465, 69.0983, 72.2679, 70.4286, 70.1493],
         [61.2143, 63.0000, 69.0357, 65.3393, 62.3214, 59.8036, 56.2730,
          54.5829, 52.8393, 52.8929, 50.8304, 52.9107, 66.4643, 69.6875,
          71.1849, 72.2678, 73.9821, 74.4643, 73.0357, 74.1250, 75.6492,
          76.2360, 75.7679, 75.6071, 75.2857, 74.9286, 74.8929, 75.1850],
         [54.9439, 62.5357, 69.7143, 72.0000, 71.2500, 74.1607, 75.9987,
          79.6416, 79.5179, 81.4822, 77.3214, 75.2143, 49.6071, 59.7513,
          71.4350, 74.4822, 73.5000, 73.8214, 72.2322, 73.7143, 73.9822,
          74.5893, 74.7322, 74.8572, 76.2947, 71.5714, 73.4822, 74.8533],
         [63.4298, 61.0357, 61.6072, 59.6697, 57.8036, 59.2322, 56.5982,
          57.2079, 55.3393, 56.3572, 56.5804, 58.7322, 79.7499, 73.1900,
          65.2423, 75.5357, 74.5356, 75.6250, 72.5893, 74.7321, 74.6135,
          75.8852, 75.6964, 75.7678, 76.4286, 74.2500, 74.7857, 76.1671],
         [63.7870, 60.3750, 67.5179, 67.5446, 66.7857, 66.2857, 66.4515,
          68.5089, 68.5714, 67.0714, 68.5982, 66.7678, 57.3929, 67.2806,
          68.9503, 72.9286, 74.0893, 73.4911, 74.2143, 73.3393, 72.4873,
          73.3916, 71.7500, 75.4821, 73.8393, 74.8750, 74.6429, 75.0906],
         [72.9260, 69.0178, 67.9643, 69.2321, 67.5178, 67.3750, 66.3814,
          64.8890, 63.8572, 64.9464, 66.9821, 66.3928, 63.0000, 64.7449,
          74.8800, 63.5178, 72.2143, 73.2321, 74.9286, 74.5893, 71.6938,
          74.8635, 73.9107, 75.5536, 75.8036, 76.2857, 76.3750, 75.2564],
         [72.1160, 69.5000, 72.0000, 69.4375, 71.2500, 70.5000, 72.3392,
          73.5982, 71.5000, 72.3750, 68.8750, 67.1249, 65.3750, 60.2856,
          61.6427, 65.3749, 67.4999, 65.0624, 70.4999, 69.4999, 65.3124,
          71.9107, 69.7499, 72.8750, 72.5625, 72.7500, 74.8750, 73.7053],
         [64.3763, 64.8571, 70.4642, 66.7857, 64.3214, 65.3928, 67.4859,
          68.7385, 67.8750, 67.8750, 71.0267, 72.8749, 67.5356, 59.4106,
          58.7625, 70.2319, 62.5534, 65.7141, 68.1249, 69.0713, 65.2013,
          72.8392, 67.1427, 71.7500, 72.8482, 72.6071, 74.4285, 74.0051],
         [69.7219, 71.8214, 67.4464, 68.6518, 66.0178, 66.1071, 65.5089,
          65.6964, 65.6964, 61.0714, 61.4375, 61.8214, 67.8214, 61.8762,
          57.3354, 66.8749, 63.8571, 60.3302, 62.9999, 67.8214, 68.9043,
          71.6365, 67.5357, 75.6250, 74.6518, 73.6071, 74.5178, 75.3877],
         [72.2857, 66.2857, 63.1964, 69.2232, 68.8214, 70.2857, 68.7895,
          70.2436, 70.1250, 66.8750, 69.9643, 66.0893, 52.8393, 60.3201,
          52.9273, 66.8571, 58.0535, 57.3035, 63.2321, 60.1785, 59.6058,
          69.9936, 69.4286, 73.4821, 72.7143, 72.8750, 72.7500, 74.0791],
         [65.7334, 56.6430, 60.7143, 67.8035, 66.5178, 65.8214, 67.6760,
          67.3061, 65.6964, 64.5893, 53.1430, 68.4820, 52.7676, 48.1604,
          48.1311, 65.3034, 51.9640, 61.8213, 59.6605, 57.3927, 54.6974,
          75.5752, 73.1250, 74.3928, 74.0446, 72.2142, 72.2857, 77.7806],
         [55.4095, 60.0893, 69.7142, 66.0892, 66.8750, 65.6607, 67.1926,
          66.3712, 63.0000, 56.9465, 41.6073, 48.6609, 61.8035, 39.7281,
          44.9195, 61.5892, 47.5891, 62.7678, 56.9641, 55.9820, 58.1236,
          70.0548, 70.3750, 69.8392, 68.1517, 72.0535, 76.5893, 65.4489],
         [60.6237, 66.5714, 67.8571, 65.7232, 66.2500, 67.6250, 66.9311,
          67.3303, 64.8214, 48.9644, 45.9019, 49.4108, 51.6608, 43.9259,
          47.5012, 38.9642, 37.5356, 66.0000, 65.5178, 49.3392, 57.3571,
          67.8252, 69.7678, 70.2143, 51.7410, 76.1607, 69.7143, 54.4056],
         [61.9643, 67.2500, 66.5000, 65.6875, 66.2500, 65.0000, 65.0625,
          65.5268, 63.7500, 49.8750, 50.4375, 53.1250, 38.7500, 25.3750,
          43.4286, 31.1250, 35.3750, 59.7500, 63.3750, 39.5000, 51.8125,
          58.6249, 69.5000, 70.1250, 48.0000, 75.8750, 48.7500, 61.4018],
         [67.8915, 65.7500, 66.3035, 66.5982, 66.0357, 64.9464, 65.4643,
          65.8074, 63.4643, 56.2325, 48.3306, 54.9467, 22.0715, 23.6990,
          29.0955, 27.3211, 29.4997, 57.8660, 68.2321, 36.9819, 50.7715,
          52.6707, 69.7143, 71.3392, 55.5534, 45.7855, 62.9463, 64.1556],
         [63.8431, 66.0893, 65.3571, 65.6161, 65.0893, 64.6964, 64.3444,
          65.1225, 62.9107, 57.4287, 57.3216, 54.9287, 26.4465, 30.5689,
          23.2499, 23.5534, 25.1605, 55.1071, 69.4643, 41.9642, 52.6619,
          59.8954, 72.0893, 79.7322, 47.2856, 64.5000, 52.9463, 81.6888],
         [64.2589, 69.9643, 71.5000, 75.2857, 77.6786, 78.6429, 76.2513,
          71.0089, 67.5536, 60.8929, 57.2501, 48.1072, 22.4821, 44.3316,
          17.5369, 24.3928, 22.8214, 45.4821, 67.8036, 35.4821, 43.7028,
          52.7806, 81.8929, 56.7321, 60.5357, 44.2321, 82.6964, 72.7500],
         [63.6748, 61.8929, 58.0001, 41.7859, 47.3037, 35.2502, 40.0525,
          63.9669, 76.1962, 74.6603, 67.2228, 43.3748, 19.9821, 37.0776,
          15.6544, 30.9823, 22.0182, 51.0984, 65.8215, 32.5717, 49.4747,
          39.5946, 49.5359, 55.7859, 40.7681, 81.7857, 76.0357, 73.2832],
         [60.0192, 53.6429, 43.5359, 44.8037, 39.9287, 48.8037, 48.3241,
          35.5882, 22.6071, 20.7142, 33.8838, 45.3570, 25.0714, 32.6657,
          26.8559, 22.9644, 27.7324, 69.4375, 62.5001, 33.9823, 48.6047,
          33.4811, 38.3930, 58.5358, 74.2857, 73.2679, 68.8572, 71.0817],
         [63.2500, 63.3393, 43.1608, 50.3751, 68.6786, 69.6429, 63.9324,
          65.5510, 59.6249, 54.3035, 40.5267, 20.6071, 32.1785, 31.9834,
          30.0791, 20.3036, 34.1073, 71.0000, 56.2322, 48.2501, 42.9695,
          37.1225, 53.7322, 68.3750, 76.2232, 72.4822, 70.6072, 72.9324],
         [63.1071, 64.1250, 65.7500, 41.7500, 26.2500, 25.6250, 25.1071,
          24.1339, 18.8750, 23.5000, 35.5625, 44.5000, 31.1250, 37.3393,
          28.3125, 23.6250, 39.3750, 67.1875, 60.7500, 53.2500, 41.6250,
          39.1339, 61.2500, 81.0000, 71.3125, 70.8750, 71.5000, 72.1339],
         [67.4796, 68.1429, 68.9821, 76.4286, 75.0893, 74.6250, 73.8419,
          72.7398, 58.4108, 44.3572, 33.2322, 19.8036, 32.6965, 29.7296,
          28.5957, 19.8750, 42.7499, 69.9196, 66.3214, 51.9285, 43.6848,
          44.9017, 64.2857, 73.2857, 71.7321, 71.4286, 73.9286, 73.5893],
         [67.7080, 67.9465, 68.0358, 69.1786, 69.1071, 69.7857, 69.0650,
          70.3635, 60.1247, 52.3744, 52.1690, 44.3031, 30.2678, 29.7014,
          20.1314, 25.4645, 45.8042, 74.2947, 63.4110, 56.0183, 49.2722,
          50.1485, 73.1251, 74.6608, 74.3036, 73.8572, 72.2322, 74.1570],
         [67.5868, 68.5179, 68.1786, 66.9018, 67.3215, 67.9822, 67.2628,
          65.4694, 49.2318, 43.7318, 39.5888, 47.7318, 29.2499, 28.3277,
          15.6326, 30.8215, 34.2502, 64.6428, 63.3572, 63.0001, 50.1688,
          51.6037, 77.5000, 75.8215, 73.7501, 74.9286, 74.3572, 74.6097]]]), 20)
torch.Size([1, 28, 28])
20
Train Epoch: 1 [0/8 (0%)]   Loss: -1.982941

Test set: Average loss: 0.0000, Accuracy: 0/2 (0%)

Error information
This output is when it ran successfully with no segfault, the segfault usually occurs 4 times out of 5. When a segfault does occur, it never occurs processing the MNIST subset, it only occurs while attempting to access the dataset either at dataset[0] or whichever 1, or literally any of them, but if I run the simple print statements enough times on any of the indices I can get it to output at least once and not crash.
Here is an occasion when it crashed more gracefully (outputted the tensor info and size/class, but crashed upon train:
torch.Size([1, 28, 28])
65
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py in _try_get_data(self, timeout)
    989         try:
--> 990             data = self._data_queue.get(timeout=timeout)
    991             return (True, data)

9 frames
/usr/lib/python3.7/queue.py in get(self, block, timeout)
    178                         raise Empty
--> 179                     self.not_empty.wait(remaining)
    180             item = self._get()

/usr/lib/python3.7/threading.py in wait(self, timeout)
    299                 if timeout > 0:
--> 300                     gotit = waiter.acquire(True, timeout)
    301                 else:

/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/signal_handling.py in handler(signum, frame)
     65         # Python can still get and update the process status successfully.
---> 66         _error_if_any_worker_fails()
     67         if previous_handler is not None:

RuntimeError: DataLoader worker (pid 1132) is killed by signal: Segmentation fault.

The above exception was the direct cause of the following exception:

RuntimeError                              Traceback (most recent call last)
<ipython-input-9-02c9a53ca811> in <module>()
     68
     69 if __name__ == '__main__':
---> 70     main()

<ipython-input-9-02c9a53ca811> in main()
     60
     61     for epoch in range(1, args.epochs + 1):
---> 62         train(args, model, device, train_loader, optimizerAdadelta, epoch)
     63         test(model, device, test_loader)
     64         scheduler.step()

<ipython-input-6-93be0b7e297c> in train(args, model, device, train_loader, optimizer, epoch)
      2 def train(args, model, device, train_loader, optimizer, epoch):
      3     model.train()
----> 4     for batch_idx, (data, target) in enumerate(train_loader):
      5         data, target = data.to(device), target.to(device)
      6         optimizer.zero_grad()

/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py in __next__(self)
    519             if self._sampler_iter is None:
    520                 self._reset()
--> 521             data = self._next_data()
    522             self._num_yielded += 1
    523             if self._dataset_kind == _DatasetKind.Iterable and \

/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py in _next_data(self)
   1184
   1185             assert not self._shutdown and self._tasks_outstanding > 0
-> 1186             idx, data = self._get_data()
   1187             self._tasks_outstanding -= 1
   1188             if self._dataset_kind == _DatasetKind.Iterable:

/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py in _get_data(self)
   1140         elif self._pin_memory:
   1141             while self._pin_memory_thread.is_alive():
-> 1142                 success, data = self._try_get_data()
   1143                 if success:
   1144                     return data

/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py in _try_get_data(self, timeout)
   1001             if len(failed_workers) > 0:
   1002                 pids_str = ', '.join(str(w.pid) for w in failed_workers)
-> 1003                 raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) from e
   1004             if isinstance(e, queue.Empty):
   1005                 return (False, None)

RuntimeError: DataLoader worker (pid(s) 1132) exited unexpectedly

Generally speaking however this issue appears to 'crash for an unknown reasons' and here is what my logs look like when that occurs:
logs
What I think is going on/what I have tried
I think that there is something wrong with the tensor information and how the image is being read. I am only working with maximum 40 images at a single time so there is no reason the disk resources or RAM on Google Colab are failing. I might be normalizing the data improperly, I have tried different values but nothing has fixed it yet. Perhaps the images are corrupt?
I don't really have a solid grasp of what could be going on, otherwise, I would have already solved it. I think I provided ample resources for it to be a glaring issue for someone of expertise in the area. I put a lot of time into this post, I hope someone is able to help me get to the bottom of the problem.
If there are any other obvious issues with my code and my use of the network and custom dataset please let me know, as this is my first time working with PyTorch.
Thank you!
Additional information that I am not sure if it is relevant:
Custom dataset class:
# ------------ Custom Dataset Class ------------
class PhytoplanktonImageDataset(Dataset):
  def __init__(self, annotations_file, img_dir, transform, target_transform):
    self.img_labels = pd.read_csv(annotations_file) # Image name and label file loaded into img_labels
    self.img_dir = img_dir # directory to find all image names
    self.transform = transform # tranforms to apply to images
    self.target_transform = target_transform

  def __len__(self):
    return len(self.img_labels) # get length of csv file

  def __getitem__(self, idx):
    img_path = os.path.join(self.img_dir, self.img_labels.iloc[idx, 0])
    image = torchvision.io.read_image(path=img_path)
    image = image.type(torch.FloatTensor)
    label = self.img_labels.iloc[idx,1]
    if self.transform:
      image = self.transform(image)
    if self.target_transform:
      label = self.target_transform(label)
    return image, label

NN class (only thing changed is nn.Linear() has 10 outputs for MNIST:
class NeuralNetwork(nn.Module):
    def __init__(self):
        super(NeuralNetwork, self).__init__()
        self.flatten = nn.Flatten()
        self.linear_relu_stack = nn.Sequential(
            nn.Linear(28*28, 512),
            nn.ReLU(),
            nn.Linear(512, 512),
            nn.ReLU(),
            nn.Linear(512, 90),
            nn.ReLU()
        )
    def forward(self, x):
        x = self.flatten(x)
        logits = self.linear_relu_stack(x)
        return logits

Args used:
args = parser.parse_args(['--batch-size', '64', '--test-batch-size', '64',
                            '--epochs', '1', '--lr', '0.01', '--gamma', '0.7', '--seed','4',
                            '--log-interval', '10'])

Edit: I was able to get the following exit gracefully on one of the runs (this traceback was a ways into the getitem call):
<ipython-input-3-ae5ff8635158> in __getitem__(self, idx)
     13     img_path = os.path.join(self.img_dir, self.img_labels.iloc[idx, 0]) # image path
     14     print(img_path)
---> 15     image = torchvision.io.read_image(path=img_path) # Reading image to 1 dimensional GRAY Tensor uint between 0-255
     16     image = image.type(torch.FloatTensor) # Now a FloatTensor (not a ByteTensor)
     17     label = self.img_labels.iloc[idx,1] # getting label from csv

/usr/local/lib/python3.7/dist-packages/torchvision/io/image.py in read_image(path, mode)
    258     """
    259     data = read_file(path)
--> 260     return decode_image(data, mode)

/usr/local/lib/python3.7/dist-packages/torchvision/io/image.py in decode_image(input, mode)
    237         output (Tensor[image_channels, image_height, image_width])
    238     """
--> 239     output = torch.ops.image.decode_image(input, mode.value)
    240     return output
    241

RuntimeError: Internal error.

Here is the image path being printed just before the decoding fails: /content/gdrive/My Drive/Colab Notebooks/all_images/sample_10/D20190926T145532_IFCB122_00013.png
and here is what that image looks like:
image
Information about this image:
Color Model: Gray
Depth: 16
Pixel Height: 50
Pixel Width: 80
Image DPI: 72 pixels per inch
file size: 3,557 bytes

Output:
I suggest to take a look at you num workers param in your dataloader.
If you have a num_workers param that is too high it may be causing this error.   Therefore, I suggest to lower it to zero or until you don't get this error.
Sarthak