Instruction: What is mean by (AttributeError: 'NoneType' object has no attribute '__array_interface__') error?

Input:
I am trying to build a ML model to detect landmarks on a cartoon image face. When I split the image dataset in to training and validation sets I got the following error. Here I am using pytorch to build the model. So what is mean by this error?
This is how I split the dataset.
# split the dataset into validation and test sets
len_valid_set = int(0.2*len(dataset))
len_train_set = len(dataset) - len_valid_set

print("The length of Train set is {}".format(len_train_set))
print("The length of Valid set is {}".format(len_valid_set))

train_dataset , valid_dataset,  = torch.utils.data.random_split(dataset , [len_train_set, len_valid_set])

# shuffle and batch the datasets
train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=32, shuffle=True, num_workers=4)
valid_loader = torch.utils.data.DataLoader(valid_dataset, batch_size=8, shuffle=True, num_workers=4)

images, landmarks = next(iter(train_loader))

This is the error I got.
The length of Train set is 105
The length of Valid set is 26

AttributeError                            Traceback (most recent call last)
<ipython-input-61-ffb86a628e37> in <module>()
----> 1 images, landmarks = next(iter(train_loader))
      2
      3 print(images.shape)
      4 print(landmarks.shape)

3 frames
/usr/local/lib/python3.6/dist-packages/torch/_utils.py in reraise(self)
    426             # have message field
    427             raise self.exc_type(message=msg)
--> 428         raise self.exc_type(msg)
    429
    430

AttributeError: Caught AttributeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/worker.py", line 198, in _worker_loop
    data = fetcher.fetch(index)
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataset.py", line 272, in __getitem__
    return self.dataset[self.indices[idx]]
  File "<ipython-input-12-5595ac89d75d>", line 38, in __getitem__
    image, landmarks = self.transform(image, landmarks, self.crops[index])
  File "<ipython-input-9-e38df55ee0d4>", line 46, in __call__
    image = Image.fromarray(image)
  File "/usr/local/lib/python3.6/dist-packages/PIL/Image.py", line 2670, in fromarray
    arr = obj.__array_interface__
AttributeError: 'NoneType' object has no attribute '__array_interface__'

Output:
Basically it says when executing the line image = Image.fromarray(image), the Image.fromarray function is expecting image to be an array and that image implements a function called __array_interface__ that will turn itself into an image. However, during execution image is actually None (a python object type for nothing). Surely you can't turn None into an image.
There could be something wrong with your data. I'd suggest not doing the random split first and check if the each item in the dataset is not None.


Instruction: Tensorflow to PyTorch

Input:
I'm transffering a Tensorflow code to a PyTorch code.
Below lines are the problem I couldn't solve yet.
I'm not familiar with PyTorch so that it's not easy for me to find the matching methods in PyTorch library.
Anyone can help me?
p.s. The shape of alpha is (batch, N).
alpha_cumsum = tf.cumsum(alpha, axis = 1)
len_batch = tf.shape(alpha_cumsum)[0]
rand_prob = tf.random_uniform(shape = [len_batch, 1], minval = 0., maxval = 1.)
alpha_relu = tf.nn.relu(rand_prob - alpha_cumsum)
alpha_index = tf.count_nonzero(alpha_relu, 1)
alpha_hard  = tf.one_hot(alpha_index, len(a))

Output:
I've put all your functions followed by the corresponding pytorch function. Most are the same name and put in the pytorch docs (https://pytorch.org/docs/stable/index.html)
tf.cumsum(alpha, axis = 1)
torch.cumsum(alpha, dim=1)

tf.shape(alpha_cumsum)[0]
alpha_cumsum.shape[0]

tf.random_uniform(shape = [len_batch, 1], minval = 0., maxval = 1.)
torch.rand([len_batch,1])

tf.nn.relu(rand_prob - alpha_cumsum)
torch.nn.functional.relu(rand_prob - alpha_cumsum)

tf.count_nonzero(alpha_relu, 1)
torch.count_nonzero(alpha_relu, dim=1)

tf.one_hot(alpha_index, len(a))
torch.nn.functional.one_hot(alpha_index, len(a)) # assuming len(a) is number of classes


Instruction: How do I reshape pytorch tensor from [a, b*c] to [b,a,c]?

Input:
I'm getting this code error, and I am unsure on how to reshape my tensor.
RuntimeError: Expected 3-dimensional input for 3-dimensional weight [32, 35, 2], but got 2-dimensional input of size [35, 64] instead

Output:
You can use reshape function from torch as follows:
a = torch.rand((35,64))
b = torch.reshape(a, (32,35,2))


Instruction: Can't install pytorch: no matching distribution found for torch==1.7.0+cpu

Input:
When trying to install PyTorch, I can't progress past these errors:
ERROR: Could not find a version that satisfies the requirement torch==1.7.0+cpu (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)
ERROR: No matching distribution found for torch==1.7.0+cpu

Output: The issue likely has to do with your Python version. PyTorch will not work with the 32-bit version of Python, so if you're using 32-bit it will give that error. I had this happen to me before, switching to 64-bit should fix the error.


Instruction: PyTorch - TypeError: forward() takes 1 positional argument but 2 were given

Input:
I do not understand where this error comes from, the number of arguments to the model seems correct, below is my model:
class MancalaModel(nn.Module):

    def __init__(self, n_inputs=16, n_outputs=16):
        super().__init__()

        n_neurons = 256

        def create_block(n_in, n_out):
            block = nn.ModuleList()
            block.append(nn.Linear(n_in, n_out))
            block.append(nn.ReLU())
            return block

        self.blocks = nn.ModuleList()
        self.blocks.append(create_block(n_inputs, n_neurons))
        for _ in range(6):
            self.blocks.append(create_block(n_neurons, n_neurons))

        self.actor_block = nn.ModuleList()
        self.critic_block = nn.ModuleList()
        for _ in range(2):
            self.actor_block.append(create_block(n_neurons, n_neurons))
            self.critic_block.append(create_block(n_neurons, n_neurons))

        self.actor_block.append(create_block(n_neurons, n_outputs))
        self.critic_block.append(create_block(n_neurons, 1))

        self.apply(init_weights)

    def forward(self, x):
        x = self.blocks(x)
        actor = F.softmax(self.actor_block(x))
        critics = self.critic_block(x)
        return actor, critics

Then I create an instance and make a forward pass with random number
model = MancalaModel()
x = model(torch.rand(1, 16))

Then I got the TypeError saying the number of arguments is not correct:
      2 model = MancalaModel()
----> 3 x = model(torch.rand(1, 16))
      4 # summary(model, (16,), device='cpu')
      5

d:\environments\python\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

D:\UOM\Year3\AI & Games\KalahPlayer\agents\model_agent.py in forward(self, x)
     54
     55     def forward(self, x):
---> 56         x = self.blocks(x)
     57         actor = F.softmax(self.actor_block(x))
     58         critics = self.critic_block(x)

d:\environments\python\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

TypeError: forward() takes 1 positional argument but 2 were given

Any help is appreciated, thanks!

Output:
TL;DR
You are trying to forward through nn.ModuleList - this is not defined.
You need to convert self.blocks to nn.Sequential:
        def create_block(n_in, n_out):
            # do not work with ModuleList here either.
            block = nn.Sequential(
              nn.Linear(n_in, n_out),
              nn.ReLU()
            )
            return block

        blocks = []  # simple list - not a member of self, for temporal use only.
        blocks.append(create_block(n_inputs, n_neurons))
        for _ in range(6):
            blocks.append(create_block(n_neurons, n_neurons))
        self.blocks = nn.Sequential(*blocks)  # convert the simple list to nn.Sequential

I was expecting you to get NotImplementedError, and not this TypeError, because your self.blocks is of type nn.ModuleList and its forward method throws NotImplementedError. I just made a pull request to fix this confusing issue.
Update (April 22nd, 2021): the PR was merged. In future versions you should see NotImplementedError when calling nn.ModuleList or nn.ModuleDict.


Instruction: creating a train and a test dataloader

Input:
I have actually a directory RealPhotos containing 17000 jpg photos. I would be interested in creating a train dataloader and a test dataloader
ls RealPhotos/
2007_000027.jpg  2008_007119.jpg  2010_001501.jpg  2011_002987.jpg
2007_000032.jpg  2008_007120.jpg  2010_001502.jpg  2011_002988.jpg
2007_000033.jpg  2008_007123.jpg  2010_001503.jpg  2011_002992.jpg
2007_000039.jpg  2008_007124.jpg  2010_001505.jpg  2011_002993.jpg
2007_000042.jpg  2008_007129.jpg  2010_001511.jpg  2011_002994.jpg
2007_000061.jpg  2008_007130.jpg  2010_001514.jpg  2011_002996.jpg
2007_000063.jpg  2008_007131.jpg  2010_001515.jpg  2011_002997.jpg
2007_000068.jpg  2008_007133.jpg  2010_001516.jpg  2011_002999.jpg
2007_000121.jpg  2008_007134.jpg  2010_001518.jpg  2011_003002.jpg
2007_000123.jpg  2008_007138.jpg  2010_001520.jpg  2011_003003.jpg
...

I know I can subclassing TensorDataset to make it compatible with unlabeled data with
class UnlabeledTensorDataset(TensorDataset):
    """Dataset wrapping unlabeled data tensors.

    Each sample will be retrieved by indexing tensors along the first
    dimension.

    Arguments:
        data_tensor (Tensor): contains sample data.
    """
    def __init__(self, data_tensor):
        self.data_tensor = data_tensor

    def __getitem__(self, index):
        return self.data_tensor[index]

And something along these lines for training the autoencoder
X_train     = rnd.random((300,100))
train       = UnlabeledTensorDataset(torch.from_numpy(X_train).float())
train_loader= data_utils.DataLoader(train, batch_size=1)

for epoch in range(50):
    for batch in train_loader:
        data = Variable(batch)
        optimizer.zero_grad()
        output = model(data)
        loss = criterion(output, data)

Output:
You first need to define a Dataset (torch.utils.data.Dataset) then you can use DataLoader on it. There is no difference between your train and test dataset, you can define a generic dataset that will look into a particular directory and map each index to a unique file.
class MyDataset(Dataset):
    def __init__(self, directory):
        self.files = os.listdir(directory)

    def __getitem__(self, index):
        img = Image.open(self.files[index]).convert('RGB')
        return T.ToTensor()(img)

Where T refers to torchvision.transform and Image is imported from PIL.
You can then instanciate a dataset with
data_set = MyDataset('./RealPhotos')

From there you can use torch.utils.data.random_split to perform the split:
train_len = int(len(data_set)*0.7)
train_set, test_set = random_split(data_set, [train_len, data_set - train_len])

Then use torch.utils.data.DataLoader as you did:
train_loader = DataLoader(train_set, batch_size=1, shuffle=True)
test_loader = DataLoader(test_set, batch_size=16, shuffle=False)


Instruction: Codes worked fine one week ago, but keep getting error since yesterday: Fine-tuning Bert model training via PyTorch on Colab

Input:
I am new to Bert. Two weeks ago I successfully ran a fine-tuning Bert model on a nlp classification task though the outcome was not brilliant. Yesterday, however, when I tried to run the same code and data, an AttributeError was always there, which says: 'str' object has no attribute 'dim'. Please know everything is on Colab and via PyTorch Transformers.
What should I do to fix it?
Here is one thing I tried when I installed transformers but turned out it did not work:
instead of
!pip install transformers ,
I tried to use previous transformers version:
!pip install --target lib --upgrade transformers==3.5.0
Any feedback will be greatly appreciated!
Please see the code and the error message as below:
Code:

train definition

# function to train the model
def train():

  model.train()

  total_loss, total_accuracy = 0, 0

  # empty list to save model predictions
  total_preds=[]

  # iterate over batches
  for step,batch in enumerate(train_dataloader):

    # progress update after every 50 batches.
    if step % 200 == 0 and not step == 0:
      print('  Batch {:>5,}  of  {:>5,}.'.format(step, len(train_dataloader)))

    # push the batch to gpu
    batch = [r.to(device) for r in batch]

    sent_id, mask, labels = batch

    # clear previously calculated gradients
    model.zero_grad()

    # get model predictions for the current batch
    preds = model(sent_id, mask)

    # compute the loss between actual and predicted values
    loss = cross_entropy(preds, labels)

    # add on to the total loss
    total_loss = total_loss + loss.item()

    # backward pass to calculate the gradients
    loss.backward()

    # clip the the gradients to 1.0. It helps in preventing the exploding gradient problem
    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)

    # update parameters
    optimizer.step()

    # update learning rate schedule
    # scheduler.step()

    # model predictions are stored on GPU. So, push it to CPU
    preds=preds.detach().cpu().numpy()

    # append the model predictions
    total_preds.append(preds)

  # compute the training loss of the epoch
  avg_loss = total_loss / len(train_dataloader)

  # predictions are in the form of (no. of batches, size of batch, no. of classes).
  # reshape the predictions in form of (number of samples, no. of classes)
  total_preds  = np.concatenate(total_preds, axis=0)

  #returns the loss and predictions
  return avg_loss, total_preds

training process

# set initial loss to infinite
best_valid_loss = float('inf')

# empty lists to store training and validation loss of each epoch
train_losses=[]
valid_losses=[]

#for each epoch
for epoch in range(epochs):

    print('\n Epoch {:} / {:}'.format(epoch + 1, epochs))

    #train model
    train_loss, _ = train()

    #evaluate model
    valid_loss, _ = evaluate()

    #save the best model
    if valid_loss < best_valid_loss:
        best_valid_loss = valid_loss
        torch.save(model.state_dict(), 'saved_weights.pt')

    # append training and validation loss
    train_losses.append(train_loss)
    valid_losses.append(valid_loss)

    print(f'\nTraining Loss: {train_loss:.3f}')
    print(f'Validation Loss: {valid_loss:.3f}')

Error message:

 Epoch 1 / 10
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-41-c5138ddf6b25> in <module>()
     12
     13     #train model
---> 14     train_loss, _ = train()
     15
     16     #evaluate model

5 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in linear(input, weight, bias)
   1686         if any([type(t) is not Tensor for t in tens_ops]) and has_torch_function(tens_ops):
   1687             return handle_torch_function(linear, tens_ops, input, weight, bias=bias)
-> 1688     if input.dim() == 2 and bias is not None:
   1689         # fused op is marginally faster
   1690         ret = torch.addmm(bias, input, weight.t())

AttributeError: 'str' object has no attribute 'dim'

Output:
As far as I remember - there was an old transformer version in colab. Something like 2.11.0. Try:
!pip install transformers~=2.11.0

Change the version number until it works.


Instruction: what does it mean when kernel depth=1 in conv3d pytorch

Input:
I would like to understand the difference between conv2d and conv3d in PyTorch.
What is the difference between:
conv3d(in, out, kernel_size(1,3,3))

and
conv2d(in,out,kernel_size(3,3))

I checked the official documentation but I couldn't quite understand the difference between the two. Should conv3d in this case be the same as conv2d since the depth is 1? Any help would be appreciated.

Output:
Well to give an intuitive understanding of how a kernel works I would recommend to look how the cells(a.k.a Neurons) of a single feature layer are obtained.
For the 2D convolution you have the hight and width of the kernel layer(KL) that goes over a single feature layer(FL) from a 2D convolutional layers(CL). Because a CL can have numerous FLs, multiple KLs are then created in parallel and placed over the different FLs, so it can process them in parallel. This parallel processing is usually illustrated as a "stacked" structure of KLs, which is usually named just Kernel. Not only the singularity of the name "Kernel" from this plural structure can leads to confusion, but also the fact that the parallel processing of KLs are illustrated commonly as a stacking of KLs, creating an illusion of a depth dimension.
So important to remember: this is not a third dimension, this is just a parallel processing of KLs-FLs.
For a 3D convolution you need now 3D KL to process the 3D FL. This is now a true third dimension. The number of 3D KLs that will be "stacked", will be automatically adjusted so it can process parallel the multiple 3D FLs from your network.
Regarding your question:
Indeed, a 3D kernel with a depth 1 is the same as a 2D kernel, however the functions are builded differently for the different cases. That is, 2D for images with two dimensions (hight, width), and 3D for Images with three dimensions (hight, width, depth).


Instruction: Pytorch model training CPU Memory leak issue

Input:
When I trained my pytorch model on GPU device,my python script was killed out of blue.Dives into OS log files , and I find script was killed by OOM killer because my CPU ran out of memory.It’s very strange that I trained my model on GPU device but I ran out of my CPU memory.
Snapshot of OOM killer log file

In order to  debug this issue,I install python memory profiler. Viewing log file from memory profiler, I find when column wise -= operation  occurred, my CPU memory gradually increased until OOM killer killed my
program.
Snapshot of Python memory profiler

It’s very strange, I try many ways to solve this issue.Finally, I found before assignment operation,I detach Tensor first.Amazingly,it solves this issue.But I don’t understand clearly why it works.Here is my original function code.
def GeneralizedNabla(self, image):
        pad_size = 2
        affinity = torch.zeros(image.shape[0], self.window_size**2, self.h, self.w).to(self.device)
        h = self.h+pad_size
        w = self.w+pad_size
        #pad = nn.ZeroPad2d(pad_size)
        image_pad = self.pad(image)
        for i in range(0, self.window_size**2):
            affinity[:, i, :, :] = image[:, :, :].detach()  # initialization
            dy = int(i/5)-2
            dx = int(i % 5)-2
            h_start = pad_size+dy
            h_end = h+dy  # if 0 <= dy else h+dy
            w_start = pad_size+dx
            w_end = w+dx  # if 0 <= dx else w+dx
            affinity[:, i, :, :] -= image_pad[:, h_start:h_end, w_start:w_end].detach()
        self.Nabla=affinity
        return

If everyone has any ideas,I will appreciate very much, thank you.

Output:
Previously when you did not use the .detach() on your tensor, you were also accumulating the computation graph as well and as you went on, you kept acumulating more and more until you ended up exuasting your memory to the point it crashed.
When you do a detach(), you are effectively getting the data without the previously entangled history thats needed for computing the gradients.


Instruction: how to add L2 regulazation to cost function pytorch?

Input:
I try add L1 and L2 regulazation in my Loss function. But I fail.
My code:
criterion = nn.NLLLoss() + nn.L1Loss()
it should be perfect for something like this:
criterion = nn.NLLLoss() + _lambda * nn.L1Loss()
How can i do it?

Output:
You need to first instantiate them both and them add them. they both expect two arguments :

nll_loss = nn.NLLLoss()
l1_loss = nn.L1Loss()
loss = nll_loss(x, y) + _lambda * l1_loss(x, y)


Instruction: Filtering rows in pytorch tensor

Input:
If i have tensor which holds information on detected images in following shape:
        [[595.00000, 179.62500, 628.00000, 283.00000,   0.89062,   0.00000],
        [142.87500, 167.62500, 201.62500, 324.00000,   0.88086,   0.00000],
        [311.75000, 170.50000, 368.75000, 320.50000,   0.87549,   0.00000],
        [555.50000, 173.75000, 593.50000, 280.50000,   0.85791,   0.00000],
        [398.50000, 179.00000, 425.50000, 265.00000,   0.84180,   0.00000],
        [445.75000, 177.75000, 479.25000, 270.75000,   0.82129,   0.00000]]

where each row represents image with following parameters :
[   top,    bottom, left,   right, confidence, class    ]

What is most effective way of dropping images (rows) that are in size smaller than some user defined input for height which is top-bottom?
Naturally I would iterate over rows and than drop each row where top-bottom < someValue with some list comprehension but I suspect there could be better way of doing this.

Output:
How about I drop some benchmarks (if that would be interesting to you)?
Benchmarks
PyTorch-ic way:
In[2]: import torch
  ...: a = torch.Tensor(
  ...:      [[595.00000, 179.62500, 628.00000, 283.00000,   0.89062,   0.00000],
  ...:         [142.87500, 167.62500, 201.62500, 324.00000,   0.88086,   0.00000],
  ...:         [311.75000, 170.50000, 368.75000, 320.50000,   0.87549,   0.00000],
  ...:         [555.50000, 173.75000, 593.50000, 280.50000,   0.85791,   0.00000],
  ...:         [398.50000, 179.00000, 425.50000, 265.00000,   0.84180,   0.00000],
  ...:         [445.75000, 177.75000, 479.25000, 270.75000,   0.82129,   0.00000]])
In[3]: %timeit a[a[:, 0] - a[:, 1] > 300]
Out[3]: 24.5 µs ± 904 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)

In Numpy terms:
In[4]: import numpy as np
In[5]: %timeit np_arr[np.where(np_arr[:, 0] - np_arr[:, 1] > 300)]
Out[5]: 4.75 µs ± 713 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
In[6]: np_arr[np.where(np_arr[:, 0] - np_arr[:, 1] > 300)]

Equity
In[7]: torch.equal(torch.from_numpy(np_arr[np.where(np_arr[:, 0] - np_arr[:, 1] > 300)]), a[a[:, 0] - a[:, 1] > 300])
Out[7]: True

Conclusion is that using numpy for your comparisons would be way faster than PyTorch.


Instruction: Name of Modules to compute sparsity

Input:
I'm writing a function that computes the sparsity of the weight matrices of the following fully connected network:
class FCN(nn.Module):
    def __init__(self):
        super(FCN, self).__init__()

        self.fc1 = nn.Linear(input_dim, hidden_dim)
        self.relu1 = nn.ReLU()
        self.fc2 = nn.Linear(hidden_dim, hidden_dim)
        self.relu2 = nn.ReLU()
        self.fc3 = nn.Linear(hidden_dim, hidden_dim)
        self.relu3 = nn.ReLU()
        self.fc4 = nn.Linear(hidden_dim, output_dim)

    def forward(self, x):

        out = self.fc1(x)
        out = self.relu1(out)
        out = self.fc2(out)
        out = self.relu2(out)
        out = self.fc3(out)
        out = self.relu3(out)
        out = self.fc4(out)

        return out

The function I have written is the following:
def print_layer_sparsity(model):
    for name,module in model.named_modules():
        if 'fc' in name:
            zeros = 100. * float(torch.sum(model.name.weight == 0))
            tot = float(model.name.weight.nelement())
            print("Sparsity in {}.weight: {:.2f}%".format(name, zeros/tot))

But it gives me the following error:

torch.nn.modules.module.ModuleAttributeError: 'FCN' object has no attribute 'name'

It works fine when I manually enter the name of the layers (e.g.,

(model.fc1.weight == 0)
(model.fc2.weight == 0)
(model.fc3.weight == 0) ....

but I'd like to make it independent from the network. In other words, I'd like to adapt my function in a way that, given any sparse network, it prints the sparsity of every layer. Any suggestions?
Thanks!!

Output:
Try:
getattr(model, name).weight

In place of
model.name.weight

Your print_layer_sparsity function becomes:
def print_layer_sparsity(model):
    for name,module in model.named_modules():
        if 'fc' in name:
            zeros = 100. * float(torch.sum(getattr(model, name).weight == 0))
            tot = float(getattr(model, name).weight.nelement())
            print("Sparsity in {}.weight: {:.2f}%".format(name, zeros/tot))

You can't do model.name because name is a str. The in-built getattr function allows you to get the member variables / attributes of an object using its name as a string.
For more information, checkout this answer.


Instruction: mat1 dim 1 must match mat2 dim 0 - PyTorch

Input:
Im new to PyTorch and I keep getting the error mat1 dim1 must match mat1 dim0
this is my code for the network
class Net(Module):
    def __init__(self):
        super(Net, self).__init__()

        self.cnn_layers = Sequential(
            Conv1d(4,4,kernel_size=2, stride=1, padding=1),
            BatchNorm1d(4),
            ReLU(inplace=True),
            MaxPool1d(kernel_size=2,stride=1),
        )

        self.linear_layers = Sequential(
            Linear(8267*4,2)
        )

    def forward(self, x):
        print(x.shape)
        x = self.cnn_layers(x)
        print(x.shape)
        x = self.linear_layers(x)
        return x

and where the print statements are:
torch.Size([8267, 4, 1])
torch.Size([8267, 4, 1])

Any help/advice?

Output:
Assuming 8267 is your batch size. The output of your CNN is 8267x4x1. So you first need to flatten dim=1 and dim=2 into a single dimension to get a shape 8267x4. Then the following layer (dense) will require 4 neurons.
self.cnn_layers = Sequential(
    Conv1d(4, 4, kernel_size=2, stride=1, padding=1),
    BatchNorm1d(4),
    ReLU(inplace=True),
    MaxPool1d(kernel_size=2, stride=1))

self.linear_layers = Sequential(
    Flatten(),
    Linear(4, 2))


Instruction: I've 2 folders.One image in 1 folder and another in another folder. I have to compare two images and find the dissimilarity

Input:
I've 2 folders.One image in 1 folder and another in another folder. I have to compare two images and find the dissimilarity but the code is written random folder.
class InferenceSiameseNetworkDataset(Dataset):

    def __init__(self,imageFolderDataset,transform=None,should_invert=True):
        self.imageFolderDataset = imageFolderDataset
        self.transform = transform
        self.should_invert = should_invert

    def __getitem__(self,index):
        img0_tuple = random.choice(self.imageFolderDataset.imgs)
        img1_tuple = random.choice(self.imageFolderDataset.imgs)
        #we need to make sure approx 50% of images are in the same class
        should_get_same_class = random.randint(0,1)
        if should_get_same_class:
            while True:
                #keep looping till the same class image is found
                img1_tuple = random.choice(self.imageFolderDataset.imgs)
                if img0_tuple[1]==img1_tuple[1]:
                    break
        else:
            while True:
                #keep looping till a different class image is found

                img1_tuple = random.choice(self.imageFolderDataset.imgs)
                if img0_tuple[1] !=img1_tuple[1]:
                    break

        img0 = Image.open(img0_tuple[0])
        img1 = Image.open(img1_tuple[0])
        img0 = img0.convert("L")
        img1 = img1.convert("L")

        if self.should_invert:
            img0 = PIL.ImageOps.invert(img0)
            img1 = PIL.ImageOps.invert(img1)

        if self.transform is not None:
            img0 = self.transform(img0)
            img1 = self.transform(img1)

        return img0, img1 , torch.from_numpy(np.array([int(img1_tuple[1]!=img0_tuple[1])],dtype=np.float32))

    def __len__(self):
        return len(self.imageFolderDataset.imgs)

I took this code from GitHub and when I'm trying to compare the two images dissimilarity it's randomly choosing the images. The input folders are 2. one image should be in one folder and another image should be in another folder. When I'm trying to test it's testing on the same image sometimes I mean it's not checking for another image in another folder.
testing_dir1 = '/content/drive/My Drive/Signature Dissimilarity/Forged_Signature_Verification/processed_dataset/training1/'
folder_dataset_test = dset.ImageFolder(root=testing_dir1)
siamese_dataset = InferenceSiameseNetworkDataset(imageFolderDataset=folder_dataset_test,
                                        transform=transforms.Compose([transforms.Resize((100,100)),
                                                                      transforms.ToTensor()
                                                                      ])
                                       ,should_invert=False)

test_dataloader = DataLoader(siamese_dataset,num_workers=6,batch_size=1,shuffle=False)
dataiter = iter(test_dataloader)
x0,_,_ = next(dataiter)

for i in range(2):
  _,x1,label2 = next(dataiter)
  concatenated = torch.cat((x0,x1),0)

  output1,output2 = net(Variable(x0).cuda(),Variable(x1).cuda())
  euclidean_distance = F.pairwise_distance(output1, output2)
  imshow(torchvision.utils.make_grid(concatenated),'Dissimilarity: {:.2f}'.format(euclidean_distance.item()))
  dis = 'Dissimilarity: {:.2f}'.format(euclidean_distance.item())
  dis1 = dis
  dis1 = dis1.replace("Dissimilarity:", "").replace(" ", "")
  print(dis)
  if float(dis1) < 0.5:
    print("It's Same Signature")
  else:
    print("It's Forged Signature")

Output:
Just by assigning should_get_same_class=0 in __getitem__ function of your custom dataset class, InferenceSiameseNetworkDataset you can ensure that two images belong to different class/folder.
Secondly, You should not concatinate samples from two batches that may not satisfy your condition. You should use x0,x1,label2 = next(dataiter) under the scope of loop followed by concatination.


Instruction: How do I add PyTorch w/ CUDA to Dask Helm Chart

Input:
Install PyTorch compiled for CUDA into the Dask helm chart, and it failed:
Install PyTorch for CUDA per the instructions on pytorch.org (see image below).
Dask helm chart example fails:
- name: EXTRA_CONDA_PACKAGES
    value: "pytorch torchvision torchaudio cudatoolkit=11.0 -c pytorch"

Output:
You may want to check out the RAPIDS helm chart, which is an extension of the Dask helm chart but with additional GPU support.
Install at runtime
The RAPIDS Docker images also support the same EXTRA_PIP_PACKAGES, EXTRA_CONDA_PACKAGES and EXTRA_APT_PACKAGES that the Dask Docker images do.
# config.yaml
dask:
  scheduler:
    image:
      repository: rapidsai/rapidsai
      tag: cuda11.0-runtime-ubuntu18.04-py3.8

  worker:
    image:
      repository: rapidsai/rapidsai
      tag: cuda11.0-runtime-ubuntu18.04-py3.8
    env:
      - name: EXTRA_CONDA_PACKAGES
        value: "-c pytorch pytorch torchvision torchaudio"

  # If you're using the bundled Jupyter Lab instance you probably want to install these here too
  jupyter:
    image:
      repository: rapidsai/rapidsai
      tag: cuda11.0-runtime-ubuntu18.04-py3.8
    env:
      - name: EXTRA_CONDA_PACKAGES
        value: "-c pytorch pytorch torchvision torchaudio"

$ helm install rapidstest rapidsai/rapidsai -f config.yaml

Install ahead of time
The above approach means the dependencies will be installed every time a worker starts. Therefore you may prefer to create your own custom Docker image with these dependencies already included.
# Dockerfile
FROM rapidsai/rapidsai:cuda11.0-runtime-ubuntu18.04-py3.8

RUN conda install -n rapids -c pytorch pytorch torchvision torchaudio

$ docker build -t jacobtomlinson/customrapids:latest .
$ docker push jacobtomlinson/customrapids:latest

# config.yaml
dask:
  scheduler:
    image:
      repository: jacobtomlinson/customrapids
      tag: latest

  worker:
    image:
      repository: jacobtomlinson/customrapids
      tag: latest

  # If you're using the bundled Jupyter Lab instance you probably want to install these here too
  jupyter:
    image:
      repository: jacobtomlinson/customrapids
      tag: latest

$ helm install rapidstest rapidsai/rapidsai -f config.yaml


Instruction: Pytorch tensor representing a 3D grid with color values

Input:
Given a list of density values (scalars) that represent the density of an X,Y,Z coordinate on a 3D grid, how would I create a single tensor that can store this information?
i.e. a tensor that has dimensions of 1x20x20x20 for example would represent a 20x20x20 grid such that:
print(tensor[:,x1,y1,z1])
0.6

print(tensor[:,x2,y2,z2])
0.4

Output:
Based on your comments, you would like to turn a (2000) 1D-tensor into a (1x20x20x20) 4D-tensor.
Assuiming you have your initial tensor layed out as something like:
X = torch.tensor([111,112,113,121,122,123,131,132,133,211,212,213,221,222,223,231,232,233,311,312,313,321,322,323,331,332,333])

This is as simple as using view on it:
xyz = X.view(3, 3, 3, 1)

And, X[0, 1, 2] will give you [123] as expected.


Instruction: Pytorch CNN NotImplementedError

Input:
Here is my network class:
class network(nn.Module):
    def __init__(self):
        super(network, self).__init__()
        self.conv1 = nn.Conv2d(3, 32, 3, stride=1, padding=1, padding_mode='zeros')
        self.conv2 = nn.Conv2d(32, 32, 3, stride=1, padding=1, padding_mode='zeros')
        self.conv3 = nn.Conv2d(32, 64, 3, stride=1, padding=1, padding_mode='zeros')
        self.conv4 = nn.Conv2d(64, 64, 3, stride=1, padding=1, padding_mode='zeros')
        self.maxpool1 = nn.MaxPool2d(2)     # 14 * 14
        self.conv5 = nn.Conv2d(64, 128, 3, stride=1, padding=1, padding_mode='zeros')
        self.conv6 = nn.Conv2d(128, 128, 3, stride=1, padding=1, padding_mode='zeros')
        self.conv7 = nn.Conv2d(128, 256, 3, stride=1, padding=1, padding_mode='zeros')
        self.conv8 = nn.Conv2d(256, 256, 3, stride=1, padding=1, padding_mode='zeros')
        self.maxpool2 = nn.MaxPool2d(2)     # 7 * 7
        self.gap = nn.AvgPool2d(7)
        self.fc1 = nn.Linear(256 * 1 * 1, 256)
        self.fc2 = nn.Linear(256, len(classes))

    def foward(self, x):
        x = torch.nn.ReLU(self.conv1(x))
        x = torch.nn.ReLU(self.conv2(x))
        x = torch.nn.ReLU(self.conv3(x))
        x = torch.nn.ReLU(self.conv4(x))

        x = self.maxpool1(x)

        x = torch.nn.ReLU(self.conv5(x))
        x = torch.nn.ReLU(self.conv6(x))
        x = torch.nn.ReLU(self.conv7(x))
        x = torch.nn.ReLU(self.conv8(x))

        x = self.maxpool2(x)
        x = self.gap(x)
        x = torch.nn.ReLU(self.fc1(x))
        x = torch.nn.Softmax(self.fc2(x))
        print(f"output shape: {x.shape}")

        return x

Main code
network = network()
print(network)
loss = nn.CrossEntropyLoss().to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-5)

for epoch in range(nb_epochs + 1):
    iter = -1
    for batch_index, sample in enumerate(dataloader):
        x_train, y_train = sample

        # Upload Dataset in GPU device
        X = x_train.to(device, dtype=torch.float)
        Y = y_train.to(device, dtype=torch.long)

        optimizer.zero_grad()
        output = network(X)
        cost = loss(output, torch.max(Y, 1)[1])
        cost.backward()
        optimizer.step()
        iter = iter + 1

        if iter % 100 == 99 or iter == dataloader.__len__():
            print(f"Epoch {epoch + 1}/{nb_epochs}   "
                  f"Iteration {iter + 1}/{dataloader.__len__()}     "
                  f"Loss {str(float(cost))[0:7]}")

print('Finished Training')

PATH = './cifar10_vgg.pth'
torch.save(model.state_dict(), PATH)

And Error displayed:
Traceback (most recent call last):
    File "D:/Files/works/0+Development/Python/0+DNN/0+torch_dc/tutorial_train_save.py", line 198, in <module>
        output = network(X)
    File "C:\Users\bolero\anaconda3\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
        result = self.forward(*input, **kwargs)
    File "C:\Users\bolero\anaconda3\lib\site-packages\torch\nn\modules\module.py", line 175, in _forward_unimplemented
        raise NotImplementedError
NotImplementedError

I just implemented a very simple Network with Pytorch, but it doesn't work.
I wrote the "init" function and the forward function separately.
I watched the pytorch tutorial, and I wrote code similarly in that tutorial.
What I am aware of is that when writing a network class, I should inherit nn.Module and create init and forward functions.
I know that the init function defines the layer type, and the forward function defines the actual operation part.
If anyone knows about this issue, please comment.

Output: Note the typo in def foward(self, x):, it should be def forward(self, x):.


Instruction: how do i deal with torchvision datasets

Input:
I am using a torchvision dataset called MNIST. I run the code in my python project to download it. Do I need to delete the code I wrote, or could I just continue writing my ai? Would the download code disturb it or redownload the whole dataset?
The code I use to download it:
train = datasets.MNIST("", train=True, download=True,
                       transform = transforms.Compose([transforms.ToTensor()]))

test = datasets.MNIST("", train=False, download=True,
                       transform = transforms.Compose([transforms.ToTensor()]))

The guy from the tutorial is using jupyter notebook, but I am using PyCharm.
What do i need to do?

Output: Just leave that in there, you need that to define your datasets: The download=True flag means that the dataset is only downloaded as long as it doesn't already exist in the specified folder: https://pytorch.org/docs/stable/torchvision/datasets.html#mnist


Instruction: PyTorch: How to multiply via broadcasting of two tensors with different shapes

Input:
I have the following two PyTorch tensors A and B.
A = torch.tensor(np.array([40, 42, 38]), dtype = torch.float64)

tensor([40., 42., 38.], dtype=torch.float64)

B = torch.tensor(np.array([[[1,2,3,4,5],[1,2,3,4,5],[1,2,3,4,5],[1,2,3,4,5],[1,2,3,4,5]], [[4,5,6,7,8],[4,5,6,7,8],[4,5,6,7,8],[4,5,6,7,8],[4,5,6,7,8]], [[7,8,9,10,11],[7,8,9,10,11],[7,8,9,10,11],[7,8,9,10,11],[7,8,9,10,11]]]), dtype = torch.float64)

tensor([[[ 1.,  2.,  3.,  4.,  5.],
         [ 1.,  2.,  3.,  4.,  5.],
         [ 1.,  2.,  3.,  4.,  5.],
         [ 1.,  2.,  3.,  4.,  5.],
         [ 1.,  2.,  3.,  4.,  5.]],

        [[ 4.,  5.,  6.,  7.,  8.],
         [ 4.,  5.,  6.,  7.,  8.],
         [ 4.,  5.,  6.,  7.,  8.],
         [ 4.,  5.,  6.,  7.,  8.],
         [ 4.,  5.,  6.,  7.,  8.]],

        [[ 7.,  8.,  9., 10., 11.],
         [ 7.,  8.,  9., 10., 11.],
         [ 7.,  8.,  9., 10., 11.],
         [ 7.,  8.,  9., 10., 11.],
         [ 7.,  8.,  9., 10., 11.]]], dtype=torch.float64)

Tensor A is of shape:
torch.Size([3])

Tensor B is of shape:
torch.Size([3, 5, 5])

How do I multiply tensor A with tensor B (using broadcasting) in such a way for eg. the first value in tensor A (ie. 40.) is multiplied with all the values in the first 'nested' tensor in tensor B, ie.
tensor([[[ 1.,  2.,  3.,  4.,  5.],
         [ 1.,  2.,  3.,  4.,  5.],
         [ 1.,  2.,  3.,  4.,  5.],
         [ 1.,  2.,  3.,  4.,  5.],
         [ 1.,  2.,  3.,  4.,  5.]],

and so on for the other 2 values in tensor A and the other two nested tensors in tensor B, respectively.
I could do this multiplication (via broadcasting) with numpy arrays if A and B are arrays of both shape (3,) - ie. A*B - but I can't seem to figure out a counterpart of this with PyTorch tensors. Any help would really be appreciated.

Output:
When applying broadcasting in pytorch (as well as in numpy) you need to start at the last dimension (check out https://pytorch.org/docs/stable/notes/broadcasting.html). If they do not match you need to reshape your tensor. In your case they can't directly be broadcasted:
      [3]  # the two values in the last dimensions are not one and do not match
[3, 5, 5]

Instead you can redefine A = A[:, None, None] before muliplying such that you get shapes
[3, 1, 1]
[3, 5, 5]

which satisfies the conditions for broadcasting.


Instruction: PyTorch import broken, unable to pip install

Input: So all of a sudden my code broke. Error: Unable to import torch, No module named torch. So I attempt to install torch; error, ModuleNotFoundError: No module named 'tools.nnwrap'. I delete my venv and re-create it, same thing. I try it outside the venv, same issue. I look up the issue, apparently I should go here. I do, try several of the install commands, with cuda, without, all broken. What do I do? Am on Arch Linux if that's relevant, maybe I broke something.

Output:
https://github.com/pytorch/pytorch/issues/47116
It's a recent issue with Python 3.9, as expected: it's sadly not compatible with PyTorch as of right now (04/12/2020), and the error message isn't explicit about it. The issue is currently still open and pending, and all you can do is revert to 3.8 for the time being.


Instruction: PyTorch: How to multiply elements in a list containing tuples of integer and tensor

Input:
I have the following list ABC_lst containing tuples of an integer and a tensor.
Code:
import numpy as np
import torch

A_int = 40
A_tsr = torch.tensor(np.array([[1,2,3,4,5], [1,2,3,4,5], [1,2,3,4,5], [1,2,3,4,5], [1,2,3,4,5]]))
A_tpl = (A_int, A_tsr)

B_int = 42
B_tsr = torch.tensor(np.array([[4,5,6,7,8], [4,5,6,7,8], [4,5,6,7,8], [4,5,6,7,8], [4,5,6,7,8]]))
B_tpl = (B_int, B_tsr)

C_int = 38
C_tsr = torch.tensor(np.array([[7,8,9,10,11], [7,8,9,10,11], [7,8,9,10,11], [7,8,9,10,11], [7,8,9,10,11]]))
C_tpl = (C_int, C_tsr)

ABC_lst = [A_tpl, B_tpl, C_tpl]
ABC_lst

Output:
[(40, tensor([[1, 2, 3, 4, 5],
          [1, 2, 3, 4, 5],
          [1, 2, 3, 4, 5],
          [1, 2, 3, 4, 5],
          [1, 2, 3, 4, 5]])),
 (42, tensor([[4, 5, 6, 7, 8],
          [4, 5, 6, 7, 8],
          [4, 5, 6, 7, 8],
          [4, 5, 6, 7, 8],
          [4, 5, 6, 7, 8]])),
 (38, tensor([[ 7,  8,  9, 10, 11],
          [ 7,  8,  9, 10, 11],
          [ 7,  8,  9, 10, 11],
          [ 7,  8,  9, 10, 11],
          [ 7,  8,  9, 10, 11]]))]

How do I multiply the integer with the corresponding tensor, for eg. multiply 40 with
tensor([[1, 2, 3, 4, 5],
          [1, 2, 3, 4, 5],
          [1, 2, 3, 4, 5],
          [1, 2, 3, 4, 5],
          [1, 2, 3, 4, 5]])

multiply 42 with
tensor([[4, 5, 6, 7, 8],
          [4, 5, 6, 7, 8],
          [4, 5, 6, 7, 8],
          [4, 5, 6, 7, 8],
          [4, 5, 6, 7, 8]])

and so on.
The returned object should be a tensor, which looks like this:
tensor([[[ 40.,  80., 120., 160., 200.],
         [ 40.,  80., 120., 160., 200.],
         [ 40.,  80., 120., 160., 200.],
         [ 40.,  80., 120., 160., 200.],
         [ 40.,  80., 120., 160., 200.]],

        [[168., 210., 252., 294., 336.],
         [168., 210., 252., 294., 336.],
         [168., 210., 252., 294., 336.],
         [168., 210., 252., 294., 336.],
         [168., 210., 252., 294., 336.]],

        [[266., 304., 342., 380., 418.],
         [266., 304., 342., 380., 418.],
         [266., 304., 342., 380., 418.],
         [266., 304., 342., 380., 418.],
         [266., 304., 342., 380., 418.]]])

In the above eg., I have 3 "sets" of integer and tensor. How do I generalize a code for the multiplication above for any arbitrary "sets" of integer and tensor?
Would really appreciate it if anyone could help.
EDIT: I need to do all the above in GPU, so need to work with tensors.

Output:
Starting from two lists: I the list of integers and X the list of tensors:
I = [torch.tensor(40), torch.tensor(42), torch.tensor(38)]
X = [
    torch.tensor([[1,2,3,4,5], [1,2,3,4,5], [1,2,3,4,5], [1,2,3,4,5], [1,2,3,4,5]]),
    torch.tensor([[4,5,6,7,8], [4,5,6,7,8], [4,5,6,7,8], [4,5,6,7,8], [4,5,6,7,8]]),
    torch.tensor([[7,8,9,10,11], [7,8,9,10,11], [7,8,9,10,11], [7,8,9,10,11], [7,8,9,10,11]]),
]

You can zip both and create a list containing all multiplications results. Then stack this list into a single tensor, like so:
torch.stack([i*x for i, x in zip(I, X)])

You can, of course, add more elements to your lists.


Instruction: Examples or explanations of pytorch dataloaders?

Input:
I am fairly new to Pytorch (and have never done advanced coding). I am trying to learn the basics of deep learning using the d2l.ai textbook but am having trouble with understanding the logic behind the code for dataloaders. I read the torch.utils.data docs and am not sure what the DataLoader class is meant for, and when for example I am supposed to use the  torch.utils.data.TensorDataset class in combination with it. For example, d2l defines a function:
def load_array(data_arrays, batch_size, is_train=True):
    """Construct a PyTorch data iterator."""
    dataset = data.TensorDataset(*data_arrays)
    return data.DataLoader(dataset, batch_size, shuffle=is_train)

I assume this is supposed to return an iterable that iterates over different batches. However, I don't understand what the data.TensorDataset part does (seems like there are a lot of options listed on the docs page). Also, the documents say that there are two types of datasets: iterable and map style. When describing the former type, it says
"This type of datasets is particularly suitable for cases where random reads are expensive or even improbable, and where the batch size depends on the fetched data."
What does it mean for "a random read to be expensive or improbable" and for the batch_size to depend on the fetched data? Can anyone give an example of this?
If there is any source where a CompSci noob like me can learn these basics, I'd really appreciate tips!
Thanks very much!

Output:
I'll give you an example of how to use dataloaders and will explain the steps:
Dataloaders are iterables over the dataset. So when you iterate over it, it will return B randomly from the dataset collected samples (including the data-sample and the target/label), where B is the batch-size.
To create such a dataloader you will first need a class which inherits from the Dataset Pytorch class. There is a standard implementation of this class in pytorch which should be TensorDataset. But the standard way is to create an own one. Here is an example for image classification:
import torch
from PIL import Image

class YourImageDataset(torch.utils.data.Dataset):
    def __init__(self, image_folder):
        self.image_folder = image_folder
        self.images = os.listdir(image_folder)

    # get sample
    def __getitem__(self, idx):
        image_file = self.images[idx]

        image = Image.open((self.image_folder + image_file))
        image = np.array(image)

        # normalize image
        image = image / 255

        # convert to tensor
        image = torch.Tensor(image).reshape(3, 512, 512)

        # get the label, in this case the label was noted in the name of the image file, ie: 1_image_28457.png where 1 is the label and the number at the end is just the id or something
        target = int(image_file.split("_")[0])
        target = torch.Tensor(target)

        return image, target

    def __len__(self):
        return len(self.images)

To get an example image you can call the class and pass some random index into the getitem function. It will then return the tensor of the image matrix and the tensor of the label at that index. For example:
dataset = YourImageDataset("/path/to/image/folder")
data, sample = dataset.__getitem__(0) # get data at index 0

Alright, so now you have created the class which preprocesses and returns ONE sample and its label. Now we have to create the datalaoder, which "wraps" around this class and then can return whole batches of samples from your dataset class.
Lets create three dataloaders, one which iterates over the train set, one for the test set and one for the validation set:
dataset = YourImageDataset("/path/to/image/folder")

# lets split the dataset into three parts (train 70%, test 15%, validation 15%)
test_size = 0.15
val_size = 0.15

test_amount, val_amount = int(dataset.__len__() * test_size), int(dataset.__len__() * val_size)

# this function will automatically randomly split your dataset but you could also implement the split yourself
train_set, val_set, test_set = torch.utils.data.random_split(dataset, [
            (dataset.__len__() - (test_amount + val_amount)),
            test_amount,
            val_amount
])

# B is your batch-size, ie. 128

train_dataloader = torch.utils.data.DataLoader(
            train_set,
            batch_size=B,
            shuffle=True,
)
val_dataloader = torch.utils.data.DataLoader(
            val_set,
            batch_size=B,
            shuffle=True,
)
test_dataloader = torch.utils.data.DataLoader(
            test_set,
            batch_size=B,
            shuffle=True,
)

Now you have created your dataloaders and are ready to train!
For example like this:

for epoch in range(epochs):

    for images, targets in train_dataloder:
        # now 'images' is a batch containing B samples
        # and 'targets' is a batch containing B targets (of the images in 'images' with the same index

        optimizer.zero_grad()
        images, targets = images.cuda(), targets.cuda()
        predictions = model.train()(images)

        . . .

Normally you would create an own file for the "YourImageDataset" class and then import to the file in which you want to create the dataloaders.
I hope I could make clear what the role of the dataloader and the Dataset class is and how to use them!

I don't know much about iter-style datasets but from what I understood: The method I showed you above, is the map-style. You use that, if your dataset is stored in a .csv, .json or whatever kind of file. So you can iterate through all rows or entries of the dataset. Iter-style will take you dataset or a part of the dataset and will convert in to an iterable. For example, if your dataset is a list, this is what an iterable of the list would look like:
dataset = [1,2,3,4]
dataset  = iter(dataset)

print(next(a))
print(next(a))
print(next(a))
print(next(a))

# output:
# >>> 1
# >>> 2
# >>> 3
# >>> 4

So the next will give you the next item of the list. Using this together with a Pytorch Dataloader is probably more efficient and faster. Normally the map-dataloader is fast enough and common to use, but the documentation supposed that when you are loading data-batches from a database (which can be slower) then iter-style dataset would be more efficient.
This explanation of iter-style is a bit vague but I hope it makes you understand what I understood. I would recommend you to use the map-style first, as I explained it in my original answer.


Instruction: Pytorch CrossEntropyLoss Tensorflow Equivalent

Input:
I am currently translating a pytorch code into tensorflow.
There is a point that i am aggregating 3 losses in a tensorflow custom loop and i get an error that i am passing a two dimensional array vs a 1 dimensional into CategoricalCrossEntropy of tensorflow which is very legit and i understand why this happens... but in the pytorch code i am passing the same shapes and it's perfectly working with CrossEntropyLoss. Does anybody know what i have to do to transfer this into TF?
the shapes that are passing in are (17000,100) vs (17000)

Output: Try using the loss loss=tf.keras.losses.sparse_categorical_crossentropy


Instruction: Feeding Multiple Inputs to LSTM for Time-Series Forecasting using PyTorch

Input:
I'm currently working on building an LSTM network to forecast time-series data using PyTorch. Following Roman's blog post, I implemented a simple LSTM for univariate time-series data, please see the class definitions below. However, it's been a few days since I ground to a halt on adding more features to the input data, say an hour of the day, day of the week, week of the year, and sorts.
class Model(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(Model, self).__init__()
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.output_size = output_size
        self.lstm = nn.LSTMCell(self.input_size, self.hidden_size)
        self.linear = nn.Linear(self.hidden_size, self.output_size)

    def forward(self, input, future=0, y=None):
        outputs = []

        # reset the state of LSTM
        # the state is kept till the end of the sequence
        h_t = torch.zeros(input.size(0), self.hidden_size, dtype=torch.float32)
        c_t = torch.zeros(input.size(0), self.hidden_size, dtype=torch.float32)

        for i, input_t in enumerate(input.chunk(input.size(1), dim=1)):
            h_t, c_t = self.lstm(input_t, (h_t, c_t))
            output = self.linear(h_t)
            outputs += [output]

        for i in range(future):
            if y is not None and random.random() > 0.5:
                output = y[:, [i]]  # teacher forcing
            h_t, c_t = self.lstm(output, (h_t, c_t))
            output = self.linear(h_t)
            outputs += [output]
        outputs = torch.stack(outputs, 1).squeeze(2)
        return outputs

class Optimization:
    "A helper class to train, test and diagnose the LSTM"

    def __init__(self, model, loss_fn, optimizer, scheduler):
        self.model = model
        self.loss_fn = loss_fn
        self.optimizer = optimizer
        self.scheduler = scheduler
        self.train_losses = []
        self.val_losses = []
        self.futures = []

    @staticmethod
    def generate_batch_data(x, y, batch_size):
        for batch, i in enumerate(range(0, len(x) - batch_size, batch_size)):
            x_batch = x[i : i + batch_size]
            y_batch = y[i : i + batch_size]
            yield x_batch, y_batch, batch

    def train(
        self,
        x_train,
        y_train,
        x_val=None,
        y_val=None,
        batch_size=100,
        n_epochs=20,
        dropout=0.2,
        do_teacher_forcing=None,
    ):
        seq_len = x_train.shape[1]
        for epoch in range(n_epochs):
            start_time = time.time()
            self.futures = []

            train_loss = 0
            for x_batch, y_batch, batch in self.generate_batch_data(x_train, y_train, batch_size):
                y_pred = self._predict(x_batch, y_batch, seq_len, do_teacher_forcing)
                self.optimizer.zero_grad()
                loss = self.loss_fn(y_pred, y_batch)
                loss.backward()
                self.optimizer.step()
                train_loss += loss.item()
            self.scheduler.step()
            train_loss /= batch
            self.train_losses.append(train_loss)

            self._validation(x_val, y_val, batch_size)

            elapsed = time.time() - start_time
            print(
                "Epoch %d Train loss: %.2f. Validation loss: %.2f. Avg future: %.2f. Elapsed time: %.2fs."
                % (epoch + 1, train_loss, self.val_losses[-1], np.average(self.futures), elapsed)
            )

    def _predict(self, x_batch, y_batch, seq_len, do_teacher_forcing):
        if do_teacher_forcing:
            future = random.randint(1, int(seq_len) / 2)
            limit = x_batch.size(1) - future
            y_pred = self.model(x_batch[:, :limit], future=future, y=y_batch[:, limit:])
        else:
            future = 0
            y_pred = self.model(x_batch)
        self.futures.append(future)
        return y_pred

    def _validation(self, x_val, y_val, batch_size):
        if x_val is None or y_val is None:
            return
        with torch.no_grad():
            val_loss = 0
            batch = 1
            for x_batch, y_batch, batch in self.generate_batch_data(x_val, y_val, batch_size):
                y_pred = self.model(x_batch)
                loss = self.loss_fn(y_pred, y_batch)
                val_loss += loss.item()
            val_loss /= batch
            self.val_losses.append(val_loss)

    def evaluate(self, x_test, y_test, batch_size, future=1):
        with torch.no_grad():
            test_loss = 0
            actual, predicted = [], []
            for x_batch, y_batch, batch in self.generate_batch_data(x_test, y_test, batch_size):
                y_pred = self.model(x_batch, future=future)
                y_pred = (
                    y_pred[:, -len(y_batch) :] if y_pred.shape[1] > y_batch.shape[1] else y_pred
                )
                loss = self.loss_fn(y_pred, y_batch)
                test_loss += loss.item()
                actual += torch.squeeze(y_batch[:, -1]).data.cpu().numpy().tolist()
                predicted += torch.squeeze(y_pred[:, -1]).data.cpu().numpy().tolist()
            test_loss /= batch
            return actual, predicted, test_loss

    def plot_losses(self):
        plt.plot(self.train_losses, label="Training loss")
        plt.plot(self.val_losses, label="Validation loss")
        plt.legend()
        plt.title("Losses")

You can find some of the helper functions that help me split and format data before feeding it to my LSTM network.
def to_dataframe(actual, predicted):
    return pd.DataFrame({"value": actual, "prediction": predicted})

def inverse_transform(scaler, df, columns):
    for col in columns:
        df[col] = scaler.inverse_transform(df[col])
    return df

def split_sequences(sequences, n_steps):
    X, y = list(), list()
    for i in range(len(sequences)):
        # find the end of this pattern
        end_ix = i + n_steps
        # check if we are beyond the dataset
        if end_ix > len(sequences):
            break
        # gather input and output parts of the pattern
        seq_x, seq_y = sequences[i:end_ix, :-1], sequences[end_ix-1, -1]
        X.append(seq_x)
        y.append(seq_y)
    return array(X), array(y)

def train_val_test_split_new(df, test_ratio=0.2, seq_len = 100):
    y = df['value']
    X = df.drop(columns = ['value'])
    tarin_ratio = 1 - test_ratio
    val_ratio = 1 - ((train_ratio - test_ratio) / train_ratio)

    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=test_ratio)
    X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=val_ratio)

    return X_train, y_train, X_val, y_val, X_test, y_test

I use the following data frames to train my model.
# df_train
value   weekday monthday    hour
timestamp
2014-07-01 00:00:00 10844   1   1   0
2014-07-01 00:30:00 8127    1   1   0
2014-07-01 01:00:00 6210    1   1   1
2014-07-01 01:30:00 4656    1   1   1
2014-07-01 02:00:00 3820    1   1   2
... ... ... ... ...
2015-01-31 21:30:00 24670   5   31  21
2015-01-31 22:00:00 25721   5   31  22
2015-01-31 22:30:00 27309   5   31  22
2015-01-31 23:00:00 26591   5   31  23
2015-01-31 23:30:00 26288   5   31  23
10320 rows × 4 columns

# x_train
weekday monthday    hour
timestamp
2014-08-26 16:30:00 1   26  16
2014-08-18 16:30:00 0   18  16
2014-10-22 20:00:00 2   22  20
2014-12-10 08:00:00 2   10  8
2014-07-27 22:00:00 6   27  22
... ... ... ...
2014-08-24 05:30:00 6   24  5
2014-11-24 12:00:00 0   24  12
2014-12-18 06:00:00 3   18  6
2014-07-27 17:00:00 6   27  17
2014-12-05 21:00:00 4   5   21
6192 rows × 3 columns

# y_train
timestamp
2014-08-26 16:30:00    14083
2014-08-18 16:30:00    14465
2014-10-22 20:00:00    25195
2014-12-10 08:00:00    21348
2014-07-27 22:00:00    16356
                       ...
2014-08-24 05:30:00     2948
2014-11-24 12:00:00    16292
2014-12-18 06:00:00     7029
2014-07-27 17:00:00    18883
2014-12-05 21:00:00    26284
Name: value, Length: 6192, dtype: int64

After transforming and splitting time-series data into smaller batches, the training data set for X and y becomes as follows:
X_data shape is (6093, 100, 3)
y_data shape is (6093,)
tensor([[[-1.0097,  1.1510,  0.6508],
         [-1.5126,  0.2492,  0.6508],
         [-0.5069,  0.7001,  1.2238],
         ...,
         [ 1.5044, -1.4417, -1.6413],
         [ 1.0016, -0.0890,  0.7941],
         [ 1.5044, -0.9908, -0.2087]],

        [[-1.5126,  0.2492,  0.6508],
         [-0.5069,  0.7001,  1.2238],
         [-0.5069, -0.6526, -0.4952],
         ...,
         [ 1.0016, -0.0890,  0.7941],
         [ 1.5044, -0.9908, -0.2087],
         [ 0.4988,  0.5874,  0.5076]],

        [[-0.5069,  0.7001,  1.2238],
         [-0.5069, -0.6526, -0.4952],
         [ 1.5044,  1.2637,  1.5104],
         ...,
         [ 1.5044, -0.9908, -0.2087],
         [ 0.4988,  0.5874,  0.5076],
         [ 0.4988,  0.5874, -0.6385]],

        ...,

        [[ 1.0016,  0.9255, -1.2115],
         [-1.0097, -0.9908,  1.0806],
         [-0.0041,  0.8128,  0.3643],
         ...,
         [ 1.5044,  0.9255, -0.9250],
         [-1.5126,  0.9255,  0.0778],
         [-0.0041,  0.2492, -0.7818]],

        [[-1.0097, -0.9908,  1.0806],
         [-0.0041,  0.8128,  0.3643],
         [-0.5069,  1.3765, -0.0655],
         ...,
         [-1.5126,  0.9255,  0.0778],
         [-0.0041,  0.2492, -0.7818],
         [ 1.5044,  1.2637,  0.7941]],

        [[-0.0041,  0.8128,  0.3643],
         [-0.5069,  1.3765, -0.0655],
         [-0.0041, -1.6672, -0.4952],
         ...,
         [-0.0041,  0.2492, -0.7818],
         [ 1.5044,  1.2637,  0.7941],
         [ 0.4988, -1.2163,  1.3671]]])
tensor([ 0.4424,  0.1169,  0.0148,  ..., -1.1653,  0.5394,  1.6037])

Finally, just to check if the dimensions of all these training, validation, and test datasets are correct, I print out their shapes.
train shape is: torch.Size([6093, 100, 3])
train label shape is: torch.Size([6093])
val shape is: torch.Size([1965, 100, 3])
val label shape is: torch.Size([1965])
test shape is: torch.Size([1965, 100, 3])
test label shape is: torch.Size([1965])

When I try to build the model as follows, I end up getting a RuntimeError pointing at inconsistent input sizes.
model_params = {'train_ratio': 0.8,
                'validation_ratio': 0.2,
                'sequence_length': 100,
                'teacher_forcing': False,
                'dropout_rate': 0.2,
                'batch_size': 100,
                'num_of_epochs': 5,
                'hidden_size': 24,
                'n_features': 3,
                'learning_rate': 1e-3
               }

train_ratio = model_params['train_ratio']
val_ratio = model_params['validation_ratio']
seq_len = model_params['sequence_length']
teacher_forcing = model_params['teacher_forcing']
dropout_rate = model_params['dropout_rate']
batch_size = model_params['batch_size']
n_epochs = model_params['num_of_epochs']
hidden_size = model_params['hidden_size']
n_features = model_params['n_features']
lr = model_params['learning_rate']

model = Model(input_size=n_features, hidden_size=hidden_size, output_size=1)
loss_fn = nn.MSELoss()
optimizer = optim.Adam(model.parameters(), lr=lr)
scheduler = optim.lr_scheduler.StepLR(optimizer, step_size=5, gamma=0.1)
optimization = Optimization(model, loss_fn, optimizer, scheduler)

start_time = datetime.now()
optimization.train(x_train, y_train, x_val, y_val,
                     batch_size=batch_size,
                     n_epochs=n_epochs,
                     dropout=dropout_rate,
                     do_teacher_forcing=teacher_forcing)

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-192-6fc406c0113d> in <module>
      6
      7 start_time = datetime.now()
----> 8 optimization.train(x_train, y_train, x_val, y_val,
      9                      batch_size=batch_size,
     10                      n_epochs=n_epochs,

<ipython-input-189-c18d20430910> in train(self, x_train, y_train, x_val, y_val, batch_size, n_epochs, dropout, do_teacher_forcing)
     68             train_loss = 0
     69             for x_batch, y_batch, batch in self.generate_batch_data(x_train, y_train, batch_size):
---> 70                 y_pred = self._predict(x_batch, y_batch, seq_len, do_teacher_forcing)
     71                 self.optimizer.zero_grad()
     72                 loss = self.loss_fn(y_pred, y_batch)

<ipython-input-189-c18d20430910> in _predict(self, x_batch, y_batch, seq_len, do_teacher_forcing)
     93         else:
     94             future = 0
---> 95             y_pred = self.model(x_batch)
     96         self.futures.append(future)
     97         return y_pred

~\Anaconda3\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

<ipython-input-189-c18d20430910> in forward(self, input, future, y)
     17
     18         for i, input_t in enumerate(input.chunk(input.size(1), dim=1)):
---> 19             h_t, c_t = self.lstm(input_t, (h_t, c_t))
     20             output = self.linear(h_t)
     21             outputs += [output]

~\Anaconda3\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

~\Anaconda3\lib\site-packages\torch\nn\modules\rnn.py in forward(self, input, hx)
    963
    964     def forward(self, input: Tensor, hx: Optional[Tuple[Tensor, Tensor]] = None) -> Tuple[Tensor, Tensor]:
--> 965         self.check_forward_input(input)
    966         if hx is None:
    967             zeros = torch.zeros(input.size(0), self.hidden_size, dtype=input.dtype, device=input.device)

~\Anaconda3\lib\site-packages\torch\nn\modules\rnn.py in check_forward_input(self, input)
    789     def check_forward_input(self, input: Tensor) -> None:
    790         if input.size(1) != self.input_size:
--> 791             raise RuntimeError(
    792                 "input has inconsistent input_size: got {}, expected {}".format(
    793                     input.size(1), self.input_size))

RuntimeError: input has inconsistent input_size: got 1, expected 3

I suspect my current LSTM model class does not support data with multiple features, and I've been trying out different approaches lately with no luck so far. Feel free to share your thoughts or point me in the right direction that could help me solve this problem.
As suggested by @stackoverflowuser2010, I printed out the shapes of the tensors input_t, h_t and c_t that is fed into the forward step before the error is thrown.
input_t
torch.Size([100, 1, 3])
h_t
torch.Size([100, 24])
c_t
torch.Size([100, 24])

Output:
After muddling through for a couple of weeks, I solved the issue. This has been a fruitful journey for me, so I'd like to share what I have discovered. If you'd like to have a look at the complete walk-through with code, please check out my Medium post on the matter.
Just as in Pandas, I found that things tend to work faster and smoother when I stick to the PyTorch way. Both libraries rely on NumPy, and I'm sure one can do pretty much all the table and matrix operations explicitly with NumPy arrays and functions. However, doing so does eliminate all the nice abstractions and performance improvements these libraries provide and turn each step into a CS exercise. It's fun until it isn't.
Rather than shaping all the training and validation sets manually to pass them to the model, PyTorch's TensorDataset and DataLoaders classes have immensely helped me. Scaling the feature and target sets for training and validation, we then have NumPy arrays. We can transform these arrays into Tensors and use these Tensors to create our TensorDataset, or a custom Dataset depending on your requirements. Finally, DataLoaders allow us to iterate over such datasets with much less hassle than otherwise as they already provide built-in batching, shuffling, and dropping the last batch options.
train_features = torch.Tensor(X_train_arr)
train_targets = torch.Tensor(y_train_arr)

val_features = torch.Tensor(X_val_arr)
val_targets = torch.Tensor(y_val_arr)

train = TensorDataset(train_features, train_targets)
train_loader = DataLoader(train, batch_size=64, shuffle=False, drop_last=True)

val = TensorDataset(val_features, val_targets)
val_loader = DataLoader(val, batch_size=64, shuffle=False, drop_last=True)

After transforming our data into iterable datasets, they can later be used to do mini-batch training. Instead of explicitly defining batches or wrestling with matrix operations, we can easily iterate over them via DataLoaders as follows.
model = LSTMModel(input_dim, hidden_dim, layer_dim, output_dim)

criterion = nn.MSELoss(reduction='mean')
optimizer = optim.Adam(model.parameters(), lr=1e-2)

train_losses = []
val_losses = []
train_step = make_train_step(model, criterion, optimizer)
device = 'cuda' if torch.cuda.is_available() else 'cpu'

for epoch in range(n_epochs):
    batch_losses = []
    for x_batch, y_batch in train_loader:
        x_batch = x_batch.view([batch_size, -1, n_features]).to(device)
        y_batch = y_batch.to(device)
        loss = train_step(x_batch, y_batch)
        batch_losses.append(loss)
    training_loss = np.mean(batch_losses)
    train_losses.append(training_loss)
    with torch.no_grad():
        batch_val_losses = []
        for x_val, y_val in val_loader:
            x_val = x_val.view([batch_size, -1, n_features]).to(device)
            y_val = y_val.to(device)
            model.eval()
            yhat = model(x_val)
            val_loss = criterion(y_val, yhat).item()
            batch_val_losses.append(val_loss)
        validation_loss = np.mean(batch_val_losses)
        val_losses.append(validation_loss)

    print(f"[{epoch+1}] Training loss: {training_loss:.4f}\t Validation loss: {validation_loss:.4f}")

Another cool feature that PyTorch provides is the view() function, which allows faster and memory-efficient reshaping of tensors. Since I earlier defined my LSTM model with batch_first = True, the batch tensor for the feature set must have the shape of (batch size, time steps, number of features). The line in the code above x_batch = x_batch.view([batch_size, -1, n_features]).to(device) just does that.
I hope this answer helps those dealing with similar problems or at least gives an idea of which direction to take. I had changed a lot in the code shared in the original post, but I'll not put them all here for the sake of simplicity. Feel free to check out the rest of it in my other SO post here.


Instruction: Efficiently filling torch.Tensor at equal index positions

Input:
I have a 6 dimensional all-zero pytorch tensor lrel_w that I want to fill with 1s at positions where the indices of the first three dimensions and the indices of the last three dimensions match. I'm currently solving this trivially using 3 nested for loops:
lrel_w = torch.zeros(
  input_size[0], input_size[1], input_size[2],
  input_size[0], input_size[1], input_size[2]
)
for c in range(input_size[0]):
  for x in range(input_size[1]):
    for y in range(input_size[2]):
      lrel_w[c,x,y,c,x,y] = 1

I'm sure there must be a more efficient way of doing this, however I have not been able to figure it out.

Output:
You can try this one.
import torch
c, m, n = input_size[0], input_size[1], input_size[2]

t = torch.zeros(c, m, n, c, m, n)
i, j, k = torch.meshgrid(torch.arange(c), torch.arange(m), torch.arange(n))
i = i.flatten()
j = j.flatten()
k = k.flatten()

t[i, j, k, i, j, k] = 1

Here is how meshgrid works in case you need reference.


Instruction: only one element tensors can be converted to python scalars

Input:
def Log(A):
    '''
    theta = arccos((tr(A)-1)/2)
    K=1/(2sin(theta))(A-A^T)
    log(A)=theta K
    '''
    theta=torch.acos(torch.tensor((torch.trace(A)-1)/2))
    K=(1/(2*torch.sin(theta)))*(torch.add(A,-torch.transpose(A,0,1)))

    return theta*K

def tensor_Log(A):
    blah=[[Log(A[i,j]) for j in range(A.shape[1])] for i in range(A.shape[0])]
    new=torch.tensor(blah)

    return new

ValueError: only one element tensors can be converted to Python scalars

during training to get the outputs of my network, the above function is producing the following error, it is called inside a custom layer and I do not know what it is referencing, any thoughts?

Output:
The issue in your list comprehension as blah is a list of list of tensors.
I would create a flat list of tensors by looping on A.shape[0] and A.shape[1] and then stack everything into a single tensor.
R = torch.stack([Log(A[i,j]) for i in range(A.shape[0]) for j in range(A.shape[1])])

You can then recover the desired format with reshape or view:
R.reshape(A.shape)


Instruction: Calculate Batch Pairwise Sinkhorn Distance in PyTorch

Input:
I have two tensors and both are of same shape. I want to calculate pairwise sinkhorn distance using GeomLoss.
What i have tried:
import torch
import geomloss  # pip install git+https://github.com/jeanfeydy/geomloss

a = torch.rand((8,4))
b = torch.rand((8,4))

geomloss.SamplesLoss('sinkhorn')(a,b)
# ^ input shape [batch, feature_dim]
# will return a scalar value

geomloss.SamplesLoss('sinkhorn')(a.unsqueeze(1),b.unsqueeze(1))
# ^ input shape [batch, n_points, feature_dim]
# will return a tensor of size [batch] of distances between a[i] and b[i] for each i

However I would like to compute pairwise distance where the resultant tensor should be of size [batch, batch]. To achieve this, I tried the following to use broadcasting:
geomloss.SamplesLoss('sinkhorn')(a.unsqueeze(0), b.unsqueeze(1))

But I got this error message:

ValueError: Samples x and y should have the same batchsize.

Output:
Since the documentation doesn't give examples on how to use the distance's forward function. Here's a way to do it, which will require you to call the distance function batch times.
We will construct the distance matrix line by line. Line i corresponds to the distances a[i]<->b[0], a[i]<->b[1], through to a[i]<->b[batch]. To do so we need to construct, for each line i, a (8x4) repeated version of tensor a[i].
This will do:
a_i = torch.stack(8*[a[i]], dim=0)

Then we calculate the distance between a[i] and each batch in b:
dist(a_i.unsqueeze(1), b.unsqueeze(1))

Having a total of batch lines we can construct our final tensor stack.

Here's the complete code:
batch = a.shape[0]
dist = geomloss.SamplesLoss('sinkhorn')
distances = [dist(torch.stack(batch*[a[i]]).unsqueeze(1), b.unsqueeze(1)) for i in range(batch)]
D = torch.stack(distances)


Instruction: Importing JPG files from a Folder for PyTorch transformation (Another error)?

Input:
I just recently posted a question about a file path error in which I thought I resolved... the folder that I have contains .jpg files. For some reason, i'm getting a new error:

RuntimeError: Found 0 files in subfolders of: C:\Users\Lyn\Desktop\UTKFaceDataSet\Train_Set
Supported extensions are: .jpg,.jpeg,.png,.ppm,.bmp,.pgm,.tif,.tiff,.webp

Here is my modified code:
data_dir = "C:\\Users\\Lyn\\Desktop\\UTKFaceDataSet"
train_dir = data_dir + '\\Train_Set'
test_dir = data_dir + '\\Test_Set'

training_transforms = transforms.Compose([transforms.Resize(100),
                                          transforms.ToTensor()])
testing_transforms = transforms.Compose([transforms.Resize(256),
                                         transforms.ToTensor()])

#Load the datasets with ImageFolder
training_dataset = datasets.ImageFolder(train_dir, transform=training_transforms)
testing_dataset = datasets.ImageFolder(test_dir, transform=testing_transforms)

Output:
You need your directory structure in the following format to use the datasets.ImageFolder():
dataset
   train_dataset
      dogs
         a.png, b.png
      cats
         c.png, d.png
   valid_dataset
      dogs
         e.png, f.png
      cats
         g.png, h.png

train_dir = 'dataset/train_dataset/'
training_dataset = datasets.ImageFolder(train_dir, transform=training_transforms)

I used two categories (dogs and cats), but it could be one also.


Instruction: Computing intermediate gradients using backward method in Pytorch

Input:
I am having trouble in understanding backward method in pytorch
x1 = tensor(2.).requires_grad_()
x2 = tensor(3.).requires_grad_() # or x2 = tensor(3.)
x3 = x1 + x2

l = (x3**2).sum()
l.backward()

print(x1)
print(x3)
print(x1.grad)
print(x3.grad)

Results are
tensor(2., requires_grad=True)
tensor(5., grad_fn=<AddBackward0>)
tensor(10.)
None

Why is x3.grad still None? Shouldn't it be tensor(10.) ?
When I run the following lines of code, x3.grad is evaluated to tensor(10.)
x3 = tensor(5.).requires_grad_()
l = (x3**2).mean()
l.backward()
print(x3.grad)

Output:
If you print x3.grad on your first example you might notice torch outputs a warning:

UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the gradient for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See here for more informations.

To save memory the gradients of the non-leaf tensors (non user-created tensors) are not buffered.
If you wish to see those gradients though you can retain the gradient on x3 by calling .retain_grad() before creating the graph (i.e. before calling .backward().
x3.retain_grad()
l.backward()
print(x3.grad)

will indeed output tensor(10.)


Instruction: Error " 'Softmax' object has no attribute 'log_softmax' " while training Neural Network with PyTorch

Input:
I am working on a classifier for MNIST dataset. When I am running the code below, I am getting the error " 'Softmax' object has no attribute 'log_softmax' " at line loss = loss_function(output, y). I have not managed to find a solution to the problem. I will appreciate if you can advise on how the issue can be resolved. Thank you.
import matplotlib.pyplot as plt
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.utils.data import DataLoader, Dataset, TensorDataset
import torchvision
import torchvision.transforms as transforms
import numpy as np

device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")
batch_size = 512

# Image transformations of Torchvision will convert to the images to tensor and normalise with mean and standard deviation
transformer = transforms.Compose([transforms.ToTensor(), transforms.Normalize(mean=(0.1307,), std=(0.3081,))])

data_train = DataLoader(torchvision.datasets.MNIST('Data/data/mnist', download=True, train=True, transform=transformer),
                batch_size=batch_size, drop_last=False, shuffle=True)
data_test = DataLoader(torchvision.datasets.MNIST('Data/data/mnist', download=True, train=False, transform=transformer),
                batch_size=batch_size, drop_last=False, shuffle=True)

class neural_nw(nn.Module):

    def __init__(self):
        super(neural_nw, self).__init__()
        self.fc1 = nn.Linear(784, 128, True)
        self.fc2 = nn.Linear(128, 128, True)
        self.fc3 = nn.Linear(128, 10, True)

    def forward(self, x):
        output = torch.sigmoid(self.fc1(x))
        output = torch.sigmoid(self.fc2(output))
        output = nn.Softmax(self.fc3(output))
        return output

MLP = neural_nw()

loss_function = nn.CrossEntropyLoss()
optimiser = optim.Adam(MLP.parameters(), lr = 0.01)

Epochs = 50

for epoch in range(Epochs):
    for X, y in data_train:
        X = X.view(X.shape[0], -1)
        optimiser.zero_grad()
        output = MLP.forward(X)
        loss = loss_function(output, y)
        loss.backward()
        optimiser.step()

Output:
nn.Softmax defines a module, nn.Modules are defined as Python classes and have attributes, e.g., a nn.LSTM module will have some internal attributes like self.hidden_size. On the other hand, F.softmax defines the operation and needs all arguments to be passed (including the weights and bias). Implicitly, the modules will usually call their functional counterpart somewhere in the forward method.
This explains why F.softmax instead of nn.Softmax resolves your issue.


Instruction: clang in neovim giving pp_file_not_found error for c++/pytorch basic example

Input:
I'm following this very basic c++/pytorch example:
pytorch_installing
And I can walk through this example with no errors. However, when creating the example-app.cpp file (or editing it at any point in time) with neovim, clang throws an error 'torch/torch.h' file not found [clang: pp_file_not_found]. My CMakeLists.txt file looks like this:
cmake_minimum_required(VERSION 3.0 FATAL_ERROR)
project(example-app)

find_package(Torch REQUIRED)
include_directories(SYSTEM /home/username/Downloads/libtorch/)
set(CMAKE_PREFIX_PATH "/home/username/Downloads/libtorch/share/cmake/Torch")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${TORCH_CXX_FLAGS}")

add_executable(example-app example-app.cpp)
target_link_libraries(example-app "${TORCH_LIBRARIES}")
set_property(TARGET example-app PROPERTY CXX_STANDARD 14)

This isn't a big issue for me with this small of a project. But I will be working on a larger project with libtorch and would like clang to recognize <torch/torch.h>. There were a couple similar Stackoverflow questions, but no answers.
Update:
I believe this is happening because clang is not seeing torch/torch.h because it is not part of the include paths. I printed the include paths for clang and it was not list. So I tried adding the include paths of libtorch to /usr/include/ but then it has issues seeing other header files referenced by header files I added. So I added the libtorch/include/torch/csrc/api/include/torch/ directory to /usr/include so that it can read #include <torch/torch.h. But then other header files are referenced in those header files with specified directory paths outside of the libtorch/include/torch/csrc/api/include/torch/ path.
For example, libtorch/include/torch/csrc/ contains WindowsTorchApiMacro.h. There are other paths within the libtorch directory that contain more header files. I tried to add all header files to \usr\include but still received an error. I'm sure the package as a whole needs to be used. The original cmake file looks like this:
cmake_minimum_required(VERSION 3.0 FATAL_ERROR)
project(example-app)

find_package(Torch REQUIRED)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${TORCH_CXX_FLAGS}")

add_executable(example-app example-app.cpp)
target_link_libraries(example-app "${TORCH_LIBRARIES}")
set_property(TARGET example-app PROPERTY CXX_STANDARD 14)

So cmake is able to find and reference all of those header files properly, and I know you can compile it at the command line with clang, but how can clang within neovim (during editing) see this package without error? I thought about using a package manager like vcpkg, but they do not have libtorch available, unfortunately. I can include output from a verbose and successful cmake run if needed.

Output:
Since the compilation succeeds with cmake-make, it is possible to ask CMake to generate the compilation database (typically a compile_commands.json file) with cmake -DCMAKE_EXPORT_COMPILE_COMMANDS=1.
Once the file is available, it can be read by the autocompletion compiler (clang in this case), which prevents the autocompleter from being lost in the filesystem, because all flags and headers are specified for all compilation units.
The compilation database just needs to be put where the autocompletion plugin expects it to be.


Instruction: Implement SeparableConv2D in Pytorch

Input:
Main objective
PyTorch equivalent for SeparableConv2D with padding = 'same':
from tensorflow.keras.layers import SeparableConv2D
x = SeparableConv2D(64, (1, 16), use_bias = False, padding = 'same')(x)

What is the PyTorch equivalent for SeparableConv2D?
This source says:

If groups = nInputPlane, kernel=(K, 1), (and before is a Conv2d layer with groups=1 and kernel=(1, K)), then it is separable.

While this source says:

Its core idea is to break down a complete convolutional acid into a two-step calculation, Depthwise Convolution and Pointwise.

This is my attempt:
class SeparableConv2d(nn.Module):
    def __init__(self, in_channels, out_channels, depth, kernel_size, bias=False):
        super(SeparableConv2d, self).__init__()
        self.depthwise = nn.Conv2d(in_channels, out_channels*depth, kernel_size=kernel_size, groups=in_channels, bias=bias)
        self.pointwise = nn.Conv2d(out_channels*depth, out_channels, kernel_size=1, bias=bias)

    def forward(self, x):
        out = self.depthwise(x)
        out = self.pointwise(out)
        return out

Is this correct? Is this equivalent to tensorflow.keras.layers.SeparableConv2D?
What about padding = 'same'?
How to ensure that my input and output size is the same while doing this?
My attempt:
x = F.pad(x, (8, 7, 0, 0), )

Because the kernel size is (1,16), I added left and right padding, 8 and 7 respectively. Is this the right way (and best way) to achieve padding = 'same'? How can I place this inside my SeparableConv2d class, and calculate on the fly given the input data dimension size?
All together
class SeparableConv2d(nn.Module):
    def __init__(self, in_channels, out_channels, depth, kernel_size, bias=False):
        super(SeparableConv2d, self).__init__()
        self.depthwise = nn.Conv2d(in_channels, out_channels*depth, kernel_size=kernel_size, groups=in_channels, bias=bias)
        self.pointwise = nn.Conv2d(out_channels*depth, out_channels, kernel_size=1, bias=bias)

    def forward(self, x):
        out = self.depthwise(x)
        out = self.pointwise(out)
        return out

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.separable_conv = SeparableConv2d(
            in_channels=32,
            out_channels=64,
            depth=1,
            kernel_size=(1,16)
        )

    def forward(self, x):
        x = F.pad(x, (8, 7, 0, 0), )
        x = self.separable_conv(x)
        return x

Any problem with these codes?

Output:
The linked definitions are generally agreeing. The best one is in the article.

"Depthwise" (not a very intuitive name since depth is not involved) - is a series of regular 2d convolutions, just applied to layers of the data separately. - "Pointwise" is same as Conv2d with 1x1 kernel.

I suggest a few correction to your SeparableConv2d class:

no need to use depth parameter - it is same as out_channels
I set padding to 1 to ensure same output size with kernel=(3,3). if kernel size is different - adjust padding accordingly, using same principles as with regular Conv2d. Your example class Net() is no longer needed - padding is done in SeparableConv2d.

This is the updated code, should be similar to tf.keras.layers.SeparableConv2D implementation:
class SeparableConv2d(nn.Module):

def __init__(self, in_channels, out_channels, kernel_size, bias=False):
    super(SeparableConv2d, self).__init__()
    self.depthwise = nn.Conv2d(in_channels, in_channels, kernel_size=kernel_size,
                               groups=in_channels, bias=bias, padding=1)
    self.pointwise = nn.Conv2d(in_channels, out_channels,
                               kernel_size=1, bias=bias)

def forward(self, x):
    out = self.depthwise(x)
    out = self.pointwise(out)
    return out


Instruction: multiplying each element of a matrix by a vector (or array)

Input:
Say I have a an input array of size (64,100)
t = torch.randn((64,100))

Now say I want to multiply each of the 6400 elements of t with 6400 separate vectors each of size 256 to produce a tensor of size [64, 100, 256]. This is what I am doing currently -
import copy
def clones(module, N):
    "Produce N identical layers."
    return nn.ModuleList([copy.deepcopy(module) for _ in range(N)])

linears = clones(nn.Linear(1,256, bias=False), 6400)

idx = 0
t_final = []
for i in range(64):
    t_bs = []
    for j in range(100):
        t1 = t[i, j] * linears[idx].weight.view(-1)
        idx += 1
        t_bs.append(t1)
    t_bs = torch.cat(t_bs).view(1, 100, 256)
    t_final.append(t_bs)
t_final = torch.cat(t_final)
print(t_final.shape)
Output: torch.Size([64, 100, 256])

Is there a faster and cleaner way of doing the same thing? I tried torch.matmul and torch.dot but couldn't do any better.

Output:
It seems broadcast is what you are looking for.
t = torch.randn((64,100)).view(6400, 1)
weights = torch.randn((6400, 256))

output = (t * weights).view(64, 100, 256)


Instruction: why autoencoder tutorial of pytorch changes the view of embedding layer output?

Input:
As shown here in PyTorch tutorials the code for an autoencoder model is like this:
class EncoderRNN(nn.Module):
    def __init__(self, input_size, hidden_size):
        super(EncoderRNN, self).__init__()
        self.hidden_size = hidden_size

        self.embedding = nn.Embedding(input_size, hidden_size)
        self.gru = nn.GRU(hidden_size, hidden_size)

    def forward(self, input, hidden):
        embedded = self.embedding(input).view(1, 1, -1)
        output = embedded
        output, hidden = self.gru(output, hidden)
        return output, hidden

    def initHidden(self):
        return torch.zeros(1, 1, self.hidden_size, device=device)

My question is what is the reason for use of view function on output of embedding layer?

Output:
The view function added extra dimension to given input shape to match expected input shape. In the function initHidden the hidden shape is initialized to (1, 1, 256).
def initHidden(self):
        return torch.zeros(1, 1, self.hidden_size, device=device)

Based on documentation, GRU input shape must have 3 dimensions, input of shape (seq_len, batch, input_size).
https://pytorch.org/docs/stable/generated/torch.nn.GRU.html
The shape of self.embedding(input) is (1, 256) and a sample output is,
tensor([[ 0.1421,  0.4135, -1.0619,  0.0149,  0.0673, -0.3770,  0.4231,  2.2803,
         -1.6939, -0.0071,  1.1131, -1.0019,  0.6593,  0.1366,  1.1033, -0.8804,
          1.3676,  0.4115, -0.5671,  0.3314, -0.2599, -0.3082,  1.3644,  0.5788,
         -0.1929, -2.0505,  0.4518,  0.8757, -0.2360, -0.4099, -0.5697, -1.5973,
         -0.6638, -1.1523,  1.4425,  1.3651,  1.9371,  0.5698, -0.3541, -1.3883,
         -0.0195, -1.0757, -1.4324, -1.6226, -2.4267,  0.3874, -0.7529,  1.4938,
         -2.5773, -1.1962,  0.3759, -0.6143, -1.0444, -0.6443, -0.8130, -1.7283,
          1.4167,  1.3945, -1.2695,  0.7289,  0.7777, -0.0094, -1.8108,  0.2126,
         -0.2018, -0.4055, -0.7779, -0.8523,  0.0162,  0.2463,  0.5588, -0.7250,
         -0.0128,  0.6272, -0.7729,  0.4259,  0.7596, -1.9500,  0.5853,  0.3764,
         -0.1112,  0.7274, -2.8535, -0.0445,  0.4225,  1.2179,  0.2219, -0.7064,
         -0.9654,  1.0501,  1.7142,  0.5312, -0.8180, -1.5697,  1.3062, -0.9321,
         -0.1652, -1.5298, -0.3575, -1.2046, -0.6571, -0.7689, -0.7032,  1.0727,
         -1.3259,  0.1200,  1.9357, -0.2519, -0.3717,  0.8054,  0.1180, -0.6921,
          1.0245, -1.5500, -0.5280, -0.7462,  0.7924,  2.2701, -1.5094, -0.1973,
         -1.5919,  0.4869,  0.6739, -0.5242,  0.2559, -0.0149, -0.5332, -1.8313,
          0.3598,  0.0804, -0.0780, -0.2930, -0.2844, -0.4752, -0.9919,  0.1809,
          0.7622, -2.5069, -0.7724, -0.9441,  1.6101,  0.6461, -0.8932,  0.0600,
          0.6911,  0.5191, -0.1719, -0.5829, -0.9168,  1.5282,  1.4399,  0.3264,
         -0.8894,  0.2880, -0.0697,  0.8977, -0.5004,  0.3844,  0.0925,  0.5592,
         -0.1664,  0.8575, -1.0348,  0.7326, -0.2124,  0.7533,  0.6270, -0.9559,
         -1.4159,  0.6788,  0.6163, -0.5951, -0.1403, -1.6088, -0.7731,  0.3876,
          1.0429, -2.0960,  0.1726,  1.7446, -0.3963,  0.0785, -0.4701,  1.0074,
          0.3319, -2.2675, -1.6163, -0.4003, -0.5468,  0.0452, -2.5586,  0.4747,
         -0.0271, -1.2161,  1.2121,  1.8738, -1.2207, -0.9218, -0.1430,  0.2512,
         -0.5236, -0.2544, -0.5868, -0.7086, -1.3328, -0.0243,  0.4759,  1.4125,
          0.4947,  0.5054,  1.6253,  0.4198, -0.9150,  0.6374,  0.4581,  1.1527,
          1.4440, -0.0590, -0.4601,  0.2490, -0.5739,  0.6798, -0.2156, -1.1386,
         -0.5011, -0.7411,  0.2825, -0.2595,  0.8070,  0.5270,  0.2595, -0.1089,
          0.4221, -0.7851,  0.7112, -0.3038,  0.6169, -0.1513, -0.5872,  0.3974,
          0.2431,  0.4934, -0.9406, -0.9372,  1.4525,  0.1376,  0.2558,  0.0661,
          0.3509,  2.1667,  2.8428,  0.9429, -0.6143, -1.0969,  0.0955,  0.0914]],
       device='cuda:0', grad_fn=<EmbeddingBackward>)

The shape of self.embedding(input).view(1, 1, -1) is (1, 1, 256) and a sample output is,
tensor([[[ 0.1421,  0.4135, -1.0619,  0.0149,  0.0673, -0.3770,  0.4231,
           2.2803, -1.6939, -0.0071,  1.1131, -1.0019,  0.6593,  0.1366,
           1.1033, -0.8804,  1.3676,  0.4115, -0.5671,  0.3314, -0.2599,
          -0.3082,  1.3644,  0.5788, -0.1929, -2.0505,  0.4518,  0.8757,
          -0.2360, -0.4099, -0.5697, -1.5973, -0.6638, -1.1523,  1.4425,
           1.3651,  1.9371,  0.5698, -0.3541, -1.3883, -0.0195, -1.0757,
          -1.4324, -1.6226, -2.4267,  0.3874, -0.7529,  1.4938, -2.5773,
          -1.1962,  0.3759, -0.6143, -1.0444, -0.6443, -0.8130, -1.7283,
           1.4167,  1.3945, -1.2695,  0.7289,  0.7777, -0.0094, -1.8108,
           0.2126, -0.2018, -0.4055, -0.7779, -0.8523,  0.0162,  0.2463,
           0.5588, -0.7250, -0.0128,  0.6272, -0.7729,  0.4259,  0.7596,
          -1.9500,  0.5853,  0.3764, -0.1112,  0.7274, -2.8535, -0.0445,
           0.4225,  1.2179,  0.2219, -0.7064, -0.9654,  1.0501,  1.7142,
           0.5312, -0.8180, -1.5697,  1.3062, -0.9321, -0.1652, -1.5298,
          -0.3575, -1.2046, -0.6571, -0.7689, -0.7032,  1.0727, -1.3259,
           0.1200,  1.9357, -0.2519, -0.3717,  0.8054,  0.1180, -0.6921,
           1.0245, -1.5500, -0.5280, -0.7462,  0.7924,  2.2701, -1.5094,
          -0.1973, -1.5919,  0.4869,  0.6739, -0.5242,  0.2559, -0.0149,
          -0.5332, -1.8313,  0.3598,  0.0804, -0.0780, -0.2930, -0.2844,
          -0.4752, -0.9919,  0.1809,  0.7622, -2.5069, -0.7724, -0.9441,
           1.6101,  0.6461, -0.8932,  0.0600,  0.6911,  0.5191, -0.1719,
          -0.5829, -0.9168,  1.5282,  1.4399,  0.3264, -0.8894,  0.2880,
          -0.0697,  0.8977, -0.5004,  0.3844,  0.0925,  0.5592, -0.1664,
           0.8575, -1.0348,  0.7326, -0.2124,  0.7533,  0.6270, -0.9559,
          -1.4159,  0.6788,  0.6163, -0.5951, -0.1403, -1.6088, -0.7731,
           0.3876,  1.0429, -2.0960,  0.1726,  1.7446, -0.3963,  0.0785,
          -0.4701,  1.0074,  0.3319, -2.2675, -1.6163, -0.4003, -0.5468,
           0.0452, -2.5586,  0.4747, -0.0271, -1.2161,  1.2121,  1.8738,
          -1.2207, -0.9218, -0.1430,  0.2512, -0.5236, -0.2544, -0.5868,
          -0.7086, -1.3328, -0.0243,  0.4759,  1.4125,  0.4947,  0.5054,
           1.6253,  0.4198, -0.9150,  0.6374,  0.4581,  1.1527,  1.4440,
          -0.0590, -0.4601,  0.2490, -0.5739,  0.6798, -0.2156, -1.1386,
          -0.5011, -0.7411,  0.2825, -0.2595,  0.8070,  0.5270,  0.2595,
          -0.1089,  0.4221, -0.7851,  0.7112, -0.3038,  0.6169, -0.1513,
          -0.5872,  0.3974,  0.2431,  0.4934, -0.9406, -0.9372,  1.4525,
           0.1376,  0.2558,  0.0661,  0.3509,  2.1667,  2.8428,  0.9429,
          -0.6143, -1.0969,  0.0955,  0.0914]]], device='cuda:0',
       grad_fn=<ViewBackward>)

Code
This code works,
rnn1 = nn.GRU(256, 128, 1)
input1 = torch.randn(100, 2, 256)
h01 = torch.randn(1, 2, 128)
output1, hn1 = rnn1(input1, h01)
print(input1.shape, h01.shape)
print(output1.shape, hn1.shape)

Output
torch.Size([100, 2, 256]) torch.Size([1, 2, 128])
torch.Size([100, 2, 128]) torch.Size([1, 2, 128])

Code
This code also works,
rnn1 = nn.GRU(256, 256)
input1 = torch.randn(1, 1, 256)
h01 = torch.randn(1, 1, 256)
output1, hn1 = rnn1(input1, h01)
print(input1.shape, h01.shape)
print(output1.shape, hn1.shape)

Output
torch.Size([1, 1, 256]) torch.Size([1, 1, 256])
torch.Size([1, 1, 256]) torch.Size([1, 1, 256])

Code
This does not work,
rnn1 = nn.GRU(256, 256)
input1 = torch.randn(1, 256)
#input1 = input1.view(1, 1, -1)
h01 = torch.randn(1, 1, 256)
output1, hn1 = rnn1(input1, h01)
print(input1.shape, h01.shape)
print(output1.shape, hn1.shape)

Output
RuntimeError: input must have 3 dimensions, got 2


Instruction: Equivalent of embeddings_regularizer in pyTorch

Input: tf.keras.layers.Embedding has parameter embeddings_regularizer. What would be equivalent of this in pyTorch or nn.Embedding?

Output:
There is no direct equivalent for PyTorch as PyTorch only supports L2 regularization on parameters via torch.optim optimizers.
For example torch.optim.SGD has weight_decay parameter. If you set it up and you optimize your nn.Embedding it will be regularized by L2 with specified strength (you can pass only nn.Embedding for weight_decay, see per-parameter-options of optimizers).
If you wish to use L1 regularization you would have to:

code it on your own
use available third party solutions

Coding on your own
Usually we add L1 regularization to loss and backpropagate, but this is inefficient approach.
It is better to populate gradients of our parameters (there are some edge cases though) with the derivative of regularization (for L1 is it sign value). Something along those lines:
import torch

# Do this in your optimization loop AT THE TOP
embedding = torch.nn.Embedding(150, 100)
embedding.weight.grad = torch.sign(embedding.weight)

# Do the rest of optimization AND clear gradients!
...

Though it is a little harder to make it work in general (stuff like batch accumulation etc.) and pretty unclear IMO. You could apply L2 on top of that also.
torchlayers third party library
Disclaimer: I'm the author of this project
You can install torchlayers-nightly and get per-layer L1 and L2 regularization.
Install via pip:
pip install -U torchlayers-nightly

In your code you could do:
import torchlayers as tl
import torch

embedding = torch.nn.Embedding(150, 100)
regularized_embedding = tl.L1(embedding)

# Do the rest as per usual

This feature is experimental for now, but should work and I've used it with success previously.
Also, you should be able to use tl.L2 the same way, see docstrings about usage of this particular layer.
For more info check github repository and read documentation here.