Instruction: evaluation about K-fold cross validation

Input:
After K-fold cross validation, which evaluation metric was averaged?
Precision and recall, or F-measure?
import pandas as pd
import numpy as np
from sklearn.model_selection import KFold
KFold(n_splits=2, random_state=None, shuffle=False)

Output:
The sklearn.model_selection.KFold function is a utility that provides the folds but does not actually perform k-fold validation. You have to implement this yourself!
See documentation description:

Provides train/test indices to split data in train/test sets. Split
dataset into k consecutive folds (without shuffling by default).
Each fold is then used once as a validation while the k - 1 remaining
folds form the training set.


Instruction: IsADirectoryError when loading my pytorch model with load_from_checkpoint

Input:
Could someone please explain to me why this function:
def train_graph_classifier(model_name, **model_kwargs):
  pl.seed_everything(42)

  # Create a PyTorch Lightning trainer with the generation callback
  root_dir = os.path.join('/home/predictor2', "GraphLevel" + model_name)
  os.makedirs(root_dir, exist_ok=True)
  trainer = pl.Trainer(default_root_dir=root_dir,
                     callbacks=[ModelCheckpoint(save_weights_only=True, mode="max", monitor="val_acc")],
                     gpus=1 if str(device).startswith("cuda") else 0,
                     max_epochs=500,
                     progress_bar_refresh_rate=0)
  trainer.logger._default_hp_metric = None # Optional logging argument that we don't need

  # Check whether pretrained model exists. If yes, load it and skip training
  pretrained_filename = os.path.join('/home/predictor2', f"GraphLevel{model_name}.ckpt")
  if os.path.isfile(pretrained_filename):
    print("Found pretrained model, loading...")
    model = GraphLevelGNN.load_from_checkpoint(pretrained_filename)
  else:
    pl.seed_everything(42)
    model = GraphLevelGNN(c_in=dataset.num_node_features,
                          c_out=1 if dataset.num_classes==2 else dataset.num_classes,  #change
                          **model_kwargs)
    trainer.fit(model, graph_train_loader, graph_val_loader)
    model = GraphLevelGNN.load_from_checkpoint(trainer.checkpoint_callback.best_model_path)

  # Test best model on validation and test set
  train_result = trainer.test(model, graph_train_loader, verbose=False)
  test_result = trainer.test(model, graph_test_loader, verbose=False)
  result = {"test": test_result[0]['test_acc'], "train": train_result[0]['test_acc']}
  return model, result

Returns the error:
Traceback (most recent call last):
  File "stability_v3_alternative_net.py", line 604, in <module>
    dp_rate=0.2)
  File "stability_v3_alternative_net.py", line 591, in train_graph_classifier
    model = GraphLevelGNN.load_from_checkpoint(trainer.checkpoint_callback.best_model_path)
  File "/root/miniconda3/lib/python3.7/site-packages/pytorch_lightning/core/saving.py", line 139, in load_from_checkpoint
    checkpoint = pl_load(checkpoint_path, map_location=lambda storage, loc: storage)
  File "/root/miniconda3/lib/python3.7/site-packages/pytorch_lightning/utilities/cloud_io.py", line 46, in load
    with fs.open(path_or_url, "rb") as f:
  File "/root/miniconda3/lib/python3.7/site-packages/fsspec/spec.py", line 1043, in open
    **kwargs,
  File "/root/miniconda3/lib/python3.7/site-packages/fsspec/implementations/local.py", line 159, in _open
    return LocalFileOpener(path, mode, fs=self, **kwargs)
  File "/root/miniconda3/lib/python3.7/site-packages/fsspec/implementations/local.py", line 254, in __init__
    self._open()
  File "/root/miniconda3/lib/python3.7/site-packages/fsspec/implementations/local.py", line 259, in _open
    self.f = open(self.path, mode=self.mode)
IsADirectoryError: [Errno 21] Is a directory: '/home/predictor'

where /home/predictor is the current directory i'm working in? (I made predictor2 directory because I get the same error when I replace predictor2 with predictor in the above code).
I understand that it's telling me that it's trying to write a file or something but it's finding that the location in a directory, I can get that from seeing other people's answers. But I can't see specifically here what the issue is becaues I don't name my working directory anywhere? The code was taken from this example.

Output:
This:
model=GraphLevelGNN.load_from_checkpoint(trainer.checkpoint_callback.best_model_path)

is failing because you are trying to open a directory and not a file. Check that trainer.checkpoint_callback.best_model_path actually is the path to your .ckpt. As it looks like it is not, then you need to figure out why your callback isn't storing the right path. You can of course hardcode it yourself for an ugly solution.


Instruction: How does `torch.gather()` construct new tensors?

Input:
I'm struggling to understand what the function torch.gather is doing.
If I have:
a = torch.tensor([[4,5,6],[7,8,9],[10,11,12]])
b = torch.tensor([[1,1],[1,2]])

Then
>>> torch.gather(input=a, dim=1, index=b)
tensor([[5, 5],
        [8, 9]])

While
>>> torch.gather(input=a, dim=0, index=b)
tensor([[ 7,  8],
        [ 7, 11]])

Can somebody explain very simply how these output tensors are actually constructed?

Output:
On the first call you index a with b along dim=1 (2nd dimension). The performed operation is:
out[i,j] = a[i, b[i,j]]

Which returns:
[[ a[0, b[0,0]], a[0, b[0,1]] ],
 [ a[1, b[1,0]], a[1, b[1,1]] ]]

While on the second call, you index a with b along dim=0 (1st dimension). This time around:
out[i,j] = a[b[i,j], j]

Which returns:
[[ a[b[0,0], 0], a[b[0,1], 1] ],
 [ a[b[1,0], 0], a[b[1,1], 1] ]]

For more explanations, read on with this thread.


Instruction: Reshaping with asterisk operator `*` in PyTorch

Input:
While reading this annotated implementation of Diffusion Probabilistic models in PyTorch, I got stuck at understanding this function
def extract(a, t, x_shape):
    batch_size = t.shape[0]
    out = a.gather(-1, t.cpu())
    return out.reshape(batch_size, *((1,) * (len(x_shape) - 1))).to(t.device)

What it's not clear it's the final return statement, what does the *((1,) mean into reshape function? Does that asterisk correspond to the unpacking operator? And if yes, how is it used here?

Output:
(1,) * (len(x_shape) - 1))

means to create a tuple with length len(x_shape) - 1 filled with just 1s
*(...)

means to spread the tuple into arguments
So it ends up being (say len(x_shape) == 5)
return out.reshape(batch_size, 1, 1, 1, 1).to(t.device)


Instruction: Get min value and index (of min value) from torch.tensor shape=(x,1)

Input:
I want to get a tensor of the min value and its index from a tensor. I have tried torch.min but am unsure how the return works to get BOTH value and index of minimum.
>>> a = torch.tensor([[10], [5], [8], [2], [8]])

# Ideal output
>>> min_value
[2]
>>> min_index
[3]

How can I achieve this using torch.min?

Output:
Here is code example of torch.min usage which returns named tuple with both values and indices of min values. It may have multiple value/indices depending on input tensor shape and dim parameter.
>>> result = torch.min(a, dim=0, keepdim=False)
>>> result.values
tensor(2)
>>> result.indices
tensor(3)

or simply a.min(dim=0)


Instruction: Pytorch giving positional argument error while calling a method

Input:
Pytorch code is giving an error of missing position argument, while I have already given x as an input argument.
Code:
import torch.nn as nn
import torch.nn.functional as F

class Network(nn.Module):
    def __init__(self):
        super().__init__()
        self.hidden = nn.Linear(8, 5)
        self.output = nn.Linear(5, 1)

    def forward(self, x):
        x = 2*F.sigmoid(self.hidden(x))
        x = F.softmax(self.output(x), dim= 0)
        return x

x = torch.tensor([1.0, 2.0, 3.0, 4.0,5.0,6.0,7.0,8.0] , dtype = torch.float32)
f = Network()
print(f(x))

tensor([1.], grad_fn=)

Network.forward(x)

--------------------------------------------------------------------------- TypeError                                 Traceback (most recent call
last) Input In [98], in <cell line: 1>()
----> 1 Network.forward(x)
TypeError: forward() missing 1 required positional argument: 'x'

Output:
Network.forward(x) - in this line you are calling method using class, not instance. It requires 2 parameters in this case: self and x.
There is no need to call forward method directly.
The following lines perform forward call implicitly and it is a proper way of using it.
f = Network()
print(f(x))

UPD
f = Network()
network_output = f(x)  # <-- this will perform `forward` method indirectly

It is described here: torch.nn.Module

Although the recipe for forward pass needs to be defined within this
function, one should call the Module instance afterwards instead of
this since the former takes care of running the registered hooks while
the latter silently ignores them.

Instead of f.forward(x) you have to do just f(x). Otherwise module functionality will be incomplete because you will not get any hooks registered for the given module. It is so for all modules in PyTorch. For example, you used self.hidden(x) in your code, not self.hidden.forward(x).


Instruction: TypeError: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first;'

Input:
I have the error being displayed whilst trying to plot the graph...
I am sharing the code in the following link:
https://colab.research.google.com/drive/1PooWIPVhm67iZquqZvxz3mdfmd6rv-3d#scrollTo=qSM7mNrKhBOt
I think I'm missing 'tensor.cpu()' somewhere but I can't really pinpoint it.. Everything else works :/ Can anyone help please?
def train_epoch(
  model,
  data_loader,
  loss_fn,
  optimizer,
  device,
  scheduler,
  n_examples
):
  model = model.train()
  losses = []
  correct_predictions = 0
  for d in data_loader:
    input_ids = d["input_ids"].to(device)
    attention_mask = d["attention_mask"].to(device)
    targets = d["targets"].to(device)
    outputs = model(
      input_ids=input_ids,
      attention_mask=attention_mask
    )
    _, preds = torch.max(outputs, dim=1)
    loss = loss_fn(outputs, targets)
    correct_predictions += torch.sum(preds == targets)
    losses.append(loss.item())
    loss.backward()
    nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
    optimizer.step()
    scheduler.step()
    optimizer.zero_grad()
  return correct_predictions.double() / n_examples, np.mean(losses)

def eval_model(model, data_loader, loss_fn, device, n_examples):
  model = model.eval()
  losses = []
  correct_predictions = 0
  with torch.no_grad():
    for d in data_loader:
      input_ids = d["input_ids"].to(device)
      attention_mask = d["attention_mask"].to(device)
      targets = d["targets"].to(device)
      outputs = model(
        input_ids=input_ids,
        attention_mask=attention_mask
      )
      _, preds = torch.max(outputs, dim=1)
      loss = loss_fn(outputs, targets)
      correct_predictions += torch.sum(preds == targets)
      losses.append(loss.item())
  return correct_predictions.double() / n_examples, np.mean(losses)

%%time
history = defaultdict(list)
best_accuracy = 0
for epoch in range(EPOCHS):
  print(f'Epoch {epoch + 1}/{EPOCHS}')
  print('-' * 10)
  train_acc, train_loss = train_epoch(
    model,
    train_data_loader,
    loss_fn,
    optimizer,
    device,
    scheduler,
    len(df_train)
  )
  print(f'Train loss {train_loss} accuracy {train_acc}')
  val_acc, val_loss = eval_model(
    model,
    val_data_loader,
    loss_fn,
    device,
    len(df_val)
  )
  print(f'Val   loss {val_loss} accuracy {val_acc}')
  print()
  history['train_acc'].append(train_acc)
  history['train_loss'].append(train_loss)
  history['val_acc'].append(val_acc)
  history['val_loss'].append(val_loss)
  if val_acc > best_accuracy:
    torch.save(model.state_dict(), 'best_model_state.bin')
    best_accuracy = val_acc

plt.plot(history['train_acc'], label='train accuracy')
plt.plot(history['val_acc'], label='validation accuracy')

plt.title('Training history')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend()
plt.ylim([0, 1]);

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
/usr/local/lib/python3.7/dist-packages/matplotlib/cbook/__init__.py in index_of(y)
   1626     try:
-> 1627         return y.index.values, y.values
   1628     except AttributeError:

AttributeError: 'builtin_function_or_method' object has no attribute 'values'

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
8 frames
<__array_function__ internals> in atleast_1d(*args, **kwargs)

/usr/local/lib/python3.7/dist-packages/torch/_tensor.py in __array__(self, dtype)
    730             return handle_torch_function(Tensor.__array__, (self,), self, dtype=dtype)
    731         if dtype is None:
--> 732             return self.numpy()
    733         else:
    734             return self.numpy().astype(dtype, copy=False)

TypeError: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.

Output:
When you are computing the number of correct predictions: correct_predictions += torch.sum(preds == targets), both preds and targets are CUDA tensors, which matplotlib knows nothing about.
In this case, we should detach the tensor (to stop Autograd tracking it), push the data from GPU to CPU, and convert it to numpy elements, like so: torch.sum(preds == targets).detach().cpu().numpy().
Further, since the number of correct predictions is just a single number, we can just do torch.sum(preds == targets).item() which is a shorthand for the above, but only if the tensor is a singleton.
This way, correct_predictions is a Python integer, & you can return float(correct_predictions) / n_examples from your methods and pass them onto matplotlib.
For further reading:

https://discuss.pytorch.org/t/what-is-the-difference-between-loss-and-loss-item/126083
https://discuss.pytorch.org/t/does-item-automatically-move-the-data-to-the-cpu/69629/6
https://discuss.pytorch.org/t/difference-between-loss-item-and-loss-detach-cpu-numpy/127773


Instruction: Slice 3d-tensor-based dataset into smaller tensor lengths

Input:
I have a dataset for training networks, formed out of two tensors, my features and my labels. The shape of my demonstration set is [351, 4, 34] for features, and [351] for labels.
Now, I would like to re-shape the dataset into chunks of size k (ideally while loading data with DataLoader), to obtain a new demonstration set for features of shape [351 * n, 4, k] and the corresponding label shape [351 * n], with n = floor(34 / k). The main aim is to reduce the length of each feature, to decrease the size of my network afterwards.
As written example: Starting from
t = [[1, 2, 3, 4],
     [5, 6, 7, 8]]

i.e. a [2, 4]-tensor, with
l = [1, 0]

as labels, I would like to be able to go to (with k = 2)
t = [[1, 2],
     [3, 4],
     [5, 6],
     [7, 8]]
l = [1, 1, 0, 0]

or to (with k = 3)
t = [[1, 2, 3],
     [5, 6, 7]]
l = [1, 0]

I found some solutions for reshaping one of the tensors (by using variations of split()), but then I would have to transfer that to my other tensor, too, and therefore I'd prefer solutions inside my DataLoader instead.
Is that possible?

Output:
You can reshape the input to the desired shape (first dimension is n times longer) while the label can be repeated with torch.repeat_interleave.
def split(x, y, k=2):
    n = floor(x.size(1) / k)
    x_ = x.reshape(len(x)*n, -1)[:,:k]
    y_ = y.repeat_interleave(len(x_)//len(y))
    return x_, y_

You can test it like so:
>>> split(t, l, k=2)
(tensor([[1, 2],
         [3, 4],
         [5, 6],
         [7, 8]]), tensor([1, 1, 0, 0]))

>>> split(t, l, k=3)
(tensor([[1, 2, 3],
         [5, 6, 7]]), tensor([1, 0]))

I recommend doing this kind of processing in your dataset class.


Instruction: Finetuning LayoutLM on FUNSD-like dataset - index out of range in self

Input:
I'm experimenting with huggingface transformers to finetune microsoft/layoutlmv2-base-uncased through AutoModelForTokenClassification on my custom dataset that is similar to FUNSD (pre-processed and normalized). After a few iterations of training I get this error :
 Traceback (most recent call last):
  File "layoutlmV2/train.py", line 137, in <module>
    trainer.train()
  File "..../lib/python3.8/site-packages/transformers/trainer.py", line 1409, in train
    return inner_training_loop(
  File "..../lib/python3.8/site-packages/transformers/trainer.py", line 1651, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "..../lib/python3.8/site-packages/transformers/trainer.py", line 2345, in training_step
    loss = self.compute_loss(model, inputs)
  File "..../lib/python3.8/site-packages/transformers/trainer.py", line 2377, in compute_loss
    outputs = model(**inputs)
  File "..../lib/python3.8/site-packages/torch/nn/modules/module.py", line 1131, in _call_impl
    return forward_call(*input, **kwargs)
  File "..../lib/python3.8/site-packages/transformers/models/layoutlmv2/modeling_layoutlmv2.py", line 1228, in forward
    outputs = self.layoutlmv2(
  File "..../lib/python3.8/site-packages/torch/nn/modules/module.py", line 1131, in _call_impl
    return forward_call(*input, **kwargs)
  File "..../lib/python3.8/site-packages/transformers/models/layoutlmv2/modeling_layoutlmv2.py", line 902, in forward
    text_layout_emb = self._calc_text_embeddings(
  File "..../lib/python3.8/site-packages/transformers/models/layoutlmv2/modeling_layoutlmv2.py", line 753, in _calc_text_embeddings
    spatial_position_embeddings = self.embeddings._calc_spatial_position_embeddings(bbox)
  File "..../lib/python3.8/site-packages/transformers/models/layoutlmv2/modeling_layoutlmv2.py", line 93, in _calc_spatial_position_embeddings
    h_position_embeddings = self.h_position_embeddings(bbox[:, :, 3] - bbox[:, :, 1])
  File "..../lib/python3.8/site-packages/torch/nn/modules/module.py", line 1131, in _call_impl
    return forward_call(*input, **kwargs)
  File "..../lib/python3.8/site-packages/torch/nn/modules/sparse.py", line 158, in forward
    return F.embedding(
  File "..../lib/python3.8/site-packages/torch/nn/functional.py", line 2203, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
IndexError: index out of range in self

After further inspection (vocab size, bboxes, dimensions, classes...) I noticed that there's negative values inside the input tensor causing the error. While input tensors of successful previous iterations have unsigned integers only. These negative numbers are returned by _calc_spatial_position_embeddings(self, bbox) in modeling_layoutlmv2.py
line 92 :
h_position_embeddings = self.h_position_embeddings(bbox[:, :, 3] - bbox[:, :, 1])

What may cause the returned input values to be negative?
What could I do to prevent this error from happening?

Example of the input tensor that triggers the error in torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse) :
tensor([[ 0, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
         11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
         11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
         11, 11, 11, 11, 11, 11, 11, 11, 11, 12, 12, 12, 12, 12, 11, 11, 11, 11,
         11, 11, 11, 11, 11, 11, 11, 11, 11,  9,  9,  9,  9,  9,  9,  9,  9,  9,
          9,  9,  9,  9,  9,  9,  9, 10, 10, 12, 12, 12, 12, 12, 12, 12, 12, 12,
         12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12,
         12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12,
         12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12,
         12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12,
         12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 11, 11, 11, 11,
         11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 12, 12, 12, 12, 12, 12, 12, 12,
         12, 12, 12, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 12, 12,
         12, 12, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
         11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
         11, 11, 11, 11, 11, 11, 11, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10,
         10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 12, 12, 12, 12,
         12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12,
         12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12,
         12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12,
         12, 12, 12, 12, 12, 12, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
         11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
         11, 11, 12, 12, 12, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
         11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 12, 12, 12, 12, 12, 12, 12,
         12, 12, 12, 12, 12, 12, 12, 12,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,
          8,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,
          8,  5,  5,  5,  5,  5,  5, -6, -6, -6, -6, -6, -6,  1,  1,  1,  1,  1,
          5,  5,  5,  5,  5,  5,  7,  5,  7,  7,  0,  0,  0,  0,  0,  0,  0,  0,
          0,  0,  0,  0,  0,  0,  0,  0]])

Output:
After double checking the dataset and specifically the coordinates of the labels, I've found that some rows bbox coordinates lead to zero width or height. Here's a simplified example:
x1, y1, x2, y2 = dataset_row["bbox"]
print((x2-x1 < 1) or (y2-y1 < 1)) #output is sometimes True

After removing these labels from the dataset, the issue was resolved.


Instruction: How to have CCC Loss Function in PyTorch?

Input: How would one write a custom loss function (in this case for CCC) for PyTorch that behaves similar to other losses like backwards()?

Output:
As long as you use PyTorch operators that are differentiable, your function will be too.
So, using torch.mean, torch.var, and torch.cov. Isn't that be what your looking for?
def CCCLoss(x, y):
    ccc = 2*torch.cov(x, y) / (x.var() + y.var() + (x.mean() - y.mean())**2)
    return ccc


Instruction: PyTorch ImageFolder Dataset conversion to 1d

Input:
I have a dataset with three labels.
First, I'm loading my data into a Dataset with the ImageFolder class and the CenterCrop transform.
So every picture has now three channels r, g, b with 224x224 values from 0..1. I thought my NN has 224×224×3 Input Nodes.
I get this error:
RuntimeError: mat1 and mat2 shapes cannot be multiplied (21504x224 and 150528x200)

Assuming that the first mat1 is the Image (I have no clue where the 21504 is coming from)  and the second one is the first nn.Linear(2242243, 200) Layer, I can see why they cannot be multiplied. So, I changed the nn.Linear(2242243) to nn.Linear(224). Now it works, but..
transform = transforms.Compose([transforms.Resize((300, 200)),
                                transforms.CenterCrop(224),
                                transforms.ToTensor()])

dataset = datasets.ImageFolder('/content/drive/MyDrive/Colab Notebooks/data', transform=transform)

# ... BatchSize = 32
model = nn.Sequential(
        nn.Linear(224, 200), # <- The Size of one Row!
        nn.Sigmoid(),
        nn.Linear(200, 40),
        nn.Sigmoid(),
        nn.Dropout(p=0.2),
        nn.Linear(40, 40),
        nn.Sigmoid(),
        nn.Linear(40, 20),
        nn.Sigmoid(),
        nn.Linear(20, 3)
    ).to(device)
# ...
prediction = model(x)

For some Reaseon my prediction has now this from.
prediction[32][3][224]

I would expect 32 items in a list for a Batch. Every Item from the Batch contains 3 values with the probability what the label is. But why does the Height / width came up here?.
I think I have to change the Format of the dataset from 224x224x3 (3d) to 150528 1d, but operations like view() did not work, probably because of the lazy loading from ImageFolder.
The NN works fine with the MINST dataset. (28x28) In and (10) Out. So my guess is: The dataset has to be transformed, but I can't figure out how.

Output:
It can work if you apply transforms.Lambda and change transforms in the following way:
transform = transforms.Compose([transforms.Resize((300, 200)),
                                transforms.CenterCrop(224),
                                transforms.ToTensor(),
                                transforms.Lambda(lambda x: torch.flatten(x))])

As a result you will have (batch_size, 150528) shape for each batch which means that you need to specify 150528 input size for the first linear layer. Not sure that it's reasonable to classify images with linear fully-connected layers in such a case. It will work slowly and unlikely converge to some good predictions. You'd better consider using convolution layers instead.


Instruction: Is there a PyTorch lightning equivalent for Tensorflow?

Input:
I saw PyTorch Lightning advertised as PyTorch but for people who don't want to worry so much about the underlying methodology. This narrative is on the PyTorch lightning website but also here for example.
For hardware reasons, does something similar exist for TensorFlow? I have a code example for neural nets here written in PyTorch and PyTorch Lightning but am not sure how to rewrite it in TensorFlow.

Output:
Probably the best association would be Keras (formerly separate from but now for some time integrated in TF - you can you Keras as a high level API).
Note that you can also use tensorflow_addons (I personally enjoy working with it) package and other libraries&wrappers that come into the aid of TensorFlow, because since Keras is integrated into TF, you will be also very likely to use them on your Keras code.


Instruction: RuntimeError: the derivative for ‘target’ is not implemented for Auto Encoder

Input:
I am getting the following error - RuntimeError: the derivative for ‘target’ is not implemented
I did have a look at similar posts however they are different from my problem. I’m trying to code an Auto Encoder from scratch. Here’s my code -
import torch
import torchvision
import torchvision.transforms as transforms
import matplotlib.pyplot as plt
from torch import nn
from torchviz import make_dot

trainset = torchvision.datasets.FashionMNIST(root='./data', train=True,
                                        download=True)
data = trainset.data.float()
# print(trainset.data.shape)
# plt.imshow(trainset.data[9])
# plt.show()
device = "cuda" if torch.cuda.is_available() else "cpu"
data = data.to(device)
print(f"Using device = {device}")
class NeuralNetwork(nn.Module):
    def __init__(self):
        super(NeuralNetwork, self).__init__()
        self.flatten = nn.Flatten()
        self.encode = nn.Sequential(
            nn.Linear(28*28, 512),
            nn.ReLU(),
            nn.Linear(512, 30),
            nn.ReLU()
        )
        self.decode = nn.Sequential(
            nn.Linear(30, 512),
            nn.ReLU(),
            nn.Linear(512, 28*28),
            nn.ReLU()
        )

    def forward(self, x):
        x = self.flatten(x)
        encoded = self.encode(x)
        decoded = self.decode(encoded)
        return decoded

model = NeuralNetwork().to(device)
# print(model)

lossFn  = nn.BCELoss()

optimizer = torch.optim.SGD(model.parameters(), lr = 1e-3)

for epoch in range(1000):
    optimizer.zero_grad()
    outputs = model(data)
    loss = lossFn(outputs, outputs)
    loss.backward()
    optimizer.step()
    model.forward(data)

Output:
The reason you're having this error message is because you are calling a function which expects its argument to not require gradient computation. More specifically, nn.BCELoss expects the target (2nd argument) to not require gradient.
You can fix this by detaching the argument from the graph:
lossFn(outputs, outputs.detach())


Instruction: TypeError: ‘tuple’ object is not callable when using flatten layer

Input:
I Have the following model:
import torch
import torch.nn as nn
import torch.nn.functional as F

class MyModel(nn.Module):

def __init__(self, input_size, num_classes):
    super(MyModel, self).__init__()

    self.layer_1      = nn.Conv1d(1,   16,  3, bias=False, stride=2)
    self.activation_1 = F.relu
    self.adap         = nn.AdaptiveAvgPool1d(1)
    self.flatten      = nn.Flatten          (),

    self.layer_2      = torch.nn.Linear(2249, 500)
    self.activation_2 = F.relu
    self.layer_3      = torch.nn.Linear(500, 2)

    pass

def forward(self, x, labels=None):

    x = x.reshape(256, 1, -1)
    x = self.layer_1(x)
    x = self.activation_1(x)
    x = self.flatten(x)
    return x

When running torchinfo
model = MyModel(input_size=4500, num_classes=2)
torchinfo.summary(model, (256, 4500))

I’m Getting error:
Input In [101], in MyModel.forward(self, x, labels)
30 x = self.activation_1(x)
—> 31 x = self.flatten(x)
32 return x

TypeError: ‘tuple’ object is not callable

What is wrong ?
What do I need to change ?

Output:
You have , at the end of the flatten line.
please remove it
import torch
import torch.nn as nn
import torch.nn.functional as F

class MyModel(nn.Module):

    def __init__(self, input_size, num_classes):
        super(MyModel, self).__init__()

        self.layer_1      = nn.Conv1d(1,   16,  3, bias=False, stride=2)
        self.activation_1 = F.relu
        self.adap         = nn.AdaptiveAvgPool1d(1)
        self.flatten      = nn.Flatten()

        self.layer_2      = torch.nn.Linear(2249, 500)
        self.activation_2 = F.relu
        self.layer_3      = torch.nn.Linear(500, 2)

        pass

    def forward(self, x, labels=None):

        x = x.reshape(256, 1, -1)
        x = self.layer_1(x)
        x = self.activation_1(x)
        x = self.flatten(x)
        return x


Instruction: A torch version compatibility error is occurred when using Conv2d

Input:
I want to use python 2.7 and torch, I am using torch 1.4.0. When I started implemneting a Conv2d example such as this example
c = nn.Conv2d(1, 1, (2, 1), stride=1)
x = torch.rand(1, 4).unsqueeze(-1)

I can execute these two line in python 2.7 and python 3, However when I call the c layer for a forward propagation it only works in python 3.
y = c(x)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/larc2/.local/lib/python2.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/larc2/.local/lib/python2.7/site-packages/torch/nn/modules/conv.py", line 343, in forward
    return self.conv2d_forward(input, self.weight)
  File "/home/larc2/.local/lib/python2.7/site-packages/torch/nn/modules/conv.py", line 340, in conv2d_forward
    self.padding, self.dilation, self.groups)
RuntimeError: Expected 4-dimensional input for 4-dimensional weight 1 1 2, but got 3-dimensional input of size [1, 4, 1] instead

Whereas, the python 3 result for this exact same code is as follows:
tensor([[[-0.9926],
         [-0.6937],
         [-0.6704]]], grad_fn=<SqueezeBackward1>)

Output:
As the error writes you need a 4-D input for a 2-D Conv: (N,C,H,W)
You are giving a 3-D input.
Please use 4-D tensor as the Docs ask: https://pytorch.org/docs/1.4.0/nn.html#torch.nn.Conv2d
The reason it works in python 3 is probably that PyTorch adds support for using a 3-D tensor in 2-D Conv (C,H,W) in a newer version and maybe you installed a newer PyTorch version in the python 3 env


Instruction: Formatting our data into PyTorch Dataset object for fine-tuning BERT

Input:
I'm using an already existing code from Towards Data Science for fine-tuning a BERT Model.
The problem I'm facing belongs to this part of the code which where try to format our data into a PyTorch data.Dataset object:
class MeditationsDataset(torch.utils.data.Dataset):
    def _init_(self, encodings, *args, **kwargs):
        self.encodings = encodings
    def _getitem_(self, idx):
        return {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}
    def _len_(self):
        return len(self.encodings.input_ids)

dataset = MeditationsDataset(inputs)

When I run the code, I face this error:
TypeError                                 Traceback (most recent call last)
<ipython-input-144-41fc3213bc25> in <module>()
----> 1 dataset = MeditationsDataset(inputs)

/usr/lib/python3.7/typing.py in __new__(cls, *args, **kwds)
    819             obj = super().__new__(cls)
    820         else:
--> 821             obj = super().__new__(cls, *args, **kwds)
    822         return obj
    823

TypeError: object.__new__() takes exactly one argument (the type to instantiate)

I already searched for this error but the problem here is that sadly I'm not familiar with either PyTorch or OOP so I couldn't fix this problem. Could you please let me know what should I add or remove from this code so I can run it? Thanks a lot in advance.
Also if needed, our data is as below:
{'input_ids': tensor([[   2, 1021, 1005,  ...,    0,    0,    0],
                      [   2, 1021, 1005,  ...,    0,    0,    0],
                      [   2, 1021, 1005,  ...,    0,    0,    0],
                      ...,
                      [   2, 1021, 1005,  ...,    0,    0,    0],
                      [   2,  103, 1005,  ...,    0,    0,    0],
                      [   2,    4,    0,  ...,    0,    0,    0]]),
 'token_type_ids': tensor([[0, 0, 0,  ..., 0, 0, 0],
                           [0, 0, 0,  ..., 0, 0, 0],
                           [0, 0, 0,  ..., 0, 0, 0],
                           ...,
                           [0, 0, 0,  ..., 0, 0, 0],
                           [0, 0, 0,  ..., 0, 0, 0],
                           [0, 0, 0,  ..., 0, 0, 0]]),
 'attention_mask': tensor([[1, 1, 1,  ..., 0, 0, 0],
                           [1, 1, 1,  ..., 0, 0, 0],
                           [1, 1, 1,  ..., 0, 0, 0],
                           ...,
                           [1, 1, 1,  ..., 0, 0, 0],
                           [1, 1, 1,  ..., 0, 0, 0],
                           [1, 1, 0,  ..., 0, 0, 0]]),
 'labels': tensor([[   2, 1021, 1005,  ...,    0,    0,    0],
                   [   2, 1021, 1005,  ...,    0,    0,    0],
                   [   2, 1021, 1005,  ...,    0,    0,    0],
                   ...,
                   [   2, 1021, 1005,  ...,    0,    0,    0],
                   [   2, 1021, 1005,  ...,    0,    0,    0],
                   [   2,    4,    0,  ...,    0,    0,    0]])}

Output:
Special functions in Python use double underscores prefix and suffix. In your case, to implement a data.Dataset, you must have __init__, __getitem__, and __len__:
class MeditationsDataset(torch.utils.data.Dataset):
    def __init__(self, encodings, *args, **kwargs):
        self.encodings = encodings
    def __getitem__(self, idx):
        return {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}
    def __len__(self):
        return len(self.encodings.input_ids)


Instruction: Create a PyTorch tensor of sequences which excludes specified value

Input:
I have a 1d PyTorch tensor containing integers between 0 and n-1. Now I need to create a 2d PyTorch tensor with n-1 columns, where each row is a sequence from 0 to n-1 excluding the value in the first tensor. How can I achieve this efficiently?
Ex:
n = 3
a = torch.Tensor([0, 1, 2, 1, 2, 0])
# desired output
b = [
    [1, 2],
    [0, 2],
    [0, 1],
    [0, 2],
    [0, 1],
    [1, 2]
  ]

Typically, the a.numel() >> n.
Detailed Explanation:
The first element of a is 0, hence it has to map to the sequence [0, 1, 2] excluding 0, which is [1, 2].
Similarly, the second element of a is 1, hence it has to map to [0, 2] and so on.
PS: I actually have an additional batch dimension, which I've excluded here for simplicity. Hence, I need the solution to be easily extendable to one additional dimension.

Output:
We can construct a tensor with the desired sequences and index with tensor a.
import torch

n = 3
a = torch.Tensor([0, 1, 2, 1, 2, 0]) # using torch.tensor is recommended

def exclude_gather(a, n):
    sequences = torch.nonzero(torch.arange(n) != torch.arange(n)[:,None], as_tuple=True)[1].reshape(-1, n-1)
    return sequences[a.long()]

exclude_gather(a, n)

Output
tensor([[1, 2],
        [0, 2],
        [0, 1],
        [0, 2],
        [0, 1],
        [1, 2]])

We can add a batch dimension with functorch.vmap
from functorch import vmap

n = 4
b = torch.Tensor([[0, 1, 2, 1, 3, 0],[0, 3, 1, 0, 2, 1]])

vmap(exclude_gather, in_dims=(0, None))(b, n)

Output
tensor([[[1, 2, 3],
         [0, 2, 3],
         [0, 1, 3],
         [0, 2, 3],
         [0, 1, 2],
         [1, 2, 3]],

        [[1, 2, 3],
         [0, 1, 2],
         [0, 2, 3],
         [1, 2, 3],
         [0, 1, 3],
         [0, 2, 3]]])


Instruction: AttributeError: 'Tensor' object has no attribute 'close'

Input:
this is the code I am working with:
I searched for a similiar error do i need to change the .close() ?
Can anyone help please?

Output:
You should remove the close() function call on the parameter tensor image in line -
image = tensor.cpu().close().detach().numpy(). This should be replaced with - image = tensor.cpu().detach().numpy()


Instruction: How to exclude loss computation on certain tensors in PyTorch?

Input:
I am trying to do pose-conditioned face generation. The idea is to use an auto-encoding network that takes an image as an input and a conditional vector containing the pose information, such that the generated image is conditioned on the conditional vector.
For computing the loss using MSE, I am generating the face landmarks of the predicted image using a face-landmark detection library. Unfortunately, during the early epochs, the network produces garble and the face-landmark detection library returns None instead of an expected tensor of the form 256 x 256 x 3 where a pixel value indicates the presence of a landmark.
What I would like to do is to ignore computing the loss when no face has been detected.
Example -- assume that my batch is of the form -> 10 x 256 x 256 x 3, where 10 is the batch_size and 256x256 is the dimension of the image with 3 channels. For the predictions, let's assume that no landmarks could be generated for 3 images in the batch. I could set the prediction tensor for which no face landmarks could be generated to NaN values and the predicted landmarks would have the form - 10 x 256 x 256 x 3. In my MSE loss function, I would like to ignore gradient computation originating from tensor containing the NaN values. To make life simple, I want to ignore those individual 3 tensors that had all NaN values.
Any help would be appreciated. I have a backup with a for loop but that is sub-optimal.
This is some sample code -
import numpy as np
import torch
from torchvision import transforms
import random

transform = transforms.Compose(
        [
            transforms.ToPILImage(),
            transforms.CenterCrop(size),
            transforms.ToTensor(),
        ]
    )

image_tensors = torch.randn(10, 3, 256, 256)
tensor_meshes = list()
for image_tensor in image_tensors:
    image_array = image_tensor.detach().cpu().numpy()

    image_landmarks = generate_mesh_from_image(image_array) # returns either None or landmark image of dimension -> 3 x 256 x 256

    if image_landmarks is None:
        image_landmarks = np.empty(3, 256, 256) # creates np matrix with Nan values

    # convert to tensor
    landmark_tensor = transform(image_landmarks)

    tensor_meshes.append(landmark_tensor)

# convert tensor_meshes to a torch tensor of dimension -> 10 x 3 x 256 x 256
## SAMPLE CODE ##

# some dummy code to simulate generating landmark images
def generate_mesh_from_image(image_array):
    rand = random.randint(0, 1)
    if rand == 0:
        return None
    else:
        return np.random.randn(3, 256, 256)

The loss now needs to be computed between the prediction tensor (10 x 3 x 256 x 256) and the ground truth tensor (10 x 3 x 256 x 256). However, the prediction tensor contains some tensor elements that have all NaN values which I would like ignore during the loss computation.

Output: You can create a mask containing ones for images with landmarks and zeros for those without. Then simply compute the loss on the whole batch and apply the mask afterwards, before performing back propagation.


Instruction: Why is my autoencoder not learning the FMNIST dataset?

Input:
I am using a simple autoencoder to learn images from the FashionMnist dataset. I have preprocessed the dataset by grayscaling and normalizing it. I did not make the network too deep, to prevent it from creating a direct mapping.
Here's my PyTorch code -
import torch
import torchvision as tv
import torchvision.transforms as transforms
import matplotlib.pyplot as plt
from torch import nn
import os
from torchviz import make_dot
transforms = tv.transforms.Compose([tv.transforms.Grayscale(num_output_channels=1)])
trainset = tv.datasets.FashionMNIST(root='./data', train=True,
                                        download=True, transform=transforms)
PATH = './ae.pth'
data = trainset.data.float()
data = data/255
# print(trainset.data.shape)
plt.imshow(trainset.data[0], cmap = 'gray')
plt.show()

class NeuralNetwork(nn.Module):
    def __init__(self):
        super(NeuralNetwork, self).__init__()
        self.flatten = nn.Flatten()
        self.encode = nn.Sequential(
            nn.Linear(28*28, 512),
            nn.ReLU(),
            nn.Linear(512, 30),
            nn.ReLU()
        )
        self.decode = nn.Sequential(
            nn.Linear(30, 512),
            nn.ReLU(),
            nn.Linear(512, 28*28),
            nn.Sigmoid()
        )

    def forward(self, x):
        x = self.flatten(x)
        encoded = self.encode(x)
        decoded = self.decode(encoded)
        return decoded
if(os.path.exists(PATH)):
    print("Loading data on cpu")
    device = torch.device('cpu')
    model = NeuralNetwork()
    model.load_state_dict(torch.load(PATH, map_location=device))

else:
    device = "cuda" if torch.cuda.is_available() else "cpu"
    data = data.to(device)
    print(f"Using device = {device}")
    model = NeuralNetwork().to(device)
    # print(model)

    lossFn  = nn.BCELoss()

    optimizer = torch.optim.SGD(model.parameters(), lr = 1e-3)

    for epoch in range(1000):
        print("Epoch = ", epoch)
        optimizer.zero_grad()
        outputs = model(data)
        loss = lossFn(outputs, data.reshape(-1, 784))
        loss.backward()
        optimizer.step()

    torch.save(model.state_dict(), PATH)
    data = data.to("cpu")
    model = model.to("cpu")

pred = model(data)
pred = pred.reshape(-1, 28, 28)
# print(pred.shape)
plt.imshow(pred.detach().numpy()[0], cmap = 'gray')
plt.show()

For testing, I am inputting the following image -

However, I get this as output -

Output:
I had an intuition that there was an issue with your loss function. When working with images, distance-based losses such as L1 or L2 losses work really well, as you are essentially measuring how far-away your predictions are from the ground-truth images. This was what I had observed as well, as the loss wasn't converging with BCE and it was rather oscillating.
I rewrote the entire thing and replaced BCE loss with MSE Loss and in just 50 epochs, the loss has gone down considerably, and it is still going down.
Here is the prediction after just 50 epochs -

The ground-truth image is -

I believe that you can get the loss down much more if you train for longer.
Here is the full code. I used a dataloader for batchifying and processing the data.
I also changed the transformations so that the resulting data is a torch tensor.
import torch
import torchvision as tv
import torchvision.transforms as transforms
import matplotlib.pyplot as plt
from torch import nn
from torch.utils.data import DataLoader

transforms = tv.transforms.Compose([
    transforms.Grayscale(num_output_channels=1),
    transforms.ToTensor()
])

trainset = tv.datasets.FashionMNIST(root='./data', train=True,
                                        download=True, transform=transforms)

loader = DataLoader(trainset, batch_size=32, num_workers=1, shuffle=True)

class NeuralNetwork(nn.Module):
    def __init__(self):
        super(NeuralNetwork, self).__init__()
        self.flatten = nn.Flatten()
        self.encode = nn.Sequential(
            nn.Linear(28*28, 512),
            nn.ReLU(),
            nn.Linear(512, 30),
            nn.ReLU()
        )
        self.decode = nn.Sequential(
            nn.Linear(30, 512),
            nn.ReLU(),
            nn.Linear(512, 28*28),
            nn.Sigmoid()
        )

    def forward(self, x):
        x = self.flatten(x)
        encoded = self.encode(x)
        decoded = self.decode(encoded)
        return decoded

model = NeuralNetwork().to(device)
lossFn  = nn.MSELoss()

optimizer = torch.optim.SGD(model.parameters(), lr = 1e-2)

epochs = 50
for epoch in range(epochs):
    for images, labels in loader:
        optimizer.zero_grad()
        images, labels = images.to(device), labels.to(device)
        outputs = model(images)
        loss = lossFn(outputs, images.reshape(-1, 28*28))
        loss.backward()
        optimizer.step()

    print(f'Loss : {loss.item()}')

    print(f'Epochs done : {epoch}')

Here is some inference code -
# infer on some test data
testset = tv.datasets.FashionMNIST(root='./data', train=False,
                                        download=False, transform=transforms)

testloader = DataLoader(testset, shuffle=False, batch_size=32, num_workers=1)

test_images, test_labels = next(iter(testloader))
test_images = test_images.to(device)
predictions = model(test_images)

prediction = predictions[0]
prediction = prediction.view(1, 28, 28)

prediction = prediction.detach().cpu().numpy()

prediction = prediction.transpose(1, 2, 0)

# plot the prediction
plt.imshow(prediction, cmap = 'gray')
plt.show()

# plot the actual image
test_image = test_images[0]
test_image = test_image.detach().cpu().numpy()
test_image = test_image.transpose(1, 2, 0)

plt.imshow(test_image, cmap='gray')
plt.show()

This is the loss going down --
Epochs done : 39
Loss : 0.04641226679086685
Epochs done : 40
Loss : 0.04445071145892143
Epochs done : 41
Loss : 0.05033266171813011
Epochs done : 42
Loss : 0.04813298210501671
Epochs done : 43
Loss : 0.0474831722676754
Epochs done : 44
Loss : 0.044186390936374664
Epochs done : 45
Loss : 0.049083154648542404
Epochs done : 46
Loss : 0.04645842686295509
Epochs done : 47
Loss : 0.04586248844861984
Epochs done : 48
Loss : 0.0467853844165802
Epochs done : 49


Instruction: How do I merge image back after splitting into blocks

Input:
I splitted an image into blocks with the code below
block = []
for x in range(0, 224,16):
    for y in range(0, 224,16):
        block.append(im[y:y+16, x:x+16])
np_block = np.array(block)

The image shape is 224,224,3 and each block is 16,16,3. How do I combine these blocks together to form the complete image?.
When I reshaped it (np_block.reshape(224, 224,3)), I got the below image which is completely different from the real image

Edit:
This has been answered in the comment section as thus:
reshaped_img = np_block.reshape(14,14,16,16,3).transpose(0,3,1,2,4).reshape(14*16,14*16,3)

The resulting image is rotated and also flipped compared to the original image. Here is the original image

And here is the resulting image from the provided method to merge the blocks together

Doing restored_img = np.flip(np.rot90(reshaped_img, k=1, axes=(0, 1)), axis=0) will correctly restore the resulting image to its original form.
However, converting restored_img into a tensor in pytorch, tensor_img = torch.from_numpy(restored_img)throws the following error:
ValueError: At least one stride in the given numpy array is negative, and tensors with negative strides are not currently supported. (You can probably work around this by making a copy of your array  with array.copy().)

I worked around this as suggested in the error message.
tensor_img = torch.from_numpy(restored_img.copy()) got rid of the error but the transformations (rotation and flipping) made on restored_img is not preserved. tensor_img stil remains the same as the last image above.
I noticed that the splitted blocks are not appended in the correct order when splitting the original image. This is evident in np_block when I visualized it (the first image above looks flipped and rotated). So, I changed the line block.append(im[y:y+16, x:x+16]) to block.append(im[x:x+16, y:y+16])
Revisualizing np_block consequently gives the following  image

This seem closer to the original image and the blocks also looks like they are in the correct order (this is what I've observed). But the blocks do not overlap with each other.
How do I work around this, please?

Output:
Hi there the problem with original solution proposed is that you have to rearange the blocks 0,3 and 1,2 if you swap those dimensions then it will be solved. check:
block = [] #create blocks
for x in range(0, 224,16):
    for y in range(0, 224,16):
        block.append(im[y:y+16, x:x+16])
np_block = np.array(block)

reshaped_img = np_block.reshape(14,14,16,16,3).transpose(1,2,0,3,4).reshape(14*16,14*16,3)
#check indices are now (1,2,0,3,4) instead of (0,3,1,2,4)

plt.imshow(np.hstack([im,reshaped_img]))

Edit
also from original comment this is just transpose output dimensions
reshaped_img.transpose(1,0), but if you can correct in the original reshape it's a better solution

Edit 2
using the corrected stacking (with x in first dimension) the code would be:
block = []
for x in range(0, 224,16):
    for y in range(0, 224,16):
        block.append(im[x:x+16, y:y+16])
np_block = np.array(block)

reshaped_img = np_block.reshape(14,14,16,16,3).transpose(0,2,1,3,4).reshape(14*16,14*16,3)

Here makes morse sence: the first dimension of reshape in block (dim 0) with the first dimension of the subblock (dim 2), are grouped, then the same for the seccond dimensions of the block and and subblock (1,3).
In any case you can check the good solution by:
plt.imshow(np.hstack([im,reshaped_img]))

Edit 3
Last edit, here a code to check the original block reshape (with code of edit 2). Here you can see every block has some sense of getting a cut in the image:

b = np_block.reshape(14,14,16,16,3)
plt.figure()
for i in range(14):
    for j in range(14):
        plt.subplot(14,14,14*i+j+1)
        plt.imshow(b[i,j])
        plt.axis('off')


Instruction: Implement multiprocessing to test two videos simultaneously in opencv for object detection

Input:
I am implementing an object detection model using a YOLO algorithm with PyTorch and OpenCV. Running my model on a single video works fine. But whenever I am trying to use multiprocessing for testing more videos at once it is freezing. Can you please explain what is wrong with this code ??
import torch
import cv2
import time
from multiprocessing import Process

model = torch.hub.load('ultralytics/yolov5', 'custom', path='runs/best.pt', force_reload=True)

def detectObject(video,name):
    cap = cv2.VideoCapture(video)
    while cap.isOpened():
        pTime = time.time()
        ret, img = cap.read()
        cTime = time.time()
        fps = str(int(1 / (cTime - pTime)))
        if img is None:
            break
        else:
            results = model(img)
            labels = results.xyxyn[0][:, -1].cpu().numpy()
            cord = results.xyxyn[0][:, :-1].cpu().numpy()
            n = len(labels)
            x_shape, y_shape = img.shape[1], img.shape[0]
            for i in range(n):
                row = cord[i]
                # If score is less than 0.3 we avoid making a prediction.
                if row[4] < 0.3:
                    continue
                x1 = int(row[0] * x_shape)
                y1 = int(row[1] * y_shape)
                x2 = int(row[2] * x_shape)
                y2 = int(row[3] * y_shape)
                bgr = (0, 255, 0)  # color of the box
                classes = model.names  # Get the name of label index
                label_font = cv2.FONT_HERSHEY_COMPLEX  # Font for the label.
                cv2.rectangle(img, (x1, y1), (x2, y2), bgr, 2)  # Plot the boxes
                cv2.putText(img, classes[int(labels[i])], (x1, y1), label_font, 2, bgr, 2)
                cv2.putText(img, f'FPS={fps}', (8, 70), label_font, 3, (100, 255, 0), 3, cv2.LINE_AA)

        img = cv2.resize(img, (700, 700))
        cv2.imshow(name, img)
        if cv2.waitKey(1) & 0xFF == ord('q'):
            break
    cap.release()

Videos = ['../Dataset/Test1.mp4','../Dataset/Test2.mp4']
for i in Videos:
    process = Process(target=detectObject, args=(i, str(i)))
    process.start()

Every time I run that code it freezes.
Here is the output :
Downloading: "https://github.com/ultralytics/yolov5/archive/master.zip" to /home/com/.cache/torch/hub/master.zip
YOLOv5  2022-6-27 Python-3.9.9 torch-1.11.0+cu102 CPU

Fusing layers...
YOLOv5s summary: 213 layers, 7023610 parameters, 0 gradients
Adding AutoShape...

Output:
I got it to work by adding torch multiprocessing code.
from torch.multiprocessing import Pool, Process, set_start_method
try:
     set_start_method('spawn', force=True)
except RuntimeError:
    pass

videos = ['videos/video1.mp4', 'videos/video2.mp4']
for i in videos:
    process = Process(target=detectObject, args=(i, str(i)))
    process.start()

I was able to run on multiple videos at once this way.


Instruction: Pytorch nn.MSELoss without specifying target

Input:
I was having difficulty with my loss getting stuck at a particular value. It would always decrease to a certain value, then stop decreasing. The code regarding the loss was:
criterion = nn.MSELoss()
loss = criterion(y_pred, y_batch.unsqueeze(1))

When I changed it to:
criterion = nn.MSELoss()
loss = criterion(y_pred, target=y_batch)

the issue was fixed.
What was happening before when the target was not specified? Does the target need to be specified for every Pytorch loss function? I found nothing in the documentation about target specifications.

Output: It looks like target is the name of the second positional argument, that's all. The only difference between the two lines is the unsqueezing of dim=1 on the second one.


Instruction: how to increase a size of image in DCGAN in python?

Input:
I am trying to generate artificial images in DCGANs of size 128x128. but in all of the examples on the internet, the generated image size is 64x64.  and I don't know where I am making a mistake.
here is a code. i have taken this code from inter and when I run this code for 64x64 image it run perfectly fine but for 128x128 image, i am getting error about this line code
---> 90         disc_fake = disc(fake.detach()).reshape(-1)
Discriminator and Generator implementation from DCGAN paper
import torch
import torch.nn as nn

class Discriminator(nn.Module):
    def __init__(self, channels_img, features_d):
        super(Discriminator, self).__init__()
        self.disc = nn.Sequential(
            # input: N x channels_img x 64 x 64
            nn.Conv2d(
                channels_img, features_d, kernel_size=4, stride=2, padding=1
            ),
            nn.LeakyReLU(0.2),
            # _block(in_channels, out_channels, kernel_size, stride, padding)
            self._block(features_d, features_d * 2, 4, 2, 1),
            self._block(features_d * 2, features_d * 4, 4, 2, 1),
            self._block(features_d * 4, features_d * 8, 4, 2, 1),
            self._block(features_d * 8, features_d * 16, 4, 2, 1),
            # After all _block img output is 4x4 (Conv2d below makes into 1x1)
            nn.Conv2d(features_d * 16, 1, kernel_size=4, stride=2, padding=0),
            nn.Sigmoid(),
        )

    def _block(self, in_channels, out_channels, kernel_size, stride, padding):
        return nn.Sequential(
            nn.Conv2d(
                in_channels,
                out_channels,
                kernel_size,
                stride,
                padding,
                bias=False,
            ),
            #nn.BatchNorm2d(out_channels),
            nn.LeakyReLU(0.2),
        )

    def forward(self, x):
        return self.disc(x)

class Generator(nn.Module):
    def __init__(self, channels_noise, channels_img, features_g):
        super(Generator, self).__init__()
        self.net = nn.Sequential(
            # Input: N x channels_noise x 1 x 1
            self._block(channels_noise, features_g * 32, 4, 1, 0),
            self._block(features_g * 32, features_g * 16, 4, 1, 1),  # img: 4x4
            self._block(features_g * 16, features_g * 8, 4, 2, 1),  # img: 8x8
            self._block(features_g * 8, features_g * 4, 4, 2, 1),  # img: 16x16
            self._block(features_g * 4, features_g * 2, 4, 2, 1),  # img: 32x32
            nn.ConvTranspose2d(
                features_g * 2, channels_img, kernel_size=4, stride=2, padding=1
            ),
            # Output: N x channels_img x 64 x 64
            nn.Tanh(),
        )

    def _block(self, in_channels, out_channels, kernel_size, stride, padding):
        return nn.Sequential(
            nn.ConvTranspose2d(
                in_channels,
                out_channels,
                kernel_size,
                stride,
                padding,
                bias=False,
            ),
            #nn.BatchNorm2d(out_channels),
            nn.ReLU(),
        )

    def forward(self, x):
        return self.net(x)

def initialize_weights(model):
    # Initializes weights according to the DCGAN paper
    for m in model.modules():
        if isinstance(m, (nn.Conv2d, nn.ConvTranspose2d, nn.BatchNorm2d)):
            nn.init.normal_(m.weight.data, 0.0, 0.02)

Training of DCGAN network on a covid dataset with Discriminator
and Generator imported from models.py

import torch
import torch.nn as nn
import torch.optim as optim
import torchvision
import torchvision.datasets as datasets
import torchvision.transforms as transforms
from torch.utils.data import DataLoader
import matplotlib.pyplot as plt
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.animation as animation
from IPython.display import HTML

from torch.utils.tensorboard import SummaryWriter

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
LEARNING_RATE = 0.00025  # could also use two lrs, one for gen and one for disc
BATCH_SIZE = 128
IMAGE_SIZE = 128
CHANNELS_IMG = 3
NOISE_DIM = 128
NUM_EPOCHS = 7500
FEATURES_DISC = 128
FEATURES_GEN = 128

img_list = []
G_loss = []
D_loss = []

transforms = transforms.Compose(
    [
        transforms.Resize(IMAGE_SIZE),
        transforms.ToTensor(),
        transforms.Normalize(
            [0.5 for _ in range(CHANNELS_IMG)], [0.5 for _ in range(CHANNELS_IMG)]
        ),
    ]
)

dataset = datasets.ImageFolder(root="/content/subset", transform=transforms)
dataloader = DataLoader(dataset, batch_size=BATCH_SIZE, shuffle=True)
gen = Generator(NOISE_DIM, CHANNELS_IMG, FEATURES_GEN).to(device)
disc = Discriminator(CHANNELS_IMG, FEATURES_DISC).to(device)
initialize_weights(gen)
initialize_weights(disc)

opt_gen = optim.Adam(gen.parameters(), lr=LEARNING_RATE, betas=(0.5, 0.999))
opt_disc = optim.Adam(disc.parameters(), lr=LEARNING_RATE, betas=(0.5, 0.999))
criterion = nn.BCELoss()

fixed_noise = torch.randn(128, NOISE_DIM, 1, 1).to(device)

writer_real = SummaryWriter(f"generated/real11")
writer_fake = SummaryWriter(f"generated/fake11")

step = 0

gen.train()
disc.train()

for epoch in range(NUM_EPOCHS):
    # Target labels not needed! <3 unsupervised
    for batch_idx, (real, _) in enumerate(dataloader):
        real = real.to(device)
        noise = torch.randn(BATCH_SIZE, NOISE_DIM, 1, 1).to(device)
        fake = gen(noise)

        ### Train Discriminator: max log(D(x)) + log(1 - D(G(z)))
        disc_real = disc(real).reshape(-1)
        loss_disc_real = criterion(disc_real, torch.ones_like(disc_real))

        disc_fake = disc(fake.detach()).reshape(-1)
        loss_disc_fake = criterion(disc_fake, torch.zeros_like(disc_fake))
        loss_disc = (loss_disc_real + loss_disc_fake) / 2
        disc.zero_grad()
        loss_disc.backward()
        opt_disc.step()

        ### Train Generator: min log(1 - D(G(z))) <-> max log(D(G(z))
        output = disc(fake).reshape(-1)
        loss_gen = criterion(output, torch.ones_like(output))
        gen.zero_grad()
        loss_gen.backward()
        opt_gen.step()

        G_loss.append(loss_gen.item())
        D_loss.append(loss_disc.item())

        # Print losses occasionally and print to tensorboard
        if batch_idx in range(BATCH_SIZE):
            print(
                f"Epoch [{epoch}/{NUM_EPOCHS}] Batch {batch_idx}/{len(dataloader)} \
                  Loss D: {loss_disc:.4f}, loss G: {loss_gen:.4f}"
            )

            with torch.no_grad():
                fake = gen(fixed_noise)
                # take out (up to) 32 examples
                img_grid_real = torchvision.utils.make_grid(
                    real[:32], normalize=True
                )
                img_grid_fake = torchvision.utils.make_grid(
                    fake[:1], normalize=True
                )
                writer_real.add_image("Real1", img_grid_real, global_step=step)
                for batch_idx in range(BATCH_SIZE):
                  torchvision.utils.save_image(img_grid_fake, f"/content/generated/generated_image/Fake_image-{batch_idx}.png", global_step=step)

                writer_fake.add_image("Fake1", img_grid_fake, global_step=step)

            step += 1

this is an error I am getting when I run this code.
RuntimeError                              Traceback (most recent call last)
<ipython-input-5-8a323fce319d> in <module>()
     88
     89
---> 90         disc_fake = disc(fake.detach()).reshape(-1)
     91         loss_disc_fake = criterion(disc_fake, torch.zeros_like(disc_fake))
     92         loss_disc = (loss_disc_real + loss_disc_fake) / 2
/usr/local/lib/python3.7/dist-packages/torch/nn/modules/conv.py in _conv_forward(self, input, weight, bias)
    442                             _pair(0), self.dilation, self.groups)
    443         return F.conv2d(input, weight, bias, self.stride,
--> 444                         self.padding, self.dilation, self.groups)
    445
    446     def forward(self, input: Tensor) -> Tensor:

RuntimeError: Calculated padded input size per channel: (2 x 2). Kernel size: (4 x 4). Kernel size can't be greater than actual input size

Output:
I checked the code and the issue stems from the fact that you are expecting your Generator to generate images of dimension - 128x3x128x128 (batch_size x channels x image_dim x image_dim). However, the way that you have written the ConvTranspose2d operations, that is not the case.
I checked the output from the intermediate layers and your generator is generating output images of dimension - 128x3x80x80 which is a size mismatch as your Discriminator is expected input images of dimension - 128x3x128x128.
Here are the shapes of the intermediate outputs from your Generator's ConvTranpose2d operations -
torch.Size([128, 4096, 4, 4])
torch.Size([128, 2048, 5, 5])
torch.Size([128, 1024, 10, 10])
torch.Size([128, 512, 20, 20])
torch.Size([128, 256, 40, 40])
torch.Size([128, 3, 80, 80])

I'd suggest, you modify your Generator's ConvTranspose2d parameters as follows -
class Generator(nn.Module):
    def __init__(self, channels_noise, channels_img, features_g):
        super(Generator, self).__init__()
        self.net = nn.Sequential(
            self._block(channels_noise, features_g*32, 4, 1, 0),
            self._block(features_g*32, features_g*16, 4, 2, 1),
            self._block(features_g*16, features_g*8, 4, 2, 1),
            self._block(features_g*8, features_g*4, 4, 2, 1),
            self._block(features_g*4, features_g*2, 4, 2, 1),
            nn.ConvTranspose2d(features_g*2, channels_img, kernel_size=4, stride=2, padding=1),
            # Output: N x channels_img x 64 x 64
            nn.Tanh(),
        )

    def _block(self, in_channels, out_channels, kernel_size, stride, padding):
        return nn.Sequential(
            nn.ConvTranspose2d(
                in_channels,
                out_channels,
                kernel_size,
                stride,
                padding,
                bias=False,
            ),
            #nn.BatchNorm2d(out_channels),
            nn.ReLU(),
        )

    def forward(self, x):
        return self.net(x)

This produces the required dimension of 128x3x128x128. The intermediate dimensions are as follows -
torch.Size([128, 4096, 4, 4])
torch.Size([128, 2048, 8, 8])
torch.Size([128, 1024, 16, 16])
torch.Size([128, 512, 32, 32])
torch.Size([128, 256, 64, 64])
torch.Size([128, 3, 128, 128])

Just replace the Generator with this one and your code should work for images of dimension 3x128x128.


Instruction: Shifting an image with bilinear interpolation in pytorch

Input:
Suppose that I have an input x of size [H,W] and also a mu_x and mu_y (which may be fractional)representing the pixels in x and y direction to shift. Is there any efficient way in pytorch without using c++ to shift the tensor x for mu_x and mu_y units with bilinear interpolation.
To be more precise, let's say we have an image. mu_x = 5 and mu_y = 3, we may want to shift the image so that the image moves rightward 5 pixels and downward 3 pixels, with the pixels out of boundary of [H,W] removed and new pixels introduced at the other end of the boundary to be 0. However, with fractional mu_x and mu_y, we need to use bilinear interpolation to estimate the resulting image.
Is it possible to be implemented with pure pytorch tensor operations? Or do I need to use c++.

Output:
I believe you can achieve this by applying grid sampling on your original input and using a grid to guide the sampling process. If you take a coordinate grid of your image and sample using that the resulting image will be equal to the original image. However you can apply a shift on this grid and therefore sample with the given shift. Grid sampling works with floating-point grids of course, which means you can apply an arbitrary non-round shift to your image and choose a sampling mode (bilinear is the default).
This can be implemented out of the box with F.grid_sampling. Given an image tensor img, we first construct a pixel grid of that image using torch.meshgrid. Keep in mind the grid used by the sampler must be normalized to [-1, -1]. Therefore pixel x=0,y=0 should be mapped to (-1,-1), pixel x=w,y=h mapped to (1,1), and the center pixel will end up at around (0,0).
Use two torch.arange with a [0,1]-normalization followed by a remapping to [-1,1]:
>>> c,h,w = img.shape
>>> x, y = torch.arange(h)/(h-1), torch.arange(w)/(w-1)
>>> grid = torch.dstack(torch.meshgrid(x, y))*2-1

So the resulting grid has a shape of (c, h, w) which will be the dimensions of the output image produced by the sampling process.
Since we are not working with batched elements, we need to unsqueeze singleton dimensions on both img and grid. Then we can apply F.grid_sample:
>>> sampled = F.grid_sample(img[None], grid[None])

Following this you can apply your arbitrary mu_x, mu_y shift and even easily use this to batches of images and shifts. The way you would define your sampling is by defining a shifted grid:
>>> x_s, y_s = (torch.arange(h)+mu_y)/(h-1), (torch.arange(w)+mu_x)/(w-1)

Where mu_x and mu_y are the values in pixels (floating point) with wish which the image is shifted on the horizontal and vertical axes respectively. To acquire the sampled image, apply F.grid_sampling on a grid made up of x_s and y_s:
>>> grid_shifted = torch.dstack(torch.meshgrid(x_s, y_s))*2-1
>>> sampled = F.grid_sample(img[None], grid_shifted[None])


Instruction: resize_token_embeddings on the a pertrained model with different embedding size

Input:
I would like to ask about the way to change the embedding size of the trained model.
I have a trained model models/BERT-pretrain-1-step-5000.pkl.
Now I am adding a new token [TRA]to the tokeniser and try to use the resize_token_embeddings to the pertained one.
from pytorch_pretrained_bert_inset import BertModel #BertTokenizer
from transformers import AutoTokenizer
from torch.nn.utils.rnn import pad_sequence
import tqdm

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
model_bert = BertModel.from_pretrained('bert-base-uncased', state_dict=torch.load('models/BERT-pretrain-1-step-5000.pkl', map_location=torch.device('cpu')))

#print(tokenizer.all_special_tokens) #--> ['[UNK]', '[SEP]', '[PAD]', '[CLS]', '[MASK]']
#print(tokenizer.all_special_ids)    #--> [100, 102, 0, 101, 103]

num_added_toks = tokenizer.add_tokens(['[TRA]'], special_tokens=True)
model_bert.resize_token_embeddings(len(tokenizer))  # --> Embedding(30523, 768)
print('[TRA] token id: ', tokenizer.convert_tokens_to_ids('[TRA]'))  # --> 30522

But I encountered the error:
AttributeError: 'BertModel' object has no attribute 'resize_token_embeddings'

I assume that it is because the model_bert(BERT-pretrain-1-step-5000.pkl) I had has the different embedding size.
I would like to know if there is any way to fit the embedding size of my modified tokeniser and the model I would like to use as the initial weights.
Thanks a lot!!

Output:
resize_token_embeddings is a huggingface transformer method. You are using the BERTModel class from pytorch_pretrained_bert_inset which does not provide such a method. Looking at the code, it seems like they have copied the BERT code from huggingface some time ago.
You can either wait for an update from INSET (maybe create a github issue) or write your own code to extend the word_embedding layer:
from torch import nn

embedding_layer = model.embeddings.word_embeddings

old_num_tokens, old_embedding_dim = embedding_layer.weight.shape

num_new_tokens = 1

# Creating new embedding layer with more entries
new_embeddings = nn.Embedding(
        old_num_tokens + num_new_tokens, old_embedding_dim
)

# Setting device and type accordingly
new_embeddings.to(
    embedding_layer.weight.device,
    dtype=embedding_layer.weight.dtype,
)

# Copying the old entries
new_embeddings.weight.data[:old_num_tokens, :] = embedding_layer.weight.data[
    :old_num_tokens, :
]

model.embeddings.word_embeddings = new_embeddings


Instruction: URL fetch failure on resnet50_weights_tf

Input:
I have to train a model from where I dont have access to Internet.
base_cnn = resnet.ResNet50(
    weights="imagenet", input_shape=target_shape + (3,), include_top=False
)

Subsequently, training is failing with:
Traceback (most recent call last):
    base_cnn = resnet.ResNet50(
      return ResNet(stack_fn, False, True, 'resnet50', include_top, weights,
      weights_path = data_utils.get_file(
      raise Exception(error_msg.format(origin, e.errno, e.reason))
Exception: URL fetch failure on https://storage.googleapis.com/tensorflow/keras-applications/resnet/resnet50_weights_tf_dim_ordering_tf_kernels_notop.h5: None -- [Errno 101] Network is unreachable

Is there a way I can load the weights from drive instead of fetching an URL?

Output:
Weights could be downloaded as:
from tensorflow.keras.applications import resnet

base_cnn = resnet.ResNet50(
    weights="imagenet", input_shape=target_shape + (3,), include_top=False
)

base_cnn.save("weights.h5")

Then load the saved weights:
from tensorflow.keras.models import load_model
base_cnn=load_model('weights.h5')


Instruction: pytorch gives me an error when I don't run it in ~/ directory

Input:
When I run the following python script in an subdirectory, for example ~/test_dir, python results in an error :
import torch
import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()
        # 1 input image channel, 6 output channels, 5x5 square convolution
        # kernel
        self.conv1 = nn.Conv2d(1, 6, 5)
        self.conv2 = nn.Conv2d(6, 16, 5)
        # an affine operation: y = Wx + b
        self.fc1 = nn.Linear(16 * 5 * 5, 120)  # 5*5 from image dimension
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        # Max pooling over a (2, 2) window
        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
        # If the size is a square, you can specify with a single number
        x = F.max_pool2d(F.relu(self.conv2(x)), 2)
        x = torch.flatten(x, 1) # flatten all dimensions except the batch dimension
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

net = Net()
print(net)

Error msg:
Traceback (most recent call last):
  File "/Users/xxxxxxx/torch/torch.py", line 1, in <module>
    import torch
  File "/Users/xxxxxxx/torch/torch.py", line 2, in <module>
    import torch.nn as nn
ModuleNotFoundError: No module named 'torch.nn'; 'torch' is not a package

If I run the same file in the home directory ~/ it does not result in the error message shown above.
➜  ~ python3 test.py
Net(
  (conv1): Conv2d(1, 6, kernel_size=(5, 5), stride=(1, 1))
  (conv2): Conv2d(6, 16, kernel_size=(5, 5), stride=(1, 1))
  (fc1): Linear(in_features=400, out_features=120, bias=True)
  (fc2): Linear(in_features=120, out_features=84, bias=True)
  (fc3): Linear(in_features=84, out_features=10, bias=True)
)

➜  ~ which python3
/Library/Frameworks/Python.framework/Versions/3.9/bin/python3

I did not encounter any similar errors with other packages, so I assume it is caused by pytorch.
I would appreciate any help or hint on how to solve this.

Output:
It seems that you named your working directory and a script torch. It causes a conflict with the installed Pytorch library, therefore you're calling your torch, not the installed one.
Try it after changing the names of your directory and script.


Instruction: Using unlabelled custom images instead of Mnist and CIFAR for a simple GAN with Pytorch

Input:
I am trying to replace standardized data from pytorch such as MNIST and CIFAR with unlabeled custom images in png format in a simple GAN. Unfortunately most examples always use such datasets and dont show the process of preparing and implementing custom data into GANs.  I have stored my  png-images (336*336, RGB) in the working directory of VS Code. Could you please provide me with a suggestion on how to go forward? Below you find the current code where I would like to replace mnist with my own images to generate new images (from #Preparing Training Data to #Plotting Samples:
import torch
from torch import nn

import math
import matplotlib.pyplot as plt
import torchvision
import torchvision.transforms as transforms

torch.manual_seed(111)

# DEVICE
device = ""
if torch.cuda.is_available():
    device = torch.device("cuda")
else:
    device = torch.device("cpu")

print(device)

***# PREPARING TRAINING DATA

transform = transforms.Compose(
    [transforms.ToTensor(), transforms.Normalize((0.5,), (0.5,))]
)

#  LOADING DATA
train_set = torchvision.datasets.MNIST(
    root=".", train=True, download=True, transform=transform
)

# CREATE DATALOADER

batch_size = 32
train_loader = torch.utils.data.DataLoader(
    train_set, batch_size=batch_size, shuffle=True
)***

# PLOTTING SAMPLES

real_samples, mnist_labels = next(iter(train_loader))
for i in range(16):
    ax = plt.subplot(4, 4, i + 1)
    plt.imshow(real_samples[i].reshape(28, 28), cmap="gray_r")
    plt.xticks([])
    plt.yticks([])
    plt.show()´

# IMPLEMENTING DISCRIMINATOR AND GENERATOR

class Discriminator(nn.Module):
    def __init__(self):
        super().__init__()
        self.model = nn.Sequential(
            nn.Linear(784, 1024),
            nn.ReLU(),
            nn.Dropout(0.3),
            nn.Linear(1024, 512),
            nn.ReLU(),
            nn.Dropout(0.3),
            nn.Linear(512, 256),
            nn.ReLU(),
            nn.Dropout(0.3),
            nn.Linear(256, 1),
            nn.Sigmoid(),
        )

    def forward(self, x):
        x = x.view(x.size(0), 784)
        output = self.model(x)
        return output

discriminator = Discriminator().to(device=device)

class Generator(nn.Module):
    def __init__(self):
        super().__init__()
        self.model = nn.Sequential(
            nn.Linear(100, 256),
            nn.ReLU(),
            nn.Linear(256, 512),
            nn.ReLU(),
            nn.Linear(512, 1024),
            nn.ReLU(),
            nn.Linear(1024, 784),
            nn.Tanh(),
        )

    def forward(self, x):
        output = self.model(x)
        output = output.view(x.size(0), 1, 28, 28)
        return output

generator = Generator().to(device=device)

# TRAINING PARAMS

lr = 0.0001
num_epochs = 100
loss_function = nn.BCELoss()

optimizer_discriminator = torch.optim.Adam(discriminator.parameters(), lr=lr)
optimizer_generator = torch.optim.Adam(generator.parameters(), lr=lr)

# TRAINING LOOP

for epoch in range(num_epochs):
    for n, (real_samples, mnist_labels) in enumerate(train_loader):
        # Data for training the discriminator
        real_samples = real_samples.to(device=device)
        real_samples_labels = torch.ones((batch_size, 1)).to(
            device=device
        )
        latent_space_samples = torch.randn((batch_size, 100)).to(
            device=device
        )
        generated_samples = generator(latent_space_samples)
        generated_samples_labels = torch.zeros((batch_size, 1)).to(
            device=device
        )
        all_samples = torch.cat((real_samples, generated_samples))
        all_samples_labels = torch.cat(
            (real_samples_labels, generated_samples_labels)
        )

        # Training the discriminator
        discriminator.zero_grad()
        output_discriminator = discriminator(all_samples)
        loss_discriminator = loss_function(
            output_discriminator, all_samples_labels
        )
        loss_discriminator.backward()
        optimizer_discriminator.step()

        # Data for training the generator
        latent_space_samples = torch.randn((batch_size, 100)).to(
            device=device
        )

        # Training the generator
        generator.zero_grad()
        generated_samples = generator(latent_space_samples)
        output_discriminator_generated = discriminator(generated_samples)
        loss_generator = loss_function(
            output_discriminator_generated, real_samples_labels
        )
        loss_generator.backward()
        optimizer_generator.step()

        # Show loss
        if n == batch_size - 1:
            print(f"Epoch: {epoch} Loss D.: {loss_discriminator}")
            print(f"Epoch: {epoch} Loss G.: {loss_generator}")

# SAMPLES

latent_space_samples = torch.randn(batch_size, 100).to(device=device)
generated_samples = generator(latent_space_samples)

generated_samples = generated_samples.cpu().detach()
for i in range(16):
    ax = plt.subplot(4, 4, i + 1)
    plt.imshow(generated_samples[i].reshape(28, 28), cmap="gray_r")
    plt.xticks([])
    plt.yticks([])
    plt.show()´´´

Output:
In the example that you shared above, you are trying to train your generator on single-channel images. Specifically, your Generator and Discriminator layers are written to handle images of dimension 1x28x28 which are the dimensions of MNIST or Fashion-MNIST datasets.
I am supposing that you are trying to train color images (3 channels) or a different dimension, in your case - 3x336x336. In your example, I have added a tensor transform that first converts an input image of any dimension to an image of dimension - 3x28x28.
Here are the code examples for creating the custom dataset and custom dataloader.
from glob import glob
from torch.utils.data import Dataset, DataLoader
from torchvision import transforms
from skimage import io

path = 'your/image/path'
image_paths = glob(path + '/*.jpg')

img_size = 28
batch_size = 32

transform = transforms.Compose(
        [
            transforms.ToPILImage(),
            transforms.Resize(img_size),
            transforms.CenterCrop(img_size),
            transforms.ToTensor(),
            transforms.Normalize([0.5, 0.5, 0.5], [0.5, 0.5, 0.5]),
        ]
    )

class ImageDataset(Dataset):
    def __init__(self, paths, transform):
        self.paths = paths
        self.transform = transform

    def __len__(self):
        return len(self.paths)

    def __getitem__(self, index):
        image_path = self.paths[index]
        image = io.imread(image_path)

        if self.transform:
            image_tensor = self.transform(image)

        return image_tensor

dataset = ImageDataset(image_paths, transform)

train_loader = DataLoader(dataset, batch_size=batch_size, num_workers=1, shuffle=True)

The dataloader generates image tensors of dimension - batch_size x img_channels x img_dim x img_dim which in this case would be - 32x3x28x28.
import torch
import torch.nn as nn

device = 'cuda' if torch.cuda.is_available() else 'cpu'

class Discriminator(nn.Module):
    def __init__(self):
        super().__init__()
        self.model = nn.Sequential(
            nn.Linear(784*3, 2048),
            nn.ReLU(),
            nn.Dropout(0.3),
            nn.Linear(2048, 1024),
            nn.ReLU(),
            nn.Dropout(0.3),
            nn.Linear(1024, 512),
            nn.ReLU(),
            nn.Dropout(0.3),
            nn.Linear(512, 256),
            nn.ReLU(),
            nn.Dropout(0.3),
            nn.Linear(256, 1),
            nn.Sigmoid(),
        )

    def forward(self, x):
        x = x.view(x.size(0), 784*3) # change required for 3 channel image
        output = self.model(x)
        return output

discriminator = Discriminator().to(device=device)

class Generator(nn.Module):
    def __init__(self):
        super().__init__()
        self.model = nn.Sequential(
            nn.Linear(100, 256),
            nn.ReLU(),
            nn.Linear(256, 512),
            nn.ReLU(),
            nn.Linear(512, 1024),
            nn.ReLU(),
            nn.Linear(1024, 2048),
            nn.ReLU(),
            nn.Linear(2048, 784*3),
            nn.Tanh(),
        )

    def forward(self, x):
        output = self.model(x)
        output = output.view(x.size(0), 3, 28, 28)
        return output

generator = Generator().to(device=device)

# TRAINING PARAMS

lr = 0.0001
num_epochs = 100
loss_function = nn.BCELoss()

optimizer_discriminator = torch.optim.Adam(discriminator.parameters(), lr=lr)
optimizer_generator = torch.optim.Adam(generator.parameters(), lr=lr)

This is the code for Generator and Discriminator. I have made slight modifications to the Generator and Discriminator. Notice the addition of the following layers in the Discriminator
nn.Linear(784*3, 2048),
nn.ReLU(),
nn.Dropout(0.3),
nn.Linear(2048, 1024),

and these in the Generator
nn.Linear(1024, 2048),
nn.ReLU(),
nn.Linear(2048, 784*3)

This is required to generate and discriminate images of the correct dimension.
Finally, this is your training loop -
for epoch in range(num_epochs):
    for n, real_samples in enumerate(train_loader):
        # Data for training the discriminator
        real_samples = real_samples.to(device=device)
        real_samples_labels = torch.ones((batch_size, 1)).to(
            device=device
        )
        latent_space_samples = torch.randn((batch_size, 100)).to(
            device=device
        )
        print(f'Latent space samples : {latent_space_samples.shape}')
        generated_samples = generator(latent_space_samples)
        generated_samples_labels = torch.zeros((batch_size, 1)).to(
            device=device
        )
        all_samples = torch.cat((real_samples, generated_samples))
        print(f'Real samples : {real_samples.shape}, generated samples : {generated_samples.shape}')
        all_samples_labels = torch.cat(
            (real_samples_labels, generated_samples_labels)
        )

        # Training the discriminator
        discriminator.zero_grad()
        output_discriminator = discriminator(all_samples)
        loss_discriminator = loss_function(
            output_discriminator, all_samples_labels
        )
        loss_discriminator.backward()
        optimizer_discriminator.step()

        # Data for training the generator
        latent_space_samples = torch.randn((batch_size, 100)).to(
            device=device
        )

        # Training the generator
        generator.zero_grad()
        generated_samples = generator(latent_space_samples)
        output_discriminator_generated = discriminator(generated_samples)
        loss_generator = loss_function(
            output_discriminator_generated, real_samples_labels
        )
        loss_generator.backward()
        optimizer_generator.step()

        # Show loss
        if n == batch_size - 1:
            print(f"Epoch: {epoch} Loss D.: {loss_discriminator}")
            print(f"Epoch: {epoch} Loss G.: {loss_generator}")

This works because the images are reshaped from the 784*3 to the 3*28*28 dimension.
This would work but if you are handling images of 3 channels, you would need to write ConvTranspose2d and Conv2d operations in your Generator and Discriminator for upsampling and downsampling the image respectively.
If you are interested in an example that uses ConvTranspose2d and Conv2d for processing multidimensional images, here it is - https://drive.google.com/file/d/1gYiBHPu-r3kialO0klsTdE2RjBR50rMs/view?usp=sharing. To handle images of different dimensions, you would have to modify the layers in the Generator and Discriminator classes.


Instruction: How to perform torch.meshgrid over multiple tensors in parallel?

Input:
Let's say we have a tensor x of size [60,9] and a tensor y of size [60,9]
Is it possible to do an operation like xx,yy = torch.meshgrid(x,y) such that xx and yy is of size [60,9,9] and xx[i,:,:], yy[i,:,:] is basically torch.meshgrid(x[i],y[i])?
The built-in torch.meshgrid operation only accepts 1d tensors, is it possible to do the above operation without using for loops (which is inefficient as it does not take use of GPU's parallel operation)?

Output:
I don't believe you will gain anything since the initialization of the tensors is not done on the GPU. So a proposed approach would indeed be to loop over x and y or using map as an iterable:
grids = map(torch.meshgrid, zip(x,y))


Instruction: Multiple parallel image-encoders in pytorch

Input:
I have an image encoder which can generate output features. I want to split my image into several patches (around 16) and feed each patch to a separate image encoder (the parameters for each encoder are different).
Initially, it's like:
Input -> Encoder -> Output
I want to change it like this:
patch-input1 -> Encoder1 -> output1
patch-input2 -> Encoder2 -> output2
...
patch-inputN -> EncoderN -> outputN

I'm using nn.Module to create the model class. The number of patches N is not fixed and is determined during the initialisation of the model so I wanted to use the list of objects (in __init__() function) to create multiple instances of the same encoder and call them over each part-input in a loop in the forward function.
When I run inference on a sample input, it doesn't show any error and I print out the number of trainable parameters using torchsummary module.
The problem is I do not see the parameters corresponding to the encoders in the number of parameters. More specifically in the single encoder pipeline there are around one million parameters and in the multiple encoder pipeline, params there are only around 200 thousand. The later's summary doesn't even show the description of various layers in the encoder.
A code snippet of my implementation would look like this:
class Patch(nn.Module):
   '''
   takes image tensor input
   returns a list of patch tensors
   '''
class Encoder(nn.Module):
   '''
   definition of the encoder
   '''
class Model(nn.Module):
   def __init__(self, patches=10, *kwargs):
      super().__init__()
      self.patch = Patch()
      self.enc = []
      for i in range(patches):
         enc.append(Encoder())
   def forward(self, x):
      '''
      patches is a list of tensors formed using an image
      tensor
      '''
      patches = self.patch(x)
      output = []
      for i in range(patches):
         output.append(self.enc[i](patch[i]))
      output_feats = torch.cat(output, dim=0)

Is there a better way to implement parallel encoders? What am I doing wrong?

Output:
You need to use a nn.ModuleList in order to properly register the list of Patch modules inside Model nn.Module:
class Model(nn.Module):
   def __init__(self, patches=10, *kwargs):
      super().__init__()
      self.patch = Patch()
      enc = []
      for i in range(patches):
         enc.append(Encoder())
      self.enc = nn.ModuleList(enc)

In a slightly refactored form:
class Model(nn.Module):
   def __init__(self, patches=10, *kwargs):
      super().__init__()
      self.patch = Patch()
      self.enc = nn.ModuleList(Encoder() for _ in range(patches))

   def forward(self, x):
      patches = self.patch(x)
      output_feats = torch.cat([enc(patch) for enc, patch in zip(self.enc, patches])


Instruction: Gradient Descent Reassignment in Pytorch

Input:
I was following a series of tutorial on youtube about deep learning, and I encountered a problem which really confuses me.
X = torch.tensor([1,2,3,4], dtype = torch.float32)
Y = torch.tensor([2,4,6,8], dtype = torch.float32)

w = torch.tensor(0.0, dtype = torch.float32, requires_grad=True)

def forward(x):
  return w*x;

def loss(y, y_predicted):
  return ((y-y_predicted)**2).mean()

print(f'Prediction before training: f(5) = {forward(5):.3f}')

learning_rate= 0.01
epoch = 20
for i in range(epoch):
  y_pred = forward(X)
  l = loss(Y, y_pred)
  l.backward()
  with torch.no_grad():
    w = w - learning_rate * w.grad
    # (w -= learning_rate * w.grad) # would not cause error in the following line

  w.grad.zero_() #error : 'NoneType' object has no attribute 'zero_'
  if i % 1 ==0:
    print(f'weight : {w}, loss : {l}')

I really wonder the difference between "w = w - learning_rate * w.grad" and "w -= learning_rate * w.grad" cause these two are the same in my expericence. Thanks!

Output:
As pointed out in the comment, the problem is in how Pytorch computes/stores gradients. In fact,
w-= learning_rate * w.grad

is an in-place operation, which will make w keep its initial properties (the requires_grad=True). Usually in Pytorch, we avoid in-place operations as it may break the computational graph used by Autograd (see Pytorch Forum Post).
But for you, this:
w = w - learning_rate * w.grad

is not in-place. Thus, w is assigned to a new copy, and because of the torch.no_grad() statement, this copy won't have a .grad attribute.


Instruction: Runtime error: mat1 dim 1 must match mat2 dim 0

Input:
I am running one classification program using GaborNet. Part of my code is
class module(nn.Module):
        def __init__(self):
            super(module, self).__init__()
            self.g0 = modConv2d(in_channels=3, out_channels=32, kernel_size=(11, 11), stride=1)
            self.c1 = nn.Conv2d(in_channels=32,out_channels=64,kernel_size=(2, 2),stride=1)
            self.c2 = nn.Conv2d(in_channels=64,out_channels=128,kernel_size=(2, 2),stride=1)

            #x = x.view(x.size(0), -1)

            #x = x.view(1, *x.shape)
            #x=x.view(-1,512*12*12)

            x = F.relu(self.fc1(x))
            print(x.shape)
            x = F.relu(self.fc2(x))
            print(x.shape)
            x = self.fc3(x)
            return x

I am getting this error at this position :
x = F.relu(self.fc1(x)

and the error is : RuntimeError: mat1 dim 1 must match mat2 dim 0

However the shape of the input image in the subsequent layers are till fc1 is:
torch.Size([64, 3, 150, 150])
torch.Size([64, 32, 140, 140])
torch.Size([64, 32, 70, 70])

Output:
You were on the right track, you indeed need to reshape your data just after the convolution layers, and before proceeding with the fully connected layers.
The best approach for flattening your tensor is to use nn.Flatten otherwise you might end up disrupting the batch size. The last spatial layer outputs a shape of (64, 128, 3, 3) and once flattened this tensor has a shape of (64, 1152) where 1152 = 128*3*3. Therefore your first fully connected layer should have 1152 neurons.
Something like this should work:
class GaborNN(nn.Module):
    def __init__(self):
        super().__init__()
        ...
        self.fc1 = nn.Linear(in_features=1152, out_features=128)
        self.fc2 = nn.Linear(in_features=128, out_features=128)
        self.fc3 = nn.Linear(in_features=128, out_features=7)
        self.flatten = nn.Flatten()

    def forward(self, x):
        ...
        x = self.flatten(x)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x