Instruction: Error loading caffe2_detectron_ops.dll when importing torch in IDE

Input:
Pytorch GPU did work for me, but after reinstalling anaconda I got this error:
Error loading “caffe2_detectron_ops.dll” (when installing the cpu version) or
Error loading “caffe2_detectron_ops_gpu.dll” (when installing the gpu version)
This error appears already when importing torch in spyder IDE.
Total error message: "OSError: [WinError 182] The operating system cannot run %1. Error loading "C:\Users\konin\anaconda3\envs\pytorch_env\lib\site-packages\torch\lib\caffe2_detectron_ops.dll" or one of its dependencies."
Simply deleting “caffe2_detectron_ops.dll” will give me a new error: Error loading “caffe2_module_test_dynamic.dll”, then error loading “caffe2_observers.dll”, …
Deleting all of them didn’t solve my problem, it ended up with error “ImportError: DLL load failed while importing _C”. I’m working in a conda environment (python 3.9.7), and starting new environments give the same errors. CPU or GPU download of pytorch won’t make a difference. Installing intel-openmp didn’t fix it. Reinstalling torch didn't help.
I've done everything I could find about this error message. It is actually working when using the CMD prompt, but not when running the file in spyder.
Any suggestion would be really appreciated, thanks

Output: Btw, for everyone curious, got it solved by resetting all of my enviroments, and  setting up my environments more cleanly. Probably this conda environment got corrupted by my base environment files.


Instruction: create one-hot encoding for values of histogram bins

Input:
Given the tensor below of size torch.Size([22])
tensor([-20.1659, -19.7022, -17.4124, -16.7115, -16.4696, -15.6848, -15.5201, -14.5384, -12.5017, -12.4227, -11.0946, -10.7844, -10.5467,  -9.3933,  -4.2351,  -4.0521,  -3.8844, -3.8668,  -3.7337,  -3.7002,  -3.6242,  -3.5820])

and the below historgram:
hist = torch.histogram(tensor, 5)
hist
torch.return_types.histogram(
hist=tensor([3., 5., 5., 1., 8.]),
bin_edges=tensor([-20.1659, -16.8491, -13.5323, -10.2156,  -6.8988,  -3.5820]))

For each value of the tensor, how to create a one hot encoding that corresponds to its bin number, so that the output is a tensor of size torch.Size([22, 5])

Output:
You can use torch.repeat_interleave
import torch

bins = torch.tensor([3, 5, 5, 1, 8])
one_hots = torch.eye(len(bins))
one_hots = torch.repeat_interleave(one_hots, bins, dim=0)
print(one_hots)

output
tensor([[1., 0., 0., 0., 0.],
        [1., 0., 0., 0., 0.],
        [1., 0., 0., 0., 0.],
        [0., 1., 0., 0., 0.],
        [0., 1., 0., 0., 0.],
        [0., 1., 0., 0., 0.],
        [0., 1., 0., 0., 0.],
        [0., 1., 0., 0., 0.],
        [0., 0., 1., 0., 0.],
        [0., 0., 1., 0., 0.],
        [0., 0., 1., 0., 0.],
        [0., 0., 1., 0., 0.],
        [0., 0., 1., 0., 0.],
        [0., 0., 0., 1., 0.],
        [0., 0., 0., 0., 1.],
        [0., 0., 0., 0., 1.],
        [0., 0., 0., 0., 1.],
        [0., 0., 0., 0., 1.],
        [0., 0., 0., 0., 1.],
        [0., 0., 0., 0., 1.],
        [0., 0., 0., 0., 1.],
        [0., 0., 0., 0., 1.]])


Instruction: Transformers: How to use CUDA for inferencing?

Input:
I have fine-tuned my models with GPU but inferencing process is very slow, I think this is because inferencing uses CPU by default. Here is my inferencing code:
txt = "This was nice place"
model = transformers.BertForSequenceClassification.from_pretrained(model_path, num_labels=24)
tokenizer = transformers.BertTokenizer.from_pretrained('TurkuNLP/bert-base-finnish-cased-v1')
encoding = tokenizer.encode_plus(txt, add_special_tokens = True, truncation = True, padding = "max_length", return_attention_mask = True, return_tensors = "pt")
output = model(**encoding)
output = output.logits.softmax(dim=-1).detach().cpu().flatten().numpy().tolist()

Here is my second inferencing code, which is using pipeline (for different model):
classifier = transformers.pipeline("sentiment-analysis", model="distilbert-base-uncased-finetuned-sst-2-english")
result = classifier(txt)

How can I force transformers library to do faster inferencing on GPU? I have tried adding model.to(torch.device("cuda")) but that throws error:
Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu

I suppose the problem is related to the data not being sent to GPU. There is a similar issue here: pytorch summary fails with huggingface model II: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu
How would I send data to GPU with and without pipeline? Any advise is highly appreciated.

Output:
You should transfer your input to CUDA as well before performing the inference:
device = torch.device('cuda')

# transfer model
model.to(device)

# define input and transfer to device
encoding = tokenizer.encode_plus(txt,
     add_special_tokens=True,
     truncation=True,
     padding="max_length",
     return_attention_mask=True,
     return_tensors="pt")

encoding = encoding.to(device)

# inference
output = model(**encoding)

Be aware nn.Module.to is in-place, while torch.Tensor.to is not (it does a copy!).


Instruction: How can reshape 2d tensor to 1d

Input:
tensor([[0, 5],
        [1, 4],
        [2, 2],
        [4, 2],
        [7, 9],
        [2, 0])

I want to reshape this tensor to
tensor([[5],
        [14],
        [22],
        [42],
        [79],
        [20])

how can i solve this problem? please help me

Output:
I don't know about tensor data type, but for normal list you can iterate over each inner list and convert them to digit
def toDigit(l):
    z = 1
    s = 0
    for v in l[::-1]:
        s += v * z
        z *= 10
    return s

a = [[0, 5],[1, 4],[2, 2],[4, 2],[7, 9],[2, 0], [0,0]]
print([toDigit(t) for t in a])

The output will be:
[5, 14, 22, 42, 79, 20, 0]


Instruction: How to extract layer shape and type from ONNX / PyTorch?

Input:
I would like to 'translate' a PyTorch model to another framework (non-tf/keras).
I'm trying to take a pytorch model, and automate the translation to the other framework, which contains similar types of layers (i.e. conv2d, dense,...).
Is there a way from pytorch directly, or through onnx to retrieve a models layers, their types, shapes and connections ? (Weights are not important so far)

Output:
From discussion in comments on your question:
each node in onnx has a list of named inputs and a list of named outputs.
For the input list accessed with node.input you have for each input index either the graph input_name that feeds that input or the name of a previous output that feeds that input. There are also initializers which are onnx parameters.
# model is an onnx model
graph = model.graph
# graph inputs
for input_name in graph.input:
    print(input_name)
# graph parameters
for init in graph.init:
    print(init.name)
# graph outputs
for output_name in graph.output:
    print(output_name)
# iterate over nodes
for node in graph.node:
    # node inputs
    for idx, node_input_name in enumerate(node.input):
        print(idx, node_input_name)
    # node outputs
    for idx, node_output_name in enumerate(node.output):
        print(idx, node_output_name)

Shape inference is talked about here and for python here
The gist for python is found here
Reproducing the gist from 3:
from onnx import shape_inference
inferred_model = shape_inference.infer_shapes(original_model)

and find the shape info in inferred_model.graph.value_info.
You can also use netron or from GitHub to have a visual representation of that information.


Instruction: (Torch tenor) Subtracting different dimension matrices

Input:
matrice1 = temp.unsqueeze(0)
print(M.shape)
matrice2 = M.permute(1, 0, 2, 3)
print(matrice2.shape)
print( torch.abs(matrice1 - matrice2).shape )

#torch.Size([1, 10, 3, 256])
#torch.Size([10, 1, 3, 256])
#torch.Size([10, 10, 3, 256])

I got the outcome above. I am wondering why the subtraction between two different dimension tensors make the outcome the tensor that has the shape as [10,10,3,256].

Output: According to the broadcast semantics of PyTorch,The two tensors are "broadcastable" in your case, so they are automatically expanded to the equal size of torch.Size([10, 10, 3, 256]).


Instruction: How can I use custom tokenizer in opennmt transformer

Input:
I'm tring to transformer for translation with opennmt-py.
And I already have the tokenizer trained by sentencepiece(unigram).
But I don't know how to use my custom tokenizer in training config yaml.
I'm refering the site of opennmt-docs (https://opennmt.net/OpenNMT-py/examples/Translation.html).
Here are my code and the error .
# original_ko_en.yaml
## Where is the vocab(s)
src_vocab: /workspace/tokenizer/t_50k.vocab
tgt_vocab: /workspace/tokenizer/t_50k.vocab

# Corpus opts:
data:
   corpus_1:
       path_src: /storage/genericdata_basemodel/train.ko
       path_tgt: /storage/genericdata_basemodel/train.en
       transforms: [sentencepiece]
       weight: 1
   valid:
       path_src: /storage/genericdata_basemodel/valid.ko
       path_tgt: /storage/genericdata_basemodel/valid.en
       transforms: [sentencepiece]

#### Subword
src_subword_model: /workspace/tokenizer/t_50k.model
tgt_subword_model: /workspace/tokenizer/t_50k.model
src_subword_nbest: 1
src_subword_alpha: 0.0
tgt_subword_nbest: 1
tgt_subword_alpha: 0.0

# filter
# src_seq_length: 200
# tgt_seq_length: 200

# silently ignore empty lines in the data
skip_empty_level: silent

# Train on a single GPU
world_size: 1
gpu_ranks: [0]

# General opts
save_model: /storage/models/opennmt_v1/opennmt
keep_checkpoint: 100
save_checkpoint_steps: 10000
average_decay: 0.0005
seed: 1234
train_steps: 500000
valid_steps: 20000
warmup_steps: 8000
report_every: 1000

# Model
decoder_type: transformer
encoder_type: transformer
layers: 6
heads: 8
word_vec_size: 512
rnn_size: 512
transformer_ff: 2048
dropout: 0.1
label_smoothing: 0.1

# Optimization
optim: adam
adam_beta1: 0.9
adam_beta2: 0.998
decay_method: noam
learning_rate: 2.0
max_grad_norm: 0.0
normalization: tokens
param_init: 0.0
param_init_glorot: 'true'
position_encoding: 'true'

# Batching
batch_size: 4096
batch_type: tokens
accum_count: 8
max_generator_batches: 2

# Visualization
tensorboard: True
tensorboard_log_dir: /workspace/runs/onmt1

and When I typing < onmt_train -config xxx.yaml >

So, the questions are two.

my sentencepiece tokenizer embedding is float. How can i resolve the int error?
When training stopped by accident or I want to train more some model.pt
what is the command to start training from the some model.pt ?

I'll look forward to any opinion.
Thanks.

Output:
I got the answers.

we can use tools/spm_to_vocab in onmt.
train_from argument is the one.


Instruction: What makes the difference in "grad attribute" in the following context?

Input:
Consider the following two contexts
with torch.no_grad():
  params = params - learning_rate * params.grad

and
with torch.no_grad():
  params -= learning_rate * params.grad

In the second case .backward() is running smoothly and in the first case it is giving the
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

What is the reason for this as it is normal to use x-= a and x = x - a interchangeably?

Output:
Note that  x -= a and x = x - a cannot be used interchangeably: The latter creates a new tensor that is assigned to the variable x, while the former performes an in place operation.
Therefore with
with torch.no_grad():
  params -= learning_rate * params.grad

everything works fine in your optimization loop, while in
with torch.no_grad():
  params = params - learning_rate * params.grad

the variable params gets overwritten with a new tensor. Since this new tensor was created within a torch.no_grad() context, this means that this new tensor has params.requires_grad=False and therefore does not have a .grad attribute. Therefore in the next iteration torch will complain that params.grad does not exist.


Instruction: When doing pre-training of a transformer model, how can I add words to the vocabulary?

Input:
Given a DistilBERT trained language model for a given language, taken from the Huggingface hub, I want to pre-train the model on a specific domain, and I want to add new words that are:

definitely non existing in the original training set
and impossible to handle via word piece toeknization - basically you can think of these words as "codes" that are a normalized form of a named entity

Consider that:

I would like to avoid to learn a new tokenizer: I am fine to add the new words, and then let the model learn their embeddings via pre-training
the number of the "words" is way larger that the "unused" tokens in the "stock" vocabulary

The only advice that I have found is the one reported here:

Append it to the end of the vocab, and write a script which generates a new checkpoint that is identical to the pre-trained checkpoint, but but with a bigger vocab where the new embeddings are randomly initialized (for initialized we used tf.truncated_normal_initializer(stddev=0.02)). This will likely require mucking around with some tf.concat() and tf.assign() calls.

Do you think this is the only way of achieve my goal?
If yes, I do not have any idea of how to write this "script": does someone has some hints at how to proceeed (sample code, documentation etc)?

Output:
As per my comment, I'm assuming that you go with a pre-trained checkpoint, if only to "avoid [learning] a new tokenizer."
Also, the solution works with PyTorch, which might be more suitable for such changes. I haven't checked Tensorflow (which is mentioned in one of your quotes), so no guarantees that this works across platforms.
To solve your problem, let us divide this into two sub-problems:

Adding the new tokens to the tokenizer, and
Re-sizing the token embedding matrix of the model accordingly.

The first can actually be achieved quite simply by using .add_tokens(). I'm referencing the slow tokenizer's implementation of it (because it's in Python), but from what I can see, this also exists for the faster Rust-based tokenizers.
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")
# Will return an integer corresponding to the number of added tokens
# The input could also be a list of strings instead of a single string
num_new_tokens = tokenizer.add_tokens("dennlinger")

You can quickly verify that this worked by looking at the encoded input ids:
print(tokenizer("This is dennlinger."))
# 'input_ids': [101, 2023, 2003, 30522, 1012, 102]

The index 30522 now corresponds to the new token with my username, so we can check the first part. However, if we look at the function docstring of .add_tokens(), it also says:

Note, hen adding new tokens to the vocabulary, you should make sure to also resize the token embedding matrix of the model so that its embedding matrix matches the tokenizer.
In order to do that, please use the PreTrainedModel.resize_token_embeddings method.

Looking at this particular function, the description is a bit confusing, but we can get a correctly resized matrix (with randomly initialized weights for new tokens), by simply passing the previous model size, plus the number of new tokens:
from transformers import AutoModel

model = AutoModel.from_pretrained("distilbert-base-uncased")
model.resize_token_embeddings(model.config.vocab_size + num_new_tokens)

# Test that everything worked correctly
model(**tokenizer("This is dennlinger", return_tensors="pt"))

EDIT: Notably, .resize_token_embeddings() also takes care of any associated weights; this means, if you are pre-training, it will also adjust the size of the language modeling head (which should have the same number of tokens), or fix tied weights that would be affected by an increased number of tokens.


Instruction: Python - Show one image from an image array

Input:
I'm going through this tutorial on pytorch. https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html
And I've been able to show real images next to the fake ones that I have generated.
# Grab a batch of real images from the dataloader
real_batch = next(iter(dataloader))

# Plot the real images
plt.figure(figsize=(15,15))
plt.subplot(1,2,1)
plt.axis("off")
plt.title("Real Images")
plt.imshow(np.transpose(vutils.make_grid(real_batch[0].to(device)[:64], padding=5, normalize=True).cpu(),(1,2,0)))

# Plot the fake images from the last epoch
plt.subplot(1,2,2)
plt.axis("off")
plt.title("Fake Images")
plt.imshow(np.transpose(img_list[-1],(1,2,0)))
plt.show()

Which from my dataset results in this:

I was wondering how I can show one image from the fake images generated. I also want to show it as a 512 X 512 image if possible.
Edit:
The img_list[-1].shape is torch.Size([3, 530, 530]).
Edit2:
This part of the training shows that img_list is a list of images with each image being a group of sub-images (not being able to separate them). Is there a way I can edit this to make img_list an image of each fake image generated?

Output:
Here is what I wanted:
noise = torch.randn(1, nz, 1, 1, device=device)
with torch.no_grad():
    newfake = netG(noise).detach().cpu()

plt.axis("off")
plt.imshow(np.transpose(newfake[0],(1,2,0)))
plt.show()

As it generates a new image, with a new noise. The img_list was combining the generated images into one image.
However, this code still only generates 64 by 64 pixel images.


Instruction: How do you know when your learning rate should be reduced?

Input:
The question is more of a general question on tips on how to know if it should be reduced. If you aren't getting good results, would a reduction in learning rate help?
Generally you have a start and and end and step down miles stones. How do you know the learning rate should not just go lower and lower? "Just try it" is probably the answer.
There was a paper where it equated learning rate with complexity. So as a model runs it's learning rate needs to be reduced to build more complex data patterns.
Any insights would be great.
Thanks.

Output:
Often it makes sense to reduce the learning rate when no more improvements can be achieved with the currently set learning rate. The whole process can also be automated. If you use Pytorch have a look here: https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
But it is important to find a good learning rate for the beginning. I often try it out for a few epochs. If you don't get good results, it may be due to the learning rate, but it doesn't have to be. It can also be too small as well as too large. If the loss values fluctuate strongly, it is probably too large and if they do not go down too small.


Instruction: Cannot install pyansys using pip install

Input:
I am on Pycharm and wish to install the pyansys package, but I keep getting this error:
Collecting pyansys
  Using cached pyansys-0.61.3.tar.gz (11 kB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [21 lines of output]

      *** PyAnsys has moved (and expanded!) ***

      To use PyAnsys you need to install the applicable packages for your
      product:

      MAPDL:

      - ``pip install ansys-mapdl-core``

      MAPDL Post-Processing:

      - ``pip install ansys-mapdl-reader``
      - ``pip install ansys-dpf-core``
      - ``pip install ansys-dpf-reader``

      PyAEDT

      - ``pip install pyaedt``

      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

So I went ahead and installed all but one of those packages. I also installed cython. The only package that cannot be installed is ansys-dpf-reader and here is its error message
ERROR: Could not find a version that satisfies the requirement ansys-pdf-reader (from versions: none)
ERROR: No matching distribution found for ansys-pdf-reader

What is going on?

Output:
So it seems like 'ansys-dpf-reader' was changed to 'ansys-dpf-post' and the 'pyansys' is not a pacakge anymore, but has now expanded into these 5 packages. Installing all of them will allow the normal use of the old pyansys.
https://docs.pyansys.com/


Instruction: How do I retrieve the weights estimated from a neural net using skorch?

Input:
I've trained a simple neural net using skorch to make it sklearn compatible and I would like to know how to retrieve the actual estimated weights.
Here's a replicable example of what I need.
The neural net presented here uses 10 features, has one hidden layer of 2 nodes, uses ReLu activation functions and linearly combines the output of the 2 nodes.
import torch
import numpy as np
from torch.autograd import Variable

# Create example data
np.random.seed(2022)
train_size = 1000
n_features= 10

X_train = np.random.rand(n_features, train_size).astype("float32")

l2_params_1 = np.random.rand(1,n_features).astype("float32")
l2_params_2 = np.random.rand(1,n_features).astype("float32")

l1_X = np.matmul(l2_params_1, X_train)
l2_X = np.matmul(l2_params_2, X_train)

y_train = l1_X + l2_X

# Defining my NN
class NNModule(torch.nn.Module):
    def __init__(self, in_features):
        super(NNModule, self).__init__()
        self.l1 =  torch.nn.Linear(in_features, 2)
        self.a1 = torch.nn.ReLU()
        self.l2 =  torch.nn.Linear(2, 1)

    def forward(self, x):
        x = self.l1(x)
        x = self.a1(x)

        return self.l2(x)

# Initialize the NN
torch.manual_seed(200)

model = NNModule(in_features = 10)

model.l1.weight.data.uniform_(0.0, 1.0)
model.l1.bias.data.uniform_(0.0, 1.0)

# Define criterion and optimizer
criterion = torch.nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)

# Train the NN
torch.manual_seed(200)

for epoch in range(100):
    inputs = Variable(torch.from_numpy(np.transpose(X_train)))
    labels = Variable(torch.from_numpy(np.transpose(y_train)))

    optimizer.zero_grad()

    outputs = model(inputs)

    loss = criterion(outputs, labels)

    loss.backward()

    optimizer.step()

The parameters at which I'm arriving are the following:
list(model.parameters())

[Output]:
[Parameter containing:
 tensor([[0.8997, 0.8345, 0.8284, 0.6950, 0.5949, 0.1217, 0.9067, 0.1824, 0.8272,
          0.2372],
         [0.7525, 0.6577, 0.4358, 0.6109, 0.8817, 0.5429, 0.5263, 0.7531, 0.1552,
          0.7066]], requires_grad=True),
 Parameter containing:
 tensor([0.6617, 0.1079], requires_grad=True),
 Parameter containing:
 tensor([[0.9225, 0.8339]], requires_grad=True),
 Parameter containing:
 tensor([0.0786], requires_grad=True)]

Now, to wrap my NNModule with skorch, I'm using this:
from skorch import NeuralNetRegressor

torch.manual_seed(200)
net = NeuralNetRegressor(
    module=NNModule(in_features=10),
    criterion=torch.nn.MSELoss,
    optimizer=torch.optim.SGD,
    optimizer__lr=0.01,
    max_epochs=100,
    verbose=0
)

net.fit(np.transpose(X_train), np.transpose(y_train))

And I'd like to retrieve the weights obtained in the training. I've used dir(net) to see if the weights are stored in any attributes to no avail.

Output:
To retrieve the weights one needs to output them like this:
list(net.module.parameters())


Instruction: Reshaping tensor 2d to 1d

Input:
tensor([[17,  0],
        [93,  0],
        [0,  0],
        [21,  0],
        [19,  0])

I want to remove 0 from this tensor, which is a two-dimensional array, and make it a one-dimensional array.
How can I make this tensor with the tensor below?
tensor([[17],
        [93],
        [0],
        [21],
        [19])

When using the code below, there is a problem that the existing zero disappears.
How should I fix this?
x = x.flatten()
x = x[x!=0]
x = np.reshape(x, ( -1, x.shape[0] ))

array([[17, 93, 21, 19]])

Output:
You can use slicing to index the 0 column
import torch

t = torch.tensor(
      [[17,  0],
       [93,  0],
       [0,  0],
       [21,  0],
       [19,  0]]
    )

print(t[:,0])

Output
tensor([17, 93,  0, 21, 19])

And if you want to keep it a 2D array then you can use numpy.reshape
import torch
import numpy as np

t = torch.tensor(
      [[17,  0],
       [93,  0],
       [0,  0],
       [21,  0],
       [19,  0]]
    )

print(np.reshape(t[:,0], (-1, 1)))

Output
array([[17],
       [93],
       [ 0],
       [21],
       [19]], dtype=int32)


Instruction: Converting PyTorch transforms.compose method into Keras

Input:
I understand that we use transforms.compose to transform images via torch.transforms. I want to do the same in Keras  and spending hours on internet I couldnt get how to write a method in keras that can do the same. Below is the Torch way:
# preprocessing
data_transform = transforms.Compose([
   transforms.ToTensor(),
   transforms.Normalize(mean=[.5], std=[.5])
])

Can someone please point me in the right direction.

Output:
It is a bit trivial in Tensorflow. Tensorflow recommends using the pre-processing/augmentation as part of the model itself.
I do not have your complete code but I assume you would be using tf.data.Dataset API to create your dataset. This is the recommended way of building the dataset in Tensorflow.
Having said that you can just prepend augmentation layers in your model.
# Following is the pre-processing pipeline for e.g
# Step 1: Image resizing.
# Step 2: Image rescaling.
# Step 3: Image normalization

# Having Data Augmentation as part of the input pipeline.
# Step 1: Random flip.
# Step 2: Random Rotate.

pre_processing_pipeline = tf.keras.Sequential([
  layers.Resizing(IMG_SIZE, IMG_SIZE),
  layers.Rescaling(1./255),
  layers.Normalization(mean=[.5], variance=[.5]),
])

data_augmentation = tf.keras.Sequential([
  layers.RandomFlip("horizontal_and_vertical"),
  layers.RandomRotation(0.2),
])

# Then add it to your model.
# This would be different in your case as you might be using a pre-trained model.

model = tf.keras.Sequential([
  # Add the preprocessing layers you created earlier.
  resize_and_rescale,
  data_augmentation,
  layers.Conv2D(16, 3, padding='same', activation='relu'),
  layers.MaxPooling2D(),
  # Rest of your model.
])

For a complete list of layers check out this link. The above-given code can be found on the website here.


Instruction: Failing to compile project using CUDA 11.0, Python 3.8, Torch 1.8

Input:
I am trying to compile DiffDVR, a differentiable renderer. This requires running cmake first, so I had to install some dependencies. I want to use the same dependencies as the ones in the repo. I'm on Linux Mint 20.3, which is based on Ubuntu 20.04.
In order, I have installed:

CUDA 11.0 (using the official runfile)
cuDNN v8.1.1 (the highest version compatible with 11.0, as per the cuDNN archive)
created a Python 3.8 environment with Torch 1.8.0 (based on the environment.yml file in the repo)
libglm-dev, libglfw3-dev and libglew-dev as they were needed for compilation.

The error I get when going to the build dir and running cmake .. is:
CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
TORCH_LIB_c10
    linked by target "Renderer" in directory /home/andrei/PycharmProjects/DiffDVR/renderer
    linked by target "Gui" in directory /home/andrei/PycharmProjects/DiffDVR/gui
TORCH_LIB_c10_cuda
    linked by target "Renderer" in directory /home/andrei/PycharmProjects/DiffDVR/renderer
    linked by target "Gui" in directory /home/andrei/PycharmProjects/DiffDVR/gui
TORCH_LIB_torch_cpu
    linked by target "Renderer" in directory /home/andrei/PycharmProjects/DiffDVR/renderer
    linked by target "Gui" in directory /home/andrei/PycharmProjects/DiffDVR/gui
TORCH_LIB_torch_cuda
    linked by target "Renderer" in directory /home/andrei/PycharmProjects/DiffDVR/renderer
    linked by target "Gui" in directory /home/andrei/PycharmProjects/DiffDVR/gui
TORCH_LIB_torch_python
    linked by target "Renderer" in directory /home/andrei/PycharmProjects/DiffDVR/renderer
    linked by target "Gui" in directory /home/andrei/PycharmProjects/DiffDVR/gui

Scrolling up in the output reveals the following problem:
Torch: full library list: /usr/lib/libtorch.so;TORCH_LIB_c10-NOTFOUND;TORCH_LIB_c10_cuda-NOTFOUND;/usr/lib/libtorch.so;TORCH_LIB_torch_cpu-NOTFOUND;TORCH_LIB_torch_cuda-NOTFOUND;TORCH_LIB_torch_python-NOTFOUND
So it seems like these libraries are not found. I tried to sudo apt-get install libtorch3-dev but apt says it's already installed (I assume it happened when conda installed the pytorch package?), and there are no libraries available with the c10, c10_cuda etc suffixes. How would I make sure these libraries are found, so I can compile the project?

I made some progress with the following (hacky) solution:
sudo ln -s /home/andrei/miniconda3/envs/py38torch18/lib/python3.8/site-packages/torch/lib/libc10.so libc10.so
sudo ln -s /home/andrei/miniconda3/envs/py38torch18/lib/python3.8/site-packages/torch/lib/libc10_cuda.so libc10_cuda.so
sudo ln -s /home/andrei/miniconda3/envs/py38torch18/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so libtorch_cpu.so
sudo ln -s /home/andrei/miniconda3/envs/py38torch18/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so libtorch_cuda.so
sudo ln -s /home/andrei/miniconda3/envs/py38torch18/lib/python3.8/site-packages/torch/lib/libtorch_python.so libtorch_python.so

Which fixed the previous errors.
Unfortunately, I'm now encountering the following error:
CMake Error at renderer/CMakeLists.txt:92 (add_library):
  CUDA_STANDARD is set to invalid value '17'

More progress:
Managed to run cmake successfully by uninstalling it via apt and reinstalling it via snap (because the apt version was too old). Unfortunately, make fails now, with the error:
[ 46%] Building CXX object renderer/CMakeFiles/Renderer.dir/volume.cpp.o
In file included from /opt/project/renderer/volume.cpp:1:
/opt/project/renderer/volume.h:9:10: fatal error: torch/types.h: No such file or directory
    9 | #include <torch/types.h>
      |          ^~~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [renderer/CMakeFiles/Renderer.dir/build.make:76: renderer/CMakeFiles/Renderer.dir/volume.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:244: renderer/CMakeFiles/Renderer.dir/all] Error 2
make: *** [Makefile:91: all] Error 2

Output:
Update: specifying all the paths manually worked. In my case, this was:
cmake .. -DTORCH_PATH=/home/andrei/miniconda3/envs/py38torch18/lib/python3.8/site-packages/torch -DTorch_DIR=/home/andrei/miniconda3/envs/py38torch18/lib/python3.8/site-packages/torch/share/cmake/Torch -DPYTHON_LIBRARY=/home/andrei/miniconda3/envs/py38torch18/lib/libpython3.8.so -DPYTHON_EXECUTABLE=/home/andrei/miniconda3/envs/py38torch18/bin/python


Instruction: Unable to load .pb while converting PyTorch model to tf.keras

Input:
Context
I'm using tf.keras for a personal project and I need to retrieve a pretrained Alexnet model.
Unfortunately, this model is not directly accessible using tf.keras only, so I downloaded the pretrained model using PyTorch, converted it into an onnx file and then exported it as a .pb file with the following code :
torch_pretrained = torchvision.models.alexnet()
torch_pretrained.load_state_dict(torch.load("alexnet.pth"))

dummy_input = Variable(torch.randn(1, 3, 224, 224))
torch.onnx.export(torch_pretrained, dummy_input, "alexnet_pretrained.onnx")

onnx_pretrained = onnx.load("alexnet_pretrained.onnx")
onnx_pretrained = prepare(onnx_pretrained)
onnx_pretrained.export_graph('alexnet')

Issue
I'm now trying to retrieve the .pb file using keras as explained here with the following code :
model = tf.keras.models.load_model("alexnet")
model.summary()

And i get an error :

AttributeError: '_UserObject' object has no attribute 'summary'

I also get a warning while loading the model, but I don't think it's relevant :

WARNING:tensorflow:SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), NOT tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.

The loaded model has a very obscure type as you can see :

<tensorflow.python.saved_model.load.Loader._recreate_base_user_object.._UserObject object at 0x0000023137981BB0>

While doing my researches, I found this which means I'm not the only one to face this issue.
Question
The easiest way would be to solve this specific issue, but if anyone knows of another way to load a pretrained Alexnet model to tf.keras, this would also solve my actual problem.
Specs
Windows 10
python 3.9.7
tensorflow 2.6.0
torch 1.10.2
torchvision 0.11.3
onnx 1.10.2
onnx-tf 1.9.0

Output:
Solution
I followed the suggestion of Jakub : I installed "pytorch2keras" (see this).
I just ran the function to convert directly the pytorch model into a keras model, and it actually worked.
I only had to modify the code of the module as there were some depencies issues (they are using onnx.optimizer, which is now called onnxoptimizer) so I just changed the import line in  :
from
from onnx import optimizer

to
import onnxoptimizer as optimizer


Instruction: What is validation set for yolov5 training and is it necessary?

Input: I'm training a yolov5 model and I have a dataset but I couldn't understand the differences between validation and test data. Should I split my dataset to train/test/val or just train/test?

Output: The validation dataset provides an unbiased assessment of a fitted model on the training dataset while fitting the model's hyper-parameters (e.g. the number of hidden units - layers and layer widths - in a neural network). Validation datasets can be used for early stop regularization (stopping training when the error in the validation dataset increases, as this is a sign of overfitting to the training dataset)


Instruction: Save MNIST dataset with added noise

Input:
I want to save the new MNIST dataset tensors after adding noise.
mnist_trainset = datasets.MNIST(root='./data', train=True, download=True, transform=transforms.ToTensor())

def add_noise(dataset):
  for data in dataset:
      img, _ = data[0], data[1]
      noisy_data = torch.tensor(random_noise(img, mode='gaussian', mean=0, var=0.05, clip=True))
      return noisy_data

train_gauss = add_noise(mnist_trainset)

The code above only saves one image of the MNIST dataset, how can I save all tensors at once?

Output:
Your return is inside the for loop, so at the end of the first iteration it already returns the noisy_data.
You should take it out of the for loop and then you could use a list to return the whole dataset at once, like so:
mnist_trainset = datasets.MNIST(root='./data', train=True, download=True, transform=transforms.ToTensor())

def add_noise(dataset):
  noisy_data = []
  for data in dataset:
      img, _ = data[0], data[1]
      noisy_data += torch.tensor(random_noise(img, mode='gaussian', mean=0, var=0.05, clip=True))
  return noisy_data

train_gauss = add_noise(mnist_trainset)


Instruction: RuntimeError: expected scalar type Float but found Double error torch.nn.CrossEntropyLoss Pytorch

Input:
I am trying to train a pytorch model. The loss function is:
cn_loss = torch.nn.CrossEntropyLoss(weight=train_label_weight, reduction='mean')

Code fragment from the training function:
for sents, targets in batch_iter(df_train, batch_size=train_batch_size, shuffle=True, bert=bert_size):
        train_iter += 1
        optimizer.zero_grad()
        batch_size = len(sents)
        pre_softmax = model(sents)
        float_targets=torch.tensor(targets, dtype=torch.float, device=device)

        loss = cn_loss(pre_softmax, float_targets)

        loss.backward()
        optimizer.step()

Data types of both pre_softmax and float_targets is torch.float32. (In the original code data type of the targets has been converted to torch.int64 using torch.tensor(targets, dtype=torch.long, device=device). However as I get the

RuntimeError: expected scalar type Float but found Double

error I converted the data type of targets to torch.float32)
Even though both parameters in cn_loss() function are torch.float32,  I get the below error when I run the code:

loss = cn_loss(pre_softmax, float_targets)
File "C:\Users\user\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "C:\Users\user\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\nn\modules\loss.py", line 1152, in forward
label_smoothing=self.label_smoothing)
File "C:\Users\user\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\nn\functional.py", line 2846, in cross_entropy
return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)
RuntimeError: expected scalar type Float but found Double

I checked the data types multiple times and changed the data type of float_targets using torch.FloatTensor(targets) also. But I get the same error.

Output: As pointed out by @aretor in comments, the data type of train_label_weight was torch.float64. When converted it to torch.float32 and changed targets to torch.long back again, the code worked perfectly


Instruction: How to translate the tensor type into numpy type?

Input:
import torch
x = torch.linspace(-5, 5, 200)
x_np = x.numpy()

it return:
UserWarning: Failed to initialize NumPy: No module named 'numpy.core._multiarray_umath' (Triggered internally at  ..\torch\csrc\utils\tensor_numpy.cpp:68.)
  _dtype_to_storage = {data_type(0).dtype: data_type for data_type in _storages}
RuntimeError: Numpy is not available

environment；
python3.7
pytorch==1.10.2

Output:
You need numpy installed
pip install numpy --upgrade


Instruction: Pytorch - vanilla Gradient Visualization

Input:
I trained a neural network on MNIST using PyTorch:
class MnistCNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d( 1, 16, 3, stride = 1, padding = 2)
        self.pool1 = nn.MaxPool2d(kernel_size = 2)
        self.conv2 = nn.Conv2d(16, 32, 3, stride = 1, padding = 2)
        self.pool2 = nn.MaxPool2d(kernel_size = 2)
        self.dropout = nn.Dropout(0.5)
        self.lin = nn.Linear(32 * 8 * 8, 10)

    def forward(self, x):
        # conv block
        x = F.relu(self.conv1(x))
        x = self.pool1(x)
        # conv block
        x = F.relu(self.conv2(x))
        x = self.pool2(x)
        # dense block
        x = x.view(x.size(0), -1)
        x = self.dropout(x)
        return self.lin(x)

I would like to implement vanilla Gradient Visualization (see reference below) on my model.
Simonyan, K., Vedaldi, A., Zisserman, A.
Deep inside convolutional networks: Visualising image classification models and saliency maps.
arXiv preprint arXiv:1312.6034 (2013)

Question: How can I implement this method in PyTorch?
If I understand correctly, vanilla gradient visualization consists in computing the partial derivatives of the loss of my model w.r.t all the pixels in my input image. So to make it short, I need to tweek my self.conv1 layer so that it computes the gradient over its input pixels instead of the gradient over its weights.
Please correct me if I'm wrong.

Output:
You do not need to change anything about your conv layer. Each layer computes gradients both w.r.t. parameters (for updates) and w.r.t. inputs (for "downstream" gradients by the chain rule). Therefore, all you need is to set your input image's x gradient property to true:
x, y = ... # get one image from MNIST
x.requires_grad_(True)  # indicate to pytorch that you would like to look at these gradients
pred = model(x)
loss = criterion(pred, y)
loss.backward()  # propagate gradients
x.grad  # <- here you should have the gradients of the loss w.r.t pixels


Instruction: How to define a normal distribution using tensors?

Input:
I want to define multivariate normal distribution with mean [0, 0, 0, 0] and variance - covariance matrix with only ones on diagonal. My code was:
import torch
import torch.distributions as td
td.Normal(loc=torch.tensor(np.zeros(4)), scale=torch.tensor(np.diag(np.repeat(1, 4)), dtype = float))

But I obtain Value Error:
ValueError: The parameter scale has invalid values

what should I do for proper definition?

Output:
torch.distributions.Normal is the univariate normal distribution.  It handles tensor arguments by computing the results as an array of univariate distributions (using broadcasting).
For example, in the following, normal implements two univariate normal distributions, one with mean 0 and std. dev 1, and the other with mean 9 and std. dev. 0.25:
In [89]: normal = td.Normal(loc=torch.tensor([0.0, 9.0]),
    ...:                    scale=torch.tensor([1, 0.25]))

Draw 5 samples from each distribution:
In [90]: normal.rsample(sample_shape=[5])
Out[90]:
tensor([[ 0.2460,  8.6748],
        [-0.8655,  9.0666],
        [ 0.6386,  8.7980],
        [ 0.5817,  9.1428],
        [-0.1418,  8.5777]], dtype=torch.float64)

For the multivariate normal distribution, use torch.distributions.multivariate_normal.MultivariateNormal.
For example,
In [98]: mvn = td.multivariate_normal.MultivariateNormal(
    ...:     loc=torch.zeros(4, dtype=torch.float64),
    ...:     covariance_matrix=torch.eye(4, dtype=torch.float64))

In [99]: mvn.rsample([5])
Out[99]:
tensor([[ 0.6445,  1.3138, -2.1718,  1.1653],
        [-1.7391, -1.0428,  0.3636,  0.1957],
        [ 1.9167, -0.2738, -0.1590,  0.7170],
        [ 0.0691, -0.5422,  1.8694,  0.7470],
        [ 0.3545,  1.0585, -0.1825, -1.9955]], dtype=torch.float64)


Instruction: How to define plenty of diagonal matricies?

Input:
Let's consider tensor:
scale = torch.tensor([[1.0824, 1.0296, 1.0065, 0.9395, 0.9424, 1.0260, 0.9805, 1.0509],
        [1.1002, 1.0358, 1.0112, 0.9466, 0.9454, 0.9942, 0.9891, 1.0485],
        [1.1060, 1.0157, 1.0216, 0.9544, 0.9378, 1.0160, 0.9671, 1.0240]])

which has shape :
scale.shape
torch.Size([3, 8])

I want to have a tensor of shape [3, 8, 8] where in which I have three diagonal matricies using values from tensor scale. In other words, first matrix will have diagonals only using scale[0], second one scale[1] and last one scale[2].
We can do it brainless:
import torch
temp = torch.tensor([])
for i in range(0, 3):
    temp = torch.cat([temp, torch.diag(scale[i])])
temp = temp.view(3, 8, 8)
temp

But I'm wondering if there is any other more efficient way to do this.

Output:
I think you are looking for diag_embed:
temp = torch.diag_embed(scale)

For example:
scale = torch.arange(24).view(3,8)
torch.diag_embed(scale)
tensor([[[ 0,  0,  0,  0,  0,  0,  0,  0],
         [ 0,  1,  0,  0,  0,  0,  0,  0],
         [ 0,  0,  2,  0,  0,  0,  0,  0],
         [ 0,  0,  0,  3,  0,  0,  0,  0],
         [ 0,  0,  0,  0,  4,  0,  0,  0],
         [ 0,  0,  0,  0,  0,  5,  0,  0],
         [ 0,  0,  0,  0,  0,  0,  6,  0],
         [ 0,  0,  0,  0,  0,  0,  0,  7]],

        [[ 8,  0,  0,  0,  0,  0,  0,  0],
         [ 0,  9,  0,  0,  0,  0,  0,  0],
         [ 0,  0, 10,  0,  0,  0,  0,  0],
         [ 0,  0,  0, 11,  0,  0,  0,  0],
         [ 0,  0,  0,  0, 12,  0,  0,  0],
         [ 0,  0,  0,  0,  0, 13,  0,  0],
         [ 0,  0,  0,  0,  0,  0, 14,  0],
         [ 0,  0,  0,  0,  0,  0,  0, 15]],

        [[16,  0,  0,  0,  0,  0,  0,  0],
         [ 0, 17,  0,  0,  0,  0,  0,  0],
         [ 0,  0, 18,  0,  0,  0,  0,  0],
         [ 0,  0,  0, 19,  0,  0,  0,  0],
         [ 0,  0,  0,  0, 20,  0,  0,  0],
         [ 0,  0,  0,  0,  0, 21,  0,  0],
         [ 0,  0,  0,  0,  0,  0, 22,  0],
         [ 0,  0,  0,  0,  0,  0,  0, 23]]])

If you insist on using a loop and torch.cat, you can use a list comprehension:
temp = torch.stack([torch.diag(s_) for s_ in scale])


Instruction: How to understand torch.arange(0, 3).view(-1, *[1]*3)

Input:
In Pytorch, for the code:
torch.arange(0, 3).view(-1, *[1]*3)

The result is:
tensor([[[[0]]],

    [[[1]]],

    [[[2]]]])

    torch.Size([3, 1, 1, 1])

Where [1] * 3 = [1, 1, 1], but I don`t understand the * before [1] * 3. What is the meaning of it? Thanks.

Output:
While links provided in the comments describe parts of the solution, whole thing might be missing, hence, let’s disentangle this view method:
.view(-1,...)

Means “all the elements”, in your case it is 3 as you have [0, 1, 2] with length of 3.
Next:
[1] * 3

Is a Python trick to create new list with single element repeated multiple times.
It is the same as
[1, 1, 1]

Now unpacking with asterisk “unpacks” values as arguments to function, in this case:
.view(-1, [1, 1, 1])

Becomes:
.view(-1, 1, 1, 1)

And the whole thing is (according to first step):
.view(3, 1, 1, 1)

BTW. Please don't do that under most circumstances, it’s pretty hard to follow as one can see above.


Instruction: Pytorch's gather, sequeeze and unsqueeze to Tensorflow Keras

Input:
I am migrating a code from pytorch to tensorflow, and in the function that calculates the loss, I have the below line that I need to migrate to tensorflow.
state_action_values = net(t_states_features).gather(1, actions_v.unsqueeze(-1)).squeeze(-1)

I found tf.gather and tf.gather_nd and I am not sure which is more suitable and how it could be used, also unsqueeze's alternative is maybe tf.expand_dims?
In an attempt to get a clearer view of the line's result, I split it into multiple parts with print statements.
  print("net result")
  state_action_values = net(t_states_features)
  print(state_action_values)
  print("gather result")
  state_action_values = state_action_values.gather(1, actions_v.unsqueeze(-1))
  print(state_action_values)
  print("last squeeze")
  state_action_values = state_action_values.squeeze(-1)

net result
tensor([[ 45.6878, -14.9495,  59.3737],
        [ 33.5737, -10.4617,  39.0078],
        [ 67.7197, -22.8818,  85.7977],
        [ 94.7701, -33.2053, 120.5519],
        [     nan,      nan,      nan],
        [ 84.7324, -29.2101, 108.0821],
        [ 67.7193, -22.7702,  86.9558],
        [113.6835, -38.7149, 142.6167],
        [ 61.9260, -20.1968,  79.8010],
        [ 51.6152, -17.7391,  66.0719],
        [ 73.6565, -21.5699,  98.9463],
        [ 84.0761, -26.5016, 107.6888],
        [ 60.9459, -20.1257,  76.4105],
        [103.2883, -35.4035, 130.4503],
        [ 37.1156, -13.5180,  47.1067],
        [     nan,      nan,      nan],
        [ 55.6286, -18.5239,  71.9837],
        [ 55.3858, -18.7892,  71.1197],
        [ 50.2419, -17.2959,  66.7059],
        [ 82.5715, -30.0302, 108.4984],
        [ -0.8662,  -1.1861,   1.6033],
        [112.4620, -38.6416, 142.4556],
        [ 57.8702, -19.8080,  74.7656],
        [ 45.8418, -15.7436,  57.3367],
        [ 81.6596, -27.5002, 104.6002],
        [ 57.1507, -21.8001,  67.7933],
        [ 35.0414, -11.8199,  47.6573],
        [ 67.7085, -23.1017,  85.4623],
        [ 40.6284, -12.4578,  58.9603],
        [ 68.6394, -23.1481,  87.0832],
        [ 27.0549,  -8.6635,  34.0150],
        [ 25.4071,  -8.5511,  34.0285],
        [ 62.9161, -22.1693,  78.7965],
        [ 85.4505, -28.1487, 108.6252],
        [ 67.6665, -23.2376,  85.7117],
        [ 60.7806, -20.2784,  77.1022],
        [ 66.5209, -21.5674,  88.5561],
        [ 61.6637, -20.9891,  72.3873],
        [ 45.1634, -15.4678,  61.4886],
        [ 66.8119, -23.1250,  85.6189],
        [     nan,      nan,      nan],
        [ 67.8166, -24.8342,  84.6706],
        [ 86.2114, -29.5941, 107.8025],
        [ 66.2716, -23.3309,  83.9700],
        [101.2122, -35.3554, 127.4772],
        [ 61.0749, -19.4720,  78.5588],
        [ 50.4058, -16.1262,  63.1010],
        [ 27.7543,  -9.3767,  35.7448],
        [ 67.7810, -23.4962,  83.6030],
        [ 35.0103, -11.7238,  44.7983],
        [ 55.7402, -19.0223,  70.3627],
        [ 67.9733, -22.0783,  85.1893],
        [ 60.5253, -20.3157,  79.7312],
        [ 67.2404, -21.5205,  81.4499],
        [ 57.9502, -20.7747,  70.9109],
        [ 87.6536, -31.4256, 112.6491],
        [ 90.3668, -30.7755, 116.6192],
        [ 59.0660, -19.6988,  75.0723],
        [ 50.0969, -17.4135,  62.6556],
        [ 28.8703,  -9.0950,  34.5749],
        [ 68.4053, -22.0715,  88.2302],
        [ 69.1397, -21.4236,  84.7833],
        [ 23.8506,  -8.1834,  30.8318],
        [ 58.4296, -20.2432,  73.8116],
        [ 87.5317, -29.0606, 110.0389],
        [     nan,      nan,      nan],
        [ 88.6387, -30.6154, 112.4239],
        [ 51.6089, -16.1073,  66.2757],
        [ 94.3989, -32.1473, 119.0358],
        [ 82.7449, -30.7778, 102.8537],
        [ 74.3067, -26.6585,  98.2536],
        [ 77.0881, -26.5706,  98.3553],
        [ 28.5688,  -9.2949,  41.1165],
        [ 86.1560, -26.9364, 107.0244],
        [ 41.8914, -16.9703,  57.3840],
        [ 88.8886, -29.7008, 108.2697],
        [ 61.1243, -20.7566,  77.2257],
        [ 85.1174, -28.7558, 107.3853],
        [ 81.7256, -27.9047, 104.5006],
        [ 51.2663, -16.5880,  67.1428],
        [ 46.9150, -12.7457,  61.3240],
        [ 36.1758, -12.9769,  47.7178],
        [ 85.5846, -29.4141, 107.9649],
        [ 59.9424, -20.8349,  75.3359],
        [ 62.6516, -22.1235,  81.6903],
        [104.7664, -34.5876, 129.9478],
        [ 64.4671, -23.3980,  83.9093],
        [ 69.6928, -23.6567,  89.6024],
        [ 60.4407, -19.6136,  75.9350],
        [ 33.4921, -10.3434,  44.9537],
        [ 57.9112, -19.4174,  74.3050],
        [ 24.8262,  -9.3637,  30.1057],
        [ 85.3776, -28.9097, 110.1310],
        [ 63.8175, -22.3843,  81.0308],
        [ 34.6040, -12.3217,  46.0356],
        [ 88.3740, -29.5049, 110.2897],
        [ 66.8196, -22.5860,  85.5386],
        [ 58.9767, -22.0601,  78.7086],
        [ 83.2090, -26.3499, 113.5105],
        [ 54.8450, -17.7980,  68.1161],
        [     nan,      nan,      nan],
        [ 85.0846, -29.2494, 107.6780],
        [ 76.9251, -26.2295,  98.4755],
        [ 98.2907, -32.8878, 124.9192],
        [ 91.1387, -30.8262, 115.3978],
        [ 73.1062, -24.9450,  90.0967],
        [ 27.6564,  -8.6114,  35.4470],
        [ 71.8508, -25.1529,  95.5165],
        [ 69.7275, -20.1357,  86.9620],
        [ 67.0907, -21.9245,  84.8853],
        [ 77.3163, -25.5980,  92.7700],
        [ 63.0082, -21.0345,  78.7311],
        [ 68.0553, -22.4280,  84.8031],
        [  5.8148,  -2.3171,   8.0620],
        [103.3399, -35.1769, 130.7801],
        [ 54.8769, -18.6822,  70.4657],
        [ 58.4446, -18.9764,  75.5509],
        [ 91.0071, -31.2706, 112.6401],
        [ 84.6577, -29.2644, 104.6046],
        [ 45.4887, -15.8309,  59.0498],
        [ 56.3384, -18.9264,  78.8834],
        [ 63.5109, -21.3169,  81.5144],
        [ 79.4635, -29.8681, 100.5056],
        [ 27.6559, -10.0517,  35.6012],
        [ 76.3909, -24.1689,  93.6133],
        [ 34.3802, -11.5272,  45.8650],
        [ 60.3553, -20.1693,  76.5371],
        [ 56.0590, -18.6468,  69.8981]], grad_fn=<AddmmBackward0>)
gather result
tensor([[ 59.3737],
        [-10.4617],
        [ 67.7197],
        [ 94.7701],
        [     nan],
        [-29.2101],
        [ 67.7193],
        [-38.7149],
        [-20.1968],
        [ 66.0719],
        [ 98.9463],
        [107.6888],
        [-20.1257],
        [-35.4035],
        [ 47.1067],
        [     nan],
        [ 55.6286],
        [-18.7892],
        [ 66.7059],
        [-30.0302],
        [  1.6033],
        [112.4620],
        [ 74.7656],
        [-15.7436],
        [ 81.6596],
        [-21.8001],
        [ 35.0414],
        [-23.1017],
        [ 40.6284],
        [ 68.6394],
        [ 34.0150],
        [ 34.0285],
        [ 78.7965],
        [-28.1487],
        [ 67.6665],
        [-20.2784],
        [-21.5674],
        [ 72.3873],
        [-15.4678],
        [ 85.6189],
        [     nan],
        [-24.8342],
        [-29.5941],
        [-23.3309],
        [101.2122],
        [-19.4720],
        [-16.1262],
        [ -9.3767],
        [-23.4962],
        [-11.7238],
        [ 70.3627],
        [-22.0783],
        [-20.3157],
        [ 67.2404],
        [-20.7747],
        [112.6491],
        [-30.7755],
        [-19.6988],
        [ 50.0969],
        [ 34.5749],
        [ 88.2302],
        [-21.4236],
        [ -8.1834],
        [ 73.8116],
        [110.0389],
        [     nan],
        [112.4239],
        [-16.1073],
        [-32.1473],
        [-30.7778],
        [ 98.2536],
        [ 98.3553],
        [ 28.5688],
        [107.0244],
        [-16.9703],
        [-29.7008],
        [ 77.2257],
        [-28.7558],
        [-27.9047],
        [ 67.1428],
        [-12.7457],
        [ 47.7178],
        [-29.4141],
        [ 59.9424],
        [-22.1235],
        [129.9478],
        [-23.3980],
        [-23.6567],
        [ 75.9350],
        [-10.3434],
        [-19.4174],
        [ 30.1057],
        [ 85.3776],
        [ 63.8175],
        [ 46.0356],
        [-29.5049],
        [-22.5860],
        [-22.0601],
        [113.5105],
        [-17.7980],
        [     nan],
        [-29.2494],
        [ 76.9251],
        [-32.8878],
        [115.3978],
        [-24.9450],
        [ 35.4470],
        [ 95.5165],
        [ 86.9620],
        [-21.9245],
        [-25.5980],
        [ 78.7311],
        [-22.4280],
        [  5.8148],
        [103.3399],
        [ 70.4657],
        [ 58.4446],
        [ 91.0071],
        [104.6046],
        [ 45.4887],
        [-18.9264],
        [ 63.5109],
        [ 79.4635],
        [-10.0517],
        [ 76.3909],
        [ 34.3802],
        [-20.1693],
        [-18.6468]], grad_fn=<GatherBackward0>)
last squeeze
tensor([ 59.3737, -10.4617,  67.7197,  94.7701,      nan, -29.2101,  67.7193,
        -38.7149, -20.1968,  66.0719,  98.9463, 107.6888, -20.1257, -35.4035,
         47.1067,      nan,  55.6286, -18.7892,  66.7059, -30.0302,   1.6033,
        112.4620,  74.7656, -15.7436,  81.6596, -21.8001,  35.0414, -23.1017,
         40.6284,  68.6394,  34.0150,  34.0285,  78.7965, -28.1487,  67.6665,
        -20.2784, -21.5674,  72.3873, -15.4678,  85.6189,      nan, -24.8342,
        -29.5941, -23.3309, 101.2122, -19.4720, -16.1262,  -9.3767, -23.4962,
        -11.7238,  70.3627, -22.0783, -20.3157,  67.2404, -20.7747, 112.6491,
        -30.7755, -19.6988,  50.0969,  34.5749,  88.2302, -21.4236,  -8.1834,
         73.8116, 110.0389,      nan, 112.4239, -16.1073, -32.1473, -30.7778,
         98.2536,  98.3553,  28.5688, 107.0244, -16.9703, -29.7008,  77.2257,
        -28.7558, -27.9047,  67.1428, -12.7457,  47.7178, -29.4141,  59.9424,
        -22.1235, 129.9478, -23.3980, -23.6567,  75.9350, -10.3434, -19.4174,
         30.1057,  85.3776,  63.8175,  46.0356, -29.5049, -22.5860, -22.0601,
        113.5105, -17.7980,      nan, -29.2494,  76.9251, -32.8878, 115.3978,
        -24.9450,  35.4470,  95.5165,  86.9620, -21.9245, -25.5980,  78.7311,
        -22.4280,   5.8148, 103.3399,  70.4657,  58.4446,  91.0071, 104.6046,
         45.4887, -18.9264,  63.5109,  79.4635, -10.0517,  76.3909,  34.3802,
        -20.1693, -18.6468], grad_fn=<SqueezeBackward1>)

Edit 1: print of actions_v
actions_v
tensor([2, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 0, 1, 0, 1, 1, 1, 2, 0, 2, 0, 1,
        2, 0, 2, 1, 1, 0, 2, 1, 0, 0, 2, 1, 1, 1, 0, 1, 0, 1, 1, 2, 1, 1, 2, 1,
        0, 2, 1, 2, 0, 2, 2, 0, 0, 1, 2, 0, 1, 2, 0, 0, 1, 1, 2, 0, 0, 2, 0, 0,
        1, 1, 2, 0, 1, 0, 2, 0, 1, 0, 0, 0, 0, 1, 2, 0, 2, 0, 1, 1, 2, 1, 2, 2,
        2, 0, 1, 1, 0, 0, 0, 1, 0, 1, 0, 1, 0, 0, 0, 0, 2, 1, 1, 0, 1, 0, 1, 2,
        2, 1, 0, 2, 0, 0, 2, 1])

Output:
gather_nd takes inputs that have the same dimension as the input tensor, and will output a tensor of values being at those indices (which is what you want).
gather will output slices (but you can give as indice shape whatever you want, the output tensor will just be a bunch of slices that are structured accordingly to the shape of indices) which is not what you want.
So you should first make the indices match the dimensions of the initial matrix:
indices = tf.transpose(tf.stack((tf.range(tf.shape(state_action_values)[0]),actions_v)))

And then gather_nd:
state_action_values  = tf.gather_nd(state_action_values,indices)

Keivan


Instruction: How to fix "Boolean value of Tensor with more than one value is ambiguous"?

Input:
I am trying to create a Model, but i'm getting a RuntimeError "Boolean value of Tensor with more than one value is ambiguous". I've searched already on other posts about this but could find a working solution so here is my own try.
I am following a tutorial, which is working with the tutorial set of Data.
The code looks as follows:
#%% import file

df = pd.read_csv('coordinates.csv', sep='\;', engine='python')
df

#%% cleanup

df=df[[ 'CATEGORY','LEFT_ANGLE']]
print(df)
df.apply(pd.to_numeric)

#%%

from tkinter import Variable
import numpy as np
from sklearn.preprocessing import MinMaxScaler

def sliding_windows(data, seq_length):
    x = []
    y = []

    for i in range(len(data)-seq_length-1):
        _x = data[i:(i+seq_length)]
        _y = data[i+seq_length]
        x.append(_x)
        y.append(_y)

    return np.array(x),np.array(y)

#%%

sc = MinMaxScaler()
training_data = sc.fit_transform(df)

seq_length = 4
x, y = sliding_windows(training_data, seq_length)

train_size = int(len(y) * 0.67)
test_size = len(y) - train_size

dataX = Variable(torch.Tensor(np.array(x)))
dataY = Variable(torch.Tensor(np.array(y)))

trainX = Variable(torch.Tensor(np.array(x[0:train_size])))
trainY = Variable(torch.Tensor(np.array(y[0:train_size])))

testX = Variable(torch.Tensor(np.array(x[train_size:len(x)])))
testY = Variable(torch.Tensor(np.array(y[train_size:len(y)])))

This is the DataFrame i want to use for training and the error:
[300 rows x 2 columns]
6
CATEGORY    LEFT_ANGLE
0   0   160
1   0   162
2   0   160
3   0   157
4   0   146
... ... ...
295 4   163
296 4   176
297 4   132
298 4   150
299 4   176
300 rows × 2 columns

Cleanup done

---------------------------------------------------------------------------

RuntimeError                              Traceback (most recent call last)

<ipython-input-12-2bfe2346f838> in <module>
      8 test_size = len(y) - train_size
      9
---> 10 dataX = Variable(torch.Tensor(np.array(x)))
     11 dataY = Variable(torch.Tensor(np.array(y)))
     12

    331             raise TypeError("name must be a string")
    332         global _varnum
--> 333         if not master:
    334             master = _default_root
    335         self._root = master._root()

RuntimeError: Boolean value of Tensor with more than one value is ambiguous

In the first place i dont understand what the class Variable is for and why the Tensor does not accept my array. I hope someone can help me.
Thanks in advance and sorry if my englisch is not the best.

Output:
Short answer:
Remove the Variable()
Explanation:
I believe your answer is in you question. If you don't know what Variable is, why are you using it? But more importantly you are importing it from tkinter which is an interface package and I'm pretty sure that's not what you want.
What you want is to use the one from torch.
I was looking for it in the doc but it is actually deprecated, the Variable() API from torch now returns a Tensor so it is not useful anymore.
see here


Instruction: Pytorch : different behaviours in GAN training with different, but conceptually equivalent, code

Input:
I'm trying to implement a simple GAN in Pytorch. The following training code works:
    for epoch in range(max_epochs):  # loop over the dataset multiple times
        print(f'epoch: {epoch}')
        running_loss = 0.0

        for batch_idx,(data,_) in enumerate(data_gen_fn):

            # data preparation
            real_data            = data
            input_shape          = real_data.shape
            inputs_generator     = torch.randn(*input_shape).detach()

            # generator forward
            fake_data            = generator(inputs_generator).detach()
            # discriminator forward
            optimizer_generator.zero_grad()
            optimizer_discriminator.zero_grad()

            #################### ALERT CODE #######################
            predictions_on_real = discriminator(real_data)
            predictions_on_fake = discriminator(fake_data)

            predictions = torch.cat((predictions_on_real,
                                     predictions_on_fake), dim=0)
           #########################################################

            # loss discriminator
            labels_real_fake           = torch.tensor([1]*batch_size + [0]*batch_size)
            loss_discriminator_batch   = criterion_discriminator(predictions,
                                                          labels_real_fake)
            # update discriminator
            loss_discriminator_batch.backward()
            optimizer_discriminator.step()

            # generator
            # zero the parameter gradients
            optimizer_discriminator.zero_grad()
            optimizer_generator.zero_grad()

            fake_data            = generator(inputs_generator) # make again fake data but without detaching
            predictions_on_fake  = discriminator(fake_data) # D(G(encoding))

            # loss generator
            labels_fake          = torch.tensor([1]*batch_size)
            loss_generator_batch = criterion_generator(predictions_on_fake,
                                                       labels_fake)

            loss_generator_batch.backward()  # dL(D(G(encoding)))/dW_{G,D}
            optimizer_generator.step()

If I plot the generated images for each iteration, I see that the generated images look like the real ones, so the training procedure seems to work well.
However, if I try to change the code in the ALERT CODE part , i.e., instead of:
   #################### ALERT CODE #######################
   predictions_on_real = discriminator(real_data)
   predictions_on_fake = discriminator(fake_data)

   predictions = torch.cat((predictions_on_real,
                            predictions_on_fake), dim=0)
   #########################################################

I use the following:
   #################### ALERT CODE #######################
   predictions = discriminator(torch.cat( (real_data, fake_data), dim=0))
   #######################################################

That is conceptually the same (in a nutshell, instead of doing two different forward on the discriminator, the former on the real, the latter on the fake data, and  finally concatenate the results, with the new code I first concatenate real and fake data, and finally I make just one forward pass on the concatenated data.
However, this code version does not work, that is the generated images seems to be always random noise.
Any explanation to this behavior?

Output:
Why do we different results?
Supplying inputs in either the same batch, or separate batches, can make a difference if the model includes dependencies between different elements of the batch. By far the most common source in current deep learning models is batch normalization. As you mentioned, the discriminator does include batchnorm, so this is likely the reason for different behaviors. Here is an example. Using single numbers and a batch size of 4:
features = [1., 2., 5., 6.]
print("mean {}, std {}".format(np.mean(features), np.std(features)))

print("normalized features", (features - np.mean(features)) / np.std(features))

>>>mean 3.5, std 2.0615528128088303
>>>normalized features [-1.21267813 -0.72760688  0.72760688  1.21267813]

Now we split the batch into two parts. First part:
features = [1., 2.]
print("mean {}, std {}".format(np.mean(features), np.std(features)))

print("normalized features", (features - np.mean(features)) / np.std(features))

>>>mean 1.5, std 0.5
>>>normalized features [-1.  1.]

Second part:
features = [5., 6.]
print("mean {}, std {}".format(np.mean(features), np.std(features)))

print("normalized features", (features - np.mean(features)) / np.std(features))

>>>mean 5.5, std 0.5
>>>normalized features [-1.  1.]

As we can see, in the split-batch version, the two batches are normalized to the exact same numbers, even though the inputs are very different. In the joint-batch version, on the other hand, the larger numbers are still larger than the smaller ones as they are normalized using the same statistics.
Why does this matter?
With deep learning, it's always hard to say, and especially with GANs and their complex training dynamics. A possible explanation is that, as we can see in the example above, the separate batches result in more similar features after normalization even if the original inputs are quite different. This may help early in training, as the generator tends to output "garbage" which has very different statistics from real data.
With a joint batch, these differing statistics make it easy for the discriminator to tell the real and generated data apart, and we end up in a situation where the discriminator "overpowers" the generator.
By using separate batches, however, the different normalizations result in the generated and real data to look more similar, which makes the task less trivial for the discriminator and allows the generator to learn.


Instruction: Pytorch error: RuntimeError: 1D target tensor expected, multi-target not supported

Input:
I am currently working on an neuronal network that can classify cats and dog and everything thats not cat nor dog. And my programm has this: error i can't solve:
" File "/home/johann/Schreibtisch/NN_v0.01/classification.py", line 146, in
train(epoch)
File "/home/johann/Schreibtisch/NN_v0.01/classification.py", line 109, in train
loss = criterion(out, target)
File "/home/johann/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/johann/.local/lib/python3.8/site-packages/torch/nn/modules/loss.py", line 1047, in forward
return F.cross_entropy(input, target, weight=self.weight,
File "/home/johann/.local/lib/python3.8/site-packages/torch/nn/functional.py", line 2693, in cross_entropy
return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
File "/home/johann/.local/lib/python3.8/site-packages/torch/nn/functional.py", line 2388, in nll_loss
ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
RuntimeError: 1D target tensor expected, multi-target not supported"
The code:
import os
from torchvision import transforms
from PIL import Image
from os import listdir
import random
import torch.optim as optim
from torch.autograd import Variable
import torch.nn.functional as F
import torch.nn as nn
import matplotlib.pyplot as plt
from tqdm import tqdm

normalize = transforms.Normalize(
    mean = [0.485, 0.456, 0.406],
    std = [0.229, 0.224, 0.225])

transform = transforms.Compose([
     transforms.Resize(256),
     transforms.CenterCrop(256),
     transforms.ToTensor(), normalize])

train_data_list = []
target_list = []
train_data = []
waited = False
files = listdir('catsanddogs/train/')
for i in range(len(listdir('catsanddogs/train/'))):
    if len(train_data) == 58 and not waited:
        waited = True
        continue
    f = random.choice(files)
    files.remove(f)
    img = Image.open("catsanddogs/train/" + f)
    img_tensor = transform(img)
    train_data_list.append(img_tensor)
    isSomething = 0
    isCat = 1 if 'cat' in f else 0
    isDog = 1 if 'dog' in f else 0
    if isDog == 0 and isCat == 0:
        isSomething = 1
    target = [isCat, isDog, isSomething] #, isSomthing
    target_list.append(target)
    if len(train_data_list) >= 256:
        train_data.append((torch.stack(train_data_list), target_list))
        train_data_list = []
        target_list = []
        print('Loaded batch ', len(train_data), 'of ', int(len(listdir('catsanddogs/train/')) / 64))
        print('Percentage Done: ', 100 * len(train_data) / int(len(listdir('catsanddogs/train/')) / 64), '%')
        if len(train_data) > 2:
            break

class Netz(nn.Module):
    def __init__(self):
        super(Netz, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, kernel_size=(5, 5))
        self.conv2 = nn.Conv2d(6, 12, kernel_size=(5, 5))
        self.conv3 = nn.Conv2d(12, 18, kernel_size=(5, 5))
        self.conv4 = nn.Conv2d(18, 24, kernel_size=(5, 5))
        self.fc1 = nn.Linear(3456, 1000)
        self.fc2 = nn.Linear(1000, 3)

    def forward(self, x):
       x = self.conv1(x)
       x = F.max_pool2d(x, 2)
       x = F.relu(x)
       x = self.conv2(x)
       x = F.max_pool2d(x, 2)
       x = F.relu(x)
       x = self.conv3(x)
       x = F.max_pool2d(x, 2)
       x = F.relu(x)
       x = self.conv4(x)
       x = F.max_pool2d(x, 2)
       x = F.relu(x)
       x = x.view(-1, 3456)
       x = F.relu(self.fc1(x))
       x = self.fc2(x)
       return F.softmax(x, dim=1)

model = Netz()
#if os.path.isfile('catdognetz.pt'):
    #model = torch.load('catdognetz.pt')

optimizer = optim.SGD(model.parameters(), lr=0.001, momentum=0.9)

def count_parameters(model):
  return sum(p.numel() for p in model.parameters() if p.requires_grad)

count_parameters(model)

train_losses = []
train_accu = []

def train(epoch):
    print('\nEpoch : %d' % epoch)
    model.train()
    running_loss = 0
    correct = 0
    total = 0
    for data, target in tqdm(train_data):
        target = torch.Tensor(target)
        data = Variable(data)
        target = Variable(target)
        optimizer.zero_grad()
        out = model(data)
        criterion = nn.CrossEntropyLoss()
        loss = criterion(out, target)
        loss.backward()
        optimizer.step()

        running_loss += loss.item()
        predicted = out.max(1)
        total += target.size(0)
        correct += predicted.eq(target).sum().item()

    train_loss = running_loss / len(train_data)
    accu = 100. * float(correct) / total

    train_accu.append(accu)
    train_losses.append(train_loss)
    print('Train Loss: %.3f | Accuracy: %.3f' % (train_loss, accu))

def test():
    model.eval()
    file = listdir('catsanddogs/test/')
    f = random.choice(file)
    img = Image.open('catsanddogs/test/' + f)
    img_eval_tensor = transform(img)
    img_eval_tensor.unsqueeze_(0)
    data = Variable(img_eval_tensor)
    out = model(data)
    what = "It's a cat"
    print(out.data.max(1, keepdim=True)[1].cpu().numpy()[0])
    if out.data.max(1, keepdim=True)[1].cpu().numpy()[0] != 0:
        what = "It's a Dog"
    if out.data.max(1, keepdim=True)[1].cpu().numpy()[0] != 0 and out.data.max(1, keepdim=True)[1].cpu().numpy()[0] != 1:
        what = "Neither"
    print(what)
    img.show()
    #x = input('')

epochs = 10
for epoch in range(1, epochs + 1):
    train(epoch)
    #test()
    torch.save(model, 'catdognetz.pt')

plt.plot(train_accu,'-o')
plt.xlabel('epoch')
plt.ylabel('accuracy')
plt.legend(['Kurve: Training'])
plt.title('Train Accuracy')

plt.show()

plt.plot(train_losses,'-o')
plt.xlabel('epoch')
plt.ylabel('loss bzw. Verlust')
plt.legend(['Kruve : Training'])
plt.title('Train Loss')

plt.show()```

Output:
The reason behind this error is that your targets list are list of lists like that:
   target =  [[1,0,0],[0,1,0],[0,0,1],...]

You should use an 1D tensor instead of the one-hot code list because CrossEntropyLoss does not accept one-hot coded tensor. For example, you can use this:
target = torch.tensor([0, 1, 2, 2, 1, 0, 0, ...])    # cat = 1, dog=2, something else= 2

You can automatically convert your targets from a one-hot coded list to a tensor of classes by:
target = torch.tensor(target)
target= torch.argmax(target ,axis=1)

Or you can change your algorithm to create a 1D tensor instead of a list of lists.
You can read more the documentation of CrossEntropyLoss here


Instruction: Upsampling images in frequency domain using Pytorch

Input:
I'm trying to upsample an RGB image in the frequency domain, using Pytorch. I'm using this article for reference on grayscale images. Since Pytorch processes the channels individually, I figure the colorspace is irrelevant here.
The basic steps outlined by this article are:

Perform FFT on the image.

Pad the FFT with zeros.

Perform inverse FFT.

I wrote the following code for the same:
import torch
import cv2
import numpy as np

img = src = cv2.imread('orig.png')
torch_img = torch.from_numpy(img).to(torch.float32).permute(2, 0, 1) / 255.
fft = torch.fft.fft2(torch_img, norm="forward")
fr = fft.real
fi = fft.imag
fr = F.pad(fr, (fft.shape[-1]//2, fft.shape[-1]//2, fft.shape[-2]//2, fft.shape[-2]//2), mode='constant', value=0)
fi = F.pad(fi, (fft.shape[-1]//2, fft.shape[-1]//2, fft.shape[-2]//2, fft.shape[-2]//2), mode='constant', value=0)

fft_hires = torch.complex(fr, fi)
inv = torch.fft.ifft2(fft_hires, norm="forward").real

print(inv.max(), inv.min())
img = (inv.permute(1, 2, 0).detach()).clamp(0, 1)
img = (255 * img).numpy().astype(np.uint8)
cv2.imwrite('hires.png', img)

The original image:

The upscaled image:

Another interesting thing to note is the maximum and minimum values of the image pixels after performing IFFT: they are 2.2729 and -1.8376 respectively. Ideally, they should be 1.0 and 0.0.
Can someone please explain what's wrong here?

Output:
The usual convention for the DFT is to treat the first sample as 0Hz component. But you need to have the 0Hz component in the center in order for padding to make sense. Most FFT tools provide a shift function to circularly shift your result so that the 0Hz component is in the center. In pytorch you  need to perform torch.fft.fftshift after the FFT and torch.fft.ifftshift right before taking the inverse FFT to put the 0Hz component back in the upper left corner.
import torch
import torch.nn.functional as F
import cv2
import numpy as np

img = src = cv2.imread('orig.png')
torch_img = torch.from_numpy(img).to(torch.float32).permute(2, 0, 1) / 255.
# note the fftshift
fft = torch.fft.fftshift(torch.fft.fft2(torch_img, norm="forward"))

fr = fft.real
fi = fft.imag
fr = F.pad(fr, (fft.shape[-1]//2, fft.shape[-1]//2, fft.shape[-2]//2, fft.shape[-2]//2), mode='constant', value=0)
fi = F.pad(fi, (fft.shape[-1]//2, fft.shape[-1]//2, fft.shape[-2]//2, fft.shape[-2]//2), mode='constant', value=0)

# note the ifftshift
fft_hires = torch.fft.ifftshift(torch.complex(fr, fi))
inv = torch.fft.ifft2(fft_hires, norm="forward").real

print(inv.max(), inv.min())
img = (inv.permute(1, 2, 0).detach()).clamp(0, 1)
img = (255 * img).numpy().astype(np.uint8)
cv2.imwrite('hires.png', img)

which produces the following hires.png


Instruction: Using RNN Trained Model without pytorch installed

Input:
I have trained an RNN model with pytorch. I need to use the model for prediction in an environment where I'm unable to install pytorch because of some strange dependency issue with glibc. However, I can install numpy and scipy and other libraries. So, I want to use the trained model, with the network definition, without pytorch.
I have the weights of the model as I save the model with its state dict and weights in the standard way, but I can also save it using just json/pickle files or similar.
I also have the network definition, which depends on pytorch in a number of ways. This is my RNN network definition.
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import random

torch.manual_seed(1)
random.seed(1)
device = torch.device('cpu')

class RNN(nn.Module):
  def __init__(self, input_size, hidden_size, output_size,num_layers, matching_in_out=False, batch_size=1):
    super(RNN, self).__init__()
    self.input_size = input_size
    self.hidden_size = hidden_size
    self.output_size = output_size
    self.num_layers = num_layers
    self.batch_size = batch_size
    self.matching_in_out = matching_in_out #length of input vector matches the length of output vector
    self.lstm = nn.LSTM(input_size, hidden_size,num_layers)
    self.hidden2out = nn.Linear(hidden_size, output_size)
    self.hidden = self.init_hidden()
  def forward(self, feature_list):
    feature_list=torch.tensor(feature_list)

    if self.matching_in_out:
      lstm_out, _ = self.lstm( feature_list.view(len( feature_list), 1, -1))
      output_space = self.hidden2out(lstm_out.view(len( feature_list), -1))
      output_scores = torch.sigmoid(output_space) #we'll need to check if we need this sigmoid
      return output_scores #output_scores
    else:
      for i in range(len(feature_list)):
        cur_ft_tensor=feature_list[i]#.view([1,1,self.input_size])
        cur_ft_tensor=cur_ft_tensor.view([1,1,self.input_size])
        lstm_out, self.hidden = self.lstm(cur_ft_tensor, self.hidden)
        outs=self.hidden2out(lstm_out)
      return outs
  def init_hidden(self):
    #return torch.rand(self.num_layers, self.batch_size, self.hidden_size)
    return (torch.rand(self.num_layers, self.batch_size, self.hidden_size).to(device),
            torch.rand(self.num_layers, self.batch_size, self.hidden_size).to(device))

I am aware of this question, but I'm willing to go as low level as possible. I can work with numpy array instead of tensors, and reshape instead of view, and I don't need a device setting.
Based on the class definition above, what I can see here is that I only need the following components from torch to get an output from the forward function:

nn.LSTM
nn.Linear
torch.sigmoid

I think I can easily implement the sigmoid function using numpy. However, can I have some implementation for the nn.LSTM and nn.Linear using something not involving pytorch? Also, how will I use the weights from the state dict into the new class?
So, the question is, how can I "translate" this RNN definition into a class that doesn't need pytorch, and how to use the state dict weights for it?
Alternatively, is there a "light" version of pytorch, that I can use just to run the model and yield a result?
EDIT
I think it might be useful to include the numpy/scipy equivalent for both nn.LSTM and nn.linear. It would help us compare the numpy output to torch output for the same code, and give us some modular code/functions to use. Specifically, a numpy equivalent for the following would be great:
rnn = nn.LSTM(10, 20, 2)
input = torch.randn(5, 3, 10)
h0 = torch.randn(2, 3, 20)
c0 = torch.randn(2, 3, 20)
output, (hn, cn) = rnn(input, (h0, c0))

and also for linear:
m = nn.Linear(20, 30)
input = torch.randn(128, 20)
output = m(input)

Output:
Basically implementing it in numpy and copying weights from your pytorch model can do the trick.  For your usecase you will only need to do a forward pass so we just need to implement that only
#Set Parameters for a small LSTM network
input_size  = 2 # size of one 'event', or sample, in our batch of data
hidden_dim  = 3 # 3 cells in the LSTM layer
output_size = 1 # desired model output

num_layers=3
torch_lstm = RNN( input_size,
                 hidden_dim ,
                 output_size,
                 num_layers,
                 matching_in_out=True
                 )

state = torch_lstm.state_dict() # state will capture the weights of your model

Now for LSTM in numpy these functions will be used:
got the below code from this link: https://towardsdatascience.com/the-lstm-reference-card-6163ca98ae87
### NOT MY CODE
import numpy as np
from scipy.special import expit as sigmoid

def forget_gate(x, h, Weights_hf, Bias_hf, Weights_xf, Bias_xf, prev_cell_state):
    forget_hidden  = np.dot(Weights_hf, h) + Bias_hf
    forget_eventx  = np.dot(Weights_xf, x) + Bias_xf
    return np.multiply( sigmoid(forget_hidden + forget_eventx), prev_cell_state )

def input_gate(x, h, Weights_hi, Bias_hi, Weights_xi, Bias_xi, Weights_hl, Bias_hl, Weights_xl, Bias_xl):
    ignore_hidden  = np.dot(Weights_hi, h) + Bias_hi
    ignore_eventx  = np.dot(Weights_xi, x) + Bias_xi
    learn_hidden   = np.dot(Weights_hl, h) + Bias_hl
    learn_eventx   = np.dot(Weights_xl, x) + Bias_xl
    return np.multiply( sigmoid(ignore_eventx + ignore_hidden), np.tanh(learn_eventx + learn_hidden) )

def cell_state(forget_gate_output, input_gate_output):
    return forget_gate_output + input_gate_output

def output_gate(x, h, Weights_ho, Bias_ho, Weights_xo, Bias_xo, cell_state):
    out_hidden = np.dot(Weights_ho, h) + Bias_ho
    out_eventx = np.dot(Weights_xo, x) + Bias_xo
    return np.multiply( sigmoid(out_eventx + out_hidden), np.tanh(cell_state) )

We would need the sigmoid function as well so
def sigmoid(x):
    return 1/(1 + np.exp(-x))

Because pytorch stores weights in stacked manner so we need to break it up for that we would need the below function
def get_slices(hidden_dim):
    slices=[]
    breaker=(hidden_dim*4)
    slices=[[i,i+3] for i in range(0, breaker, breaker//4)]
    return slices

Now we have the functions ready for lstm, now we create an lstm class to copy the weights from pytorch class and get the output from it.
class numpy_lstm:
    def __init__( self, layer_num=0, hidden_dim=1, matching_in_out=False):
        self.matching_in_out=matching_in_out
        self.layer_num=layer_num
        self.hidden_dim=hidden_dim

    def init_weights_from_pytorch(self, state):
        slices=get_slices(self.hidden_dim)
        print (slices)

        #Event (x) Weights and Biases for all gates

        lstm_weight_ih='lstm.weight_ih_l'+str(self.layer_num)
        self.Weights_xi = state[lstm_weight_ih][slices[0][0]:slices[0][1]].numpy()  # shape  [h, x]
        self.Weights_xf = state[lstm_weight_ih][slices[1][0]:slices[1][1]].numpy()  # shape  [h, x]
        self.Weights_xl = state[lstm_weight_ih][slices[2][0]:slices[2][1]].numpy()  # shape  [h, x]
        self.Weights_xo = state[lstm_weight_ih][slices[3][0]:slices[3][1]].numpy() # shape  [h, x]

        lstm_bias_ih='lstm.bias_ih_l'+str(self.layer_num)
        self.Bias_xi = state[lstm_bias_ih][slices[0][0]:slices[0][1]].numpy()  #shape is [h, 1]
        self.Bias_xf = state[lstm_bias_ih][slices[1][0]:slices[1][1]].numpy()  #shape is [h, 1]
        self.Bias_xl = state[lstm_bias_ih][slices[2][0]:slices[2][1]].numpy()  #shape is [h, 1]
        self.Bias_xo = state[lstm_bias_ih][slices[3][0]:slices[3][1]].numpy() #shape is [h, 1]

        lstm_weight_hh='lstm.weight_hh_l'+str(self.layer_num)

        #Hidden state (h) Weights and Biases for all gates
        self.Weights_hi = state[lstm_weight_hh][slices[0][0]:slices[0][1]].numpy()  #shape is [h, h]
        self.Weights_hf = state[lstm_weight_hh][slices[1][0]:slices[1][1]].numpy()  #shape is [h, h]
        self.Weights_hl = state[lstm_weight_hh][slices[2][0]:slices[2][1]].numpy()  #shape is [h, h]
        self.Weights_ho = state[lstm_weight_hh][slices[3][0]:slices[3][1]].numpy() #shape is [h, h]

        lstm_bias_hh='lstm.bias_hh_l'+str(self.layer_num)

        self.Bias_hi = state[lstm_bias_hh][slices[0][0]:slices[0][1]].numpy()  #shape is [h, 1]
        self.Bias_hf = state[lstm_bias_hh][slices[1][0]:slices[1][1]].numpy()  #shape is [h, 1]
        self.Bias_hl = state[lstm_bias_hh][slices[2][0]:slices[2][1]].numpy()  #shape is [h, 1]
        self.Bias_ho = state[lstm_bias_hh][slices[3][0]:slices[3][1]].numpy() #shape is [h, 1]
    def forward_lstm_pass(self,input_data):
        h = np.zeros(self.hidden_dim)
        c = np.zeros(self.hidden_dim)

        output_list=[]
        for eventx in input_data:
            f = forget_gate(eventx, h, self.Weights_hf, self.Bias_hf, self.Weights_xf, self.Bias_xf, c)
            i =  input_gate(eventx, h, self.Weights_hi, self.Bias_hi, self.Weights_xi, self.Bias_xi,
                        self.Weights_hl, self.Bias_hl, self.Weights_xl, self.Bias_xl)
            c = cell_state(f,i)
            h = output_gate(eventx, h, self.Weights_ho, self.Bias_ho, self.Weights_xo, self.Bias_xo, c)
            if self.matching_in_out: # doesnt make sense but it was as it was in main code :(
                output_list.append(h)
        if self.matching_in_out:
            return output_list
        else:
            return h

Similarly for fully connected layer,

class fully_connected_layer:
    def __init__(self,state, dict_name='fc', ):
        self.fc_Weight = state[dict_name+'.weight'][0].numpy()
        self.fc_Bias = state[dict_name+'.bias'][0].numpy() #shape is [,output_size]

    def forward(self,lstm_output, is_sigmoid=True):
        res=np.dot(self.fc_Weight, lstm_output)+self.fc_Bias
        print (res)
        if is_sigmoid:
            return sigmoid(res)
        else:
            return res

Now we would need one class to call all of them together and generalise them with respect to multiple layers
You can modify the below class if you need more Fully connected layers or want to set false condition for sigmoid etc.

class RNN_model_Numpy:
    def __init__(self, state, input_size, hidden_dim, output_size, num_layers, matching_in_out=True):
        self.lstm_layers=[]
        for i in range(0, num_layers):
            lstm_layer_obj=numpy_lstm(layer_num=i, hidden_dim=hidden_dim, matching_in_out=True)
            lstm_layer_obj.init_weights_from_pytorch(state)
            self.lstm_layers.append(lstm_layer_obj)

        self.hidden2out=fully_connected_layer(state, dict_name='hidden2out')

    def forward(self, feature_list):
        for x in self.lstm_layers:
            lstm_output=x.forward_lstm_pass(feature_list)
            feature_list=lstm_output

        return self.hidden2out.forward(feature_list, is_sigmoid=False)

Sanity check on a numpy variable:
data = np.array(
           [[1,1],
            [2,2],
            [3,3]])

check=RNN_model_Numpy(state, input_size, hidden_dim, output_size, num_layers)
check.forward(data)

EXPLANATION:
Since we just need forward pass, we would need certain functions that are required in LSTM, for that we have the forget gate, input gate, cell gate and output gate. They are just some operations that are done on the input that you give.
For get_slices function, this is used to break down the weight matrix that we get from pytorch state dictionary (state dictionary) is the dictionary which contains the weights of all the layers that we have in our network.
For LSTM particularly have it in this order  ignore, forget, learn, output. So for that we would need to break it up for different LSTM cells.
For numpy_lstm class, we have init_weights_from_pytorch function which must be called, what it will do is that it will extract the weights from state dictionary which we got earlier from pytorch model object and then populate the numpy array weights with the pytorch weights. You can first train your model and then save the state dictionary through pickle and then use it.
The fully connected layer class just implements the hidden2out neural network.
Finally our rnn_model_numpy class is there to ensure that if you have multiple layers then it is able to send the output of one layer of lstm to other layer of lstm.
Lastly there is a small sanity check on data variable.
IMPORTANT NOTE: PLEASE NOTE THAT YOU MIGHT GET DIMENSION ERROR AS PYTORCH WAY OF HANDLING INPUT IS COMPLETELY DIFFERENT SO PLEASE ENSURE THAT YOU INPUT NUMPY IS OF SIMILAR SHAPE AS DATA VARIABLE.
Important references:
https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html
https://christinakouridi.blog/2019/06/19/backpropagation-lstm/


Instruction: Pytorch getting accuracy of train_loop doesn't work

Input:
I want to get the accuracy of my train section of my neuronal network
But i get this error:
correct += (prediction.argmax(1) == y).type(torch.float).item()
ValueError: only one element tensors can be converted to Python scalars
With this code :
def train_loop(dataloader, model, optimizer):
    model.train()
    size = len(dataloader.dataset)
    correct = 0, 0
    l_loss = 0
    for batch, (X, y) in enumerate(dataloader):
        prediction = model(X)
        loss = cross_entropy(prediction, y)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        correct += (prediction.argmax(1) == y).type(torch.float).sum().item()
        loss, current = loss.item(), batch * len(X)
        l_loss = loss
        print(f"loss: {loss:>7f}  [{current:>5d}/{size:>5d}]")
    correct /= size
    accu = 100 * correct

    train_loss.append(l_loss)
    train_accu.append(accu)
    print(f"Accuracy: {accu:>0.1f}%")

I don't understand why it is not working becaus in my test section it work perfektly fine with execly the same code line.

Output:
item function is used to convert a one-element tensor to a standard python number as stated in the here. Please try to make sure that the result of the sum() is only a one-element tensor before using item().
x = torch.tensor([1.0,2.0]) # a tensor contains 2 elements
x.item()

error message: ValueError: only one element tensors can be converted to Python scalars
Try to use this:
prediction = prediction.argmax(1)
correct = prediction.eq(y)
correct = correct.sum()
print(correct) # to check if it is a one value tensor
correct_sum += correct.item()


Instruction: Why does Anaconda install pytorch cpuonly when I install cuda?

Input:
I have created a Python 3.7 conda virtual environment and installed the following packages using this command:
conda install pytorch torchvision torchaudio cudatoolkit=11.3 matplotlib scipy opencv -c pytorch
They install fine, but then when I come to run my program I get the following error which suggests that a CUDA enabled device is not found:
    raise RuntimeError('Attempting to deserialize object on a CUDA '
RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.

I have an NVIDIA RTX 3060ti GPU, which as far as I am aware is cuda enabled, but whenever I go into the Python interactive shell within my conda environment I get False when evaluating torch.cuda.is_available() suggesting that perhaps CUDA is not installed properly or is not found.
When I then perform a conda list to view my installed packages:
# packages in environment at /home/user/anaconda3/envs/FGVC:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main
_openmp_mutex             4.5                       1_gnu
blas                      1.0                         mkl
brotli                    1.0.9                he6710b0_2
bzip2                     1.0.8                h7b6447c_0
ca-certificates           2021.10.26           h06a4308_2
cairo                     1.16.0               hf32fb01_1
certifi                   2021.10.8        py37h06a4308_2
cpuonly                   1.0                           0    pytorch
cudatoolkit               11.3.1               h2bc3f7f_2
cycler                    0.11.0             pyhd3eb1b0_0
dbus                      1.13.18              hb2f20db_0
expat                     2.4.4                h295c915_0
ffmpeg                    4.0                  hcdf2ecd_0
fontconfig                2.13.1               h6c09931_0
fonttools                 4.25.0             pyhd3eb1b0_0
freeglut                  3.0.0                hf484d3e_5
freetype                  2.11.0               h70c0345_0
giflib                    5.2.1                h7b6447c_0
glib                      2.69.1               h4ff587b_1
graphite2                 1.3.14               h23475e2_0
gst-plugins-base          1.14.0               h8213a91_2
gstreamer                 1.14.0               h28cd5cc_2
harfbuzz                  1.8.8                hffaf4a1_0
hdf5                      1.10.2               hba1933b_1
icu                       58.2                 he6710b0_3
imageio                   2.16.0                   pypi_0    pypi
imageio-ffmpeg            0.4.5                    pypi_0    pypi
imutils                   0.5.4                    pypi_0    pypi
intel-openmp              2021.4.0          h06a4308_3561
jasper                    2.0.14               hd8c5072_2
jpeg                      9d                   h7f8727e_0
kiwisolver                1.3.2            py37h295c915_0
lcms2                     2.12                 h3be6417_0
ld_impl_linux-64          2.35.1               h7274673_9
libffi                    3.3                  he6710b0_2
libgcc-ng                 9.3.0               h5101ec6_17
libgfortran-ng            7.5.0               ha8ba4b0_17
libgfortran4              7.5.0               ha8ba4b0_17
libglu                    9.0.0                hf484d3e_1
libgomp                   9.3.0               h5101ec6_17
libopencv                 3.4.2                hb342d67_1
libopus                   1.3.1                h7b6447c_0
libpng                    1.6.37               hbc83047_0
libstdcxx-ng              9.3.0               hd4cf53a_17
libtiff                   4.2.0                h85742a9_0
libuuid                   1.0.3                h7f8727e_2
libuv                     1.40.0               h7b6447c_0
libvpx                    1.7.0                h439df22_0
libwebp                   1.2.0                h89dd481_0
libwebp-base              1.2.0                h27cfd23_0
libxcb                    1.14                 h7b6447c_0
libxml2                   2.9.12               h03d6c58_0
lz4-c                     1.9.3                h295c915_1
matplotlib                3.5.0            py37h06a4308_0
matplotlib-base           3.5.0            py37h3ed280b_0
mkl                       2021.4.0           h06a4308_640
mkl-service               2.4.0            py37h7f8727e_0
mkl_fft                   1.3.1            py37hd3c417c_0
mkl_random                1.2.2            py37h51133e4_0
munkres                   1.1.4                      py_0
ncurses                   6.3                  h7f8727e_2
networkx                  2.6.3                    pypi_0    pypi
ninja                     1.10.2           py37hd09550d_3
numpy                     1.21.2           py37h20f2e39_0
numpy-base                1.21.2           py37h79a1101_0
olefile                   0.46                     py37_0
opencv                    3.4.2            py37h6fd60c2_1
openssl                   1.1.1m               h7f8727e_0
packaging                 21.3               pyhd3eb1b0_0
pcre                      8.45                 h295c915_0
pillow                    8.4.0            py37h5aabda8_0
pip                       21.2.2           py37h06a4308_0
pixman                    0.40.0               h7f8727e_1
py-opencv                 3.4.2            py37hb342d67_1
pyparsing                 3.0.4              pyhd3eb1b0_0
pyqt                      5.9.2            py37h05f1152_2
python                    3.7.11               h12debd9_0
python-dateutil           2.8.2              pyhd3eb1b0_0
pytorch                   1.7.0               py3.7_cpu_0  [cpuonly]  pytorch
pywavelets                1.2.0                    pypi_0    pypi
qt                        5.9.7                h5867ecd_1
readline                  8.1.2                h7f8727e_1
scikit-image              0.19.1                   pypi_0    pypi
scipy                     1.7.3            py37hc147768_0
setuptools                58.0.4           py37h06a4308_0
sip                       4.19.8           py37hf484d3e_0
six                       1.16.0             pyhd3eb1b0_1
sqlite                    3.37.2               hc218d9a_0
tifffile                  2021.11.2                pypi_0    pypi
tk                        8.6.11               h1ccaba5_0
torchaudio                0.7.0                      py37    pytorch
torchvision               0.8.1                  py37_cpu  [cpuonly]  pytorch
tornado                   6.1              py37h27cfd23_0
typing_extensions         3.10.0.2           pyh06a4308_0
wheel                     0.37.1             pyhd3eb1b0_0
xz                        5.2.5                h7b6447c_0
zlib                      1.2.11               h7f8727e_4
zstd                      1.4.9                haebb681_0

There seems to be a lot of things saying cpuonly, but I am not sure how they came about, since I did not install them.
I am running Ubuntu version 20.04.4 LTS

Output:
I believe I had the following things wrong that prevented me from using Cuda. Despite having cuda installed the nvcc --version command indicated that Cuda was not installed and so what I did was add it to the path using this answer.
Despite doing that and deleting my original conda environment and using the conda install pytorch torchvision torchaudio cudatoolkit=11.3 matplotlib scipy opencv -c pytorch command again I still got False when evaluating torch.cuda.is_available().
I then used this command conda install pytorch torchvision torchaudio cudatoolkit=10.2 matplotlib scipy opencv -c pytorch changing cudatoolkit from verison 11.3 to version 10.2 and then it worked!
Now torch.cuda.is_available() evaluates to True
Unfortunately, Cuda version 10.2 was incompatible with my RTX 3060 gpu (and I'm assuming it is not compatible with all RTX 3000 cards). Cuda version 11.0 was giving me errors and Cuda version 11.3 only installs the CPU only versions for some reason. Cuda version 11.1 worked perfectly though!
This is the command I used to get it to work in the end:
pip install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html


Instruction: Attribute Error: `loss.backward()` returns None

Input:
I'm trying to implement the Learner object and its steps and facing an issue with the loss.backward() function as it raises and AttributeError: 'NoneType' object has no attribute 'data'
The entire process works when I follow the Chapter 04 MNIST Basics. However, implementing within a class raises this error. Could anybody guide me on why this occurs and ways to fix this?
Here's the code below:
class Basic_Optim:

    def __init__(self, params, lr):
        self.params = list(params)
        self.lr = lr

    def step(self):
        for p in self.params:
            p.data -= self.lr * p.grad.data

    def zero(self):
        for p in self.params:
            p.grad = None

class Learner_self:

    def __init__(self, train, valid, model, loss, metric, params, lr):
        self.x = train
        self.y = valid
        self.model = model
        self.loss = loss
        self.metric = metric
        self.opt_func = Basic_Optim(params, lr)

    def fit(self, epochs):
        for epoch in range(epochs):
            self.train_data()
            score = self.valid_data()
            print(score, end = ' | ')

    def train_data(self):
        for x, y in self.x:
            preds = self.model(x)
            loss = self.loss(preds, y)
            loss_b = loss.backward()
            print(f'Loss: {loss:.4f}, Loss Backward: {loss_b}')

            self.opt_func.step()
            self.opt_func.zero()

    def valid_data(self):
        accuracy = [self.metric(xb, yb) for xb, yb in self.y]
        return round(torch.stack(accuracy).mean().item(), 4)

learn = Learner_self(dl, valid_dl, simple_net, mnist_loss, metric=batch_accuracy,
                     params=linear_model.parameters(), lr = 1)
learn.fit(10)

OUTPUT from the print statement inside the train_data prints: Loss: 0.0516, Loss Backward: None and then raises the Attribute error shared above.
Please let me know if you want any more details. Every other function such as mnist_loss, batch_accuracy, simple_net are exactly the same from the book.
Thank you in advance.

Output:
It seems like your optimizer and your trainer do not work on the same model.
You have model=simple_net, while the parameters for the optimizer are those of a different model params=linear_model.parameters().
Try passing params=simple_net.parameters() -- that is, make sure the trainer's params are those of model.