Instruction: RuntimeError: Expected hidden[0] size (1, 1, 512), got (1, 128, 512) for LSTM pytorch

Input:
I trained the LSTM with a batch size of 128 and during testing my batch size is 1, why do I get this error? I'm suppose to initialize the hidden size when doing testing?
Here is the code that i'm using, I initialize the hidden state init_hidden function as (number_of_layers, batch_size, hidden_size) since batch_first=True
class ImageLSTM(nn.Module):
def __init__(self,  n_inputs:int=49,
                    n_outputs:int=4096,
                    n_hidden:int=256,
                    n_layers:int=1,
                    bidirectional:bool=False):
    """
    Takes a 1D flatten images.
    """
    super(ImageLSTM, self).__init__()
    self.n_inputs   = n_inputs
    self.n_hidden   = n_hidden
    self.n_outputs  = n_outputs
    self.n_layers   = n_layers
    self.bidirectional = bidirectional
    self.lstm       = nn.LSTM(  input_size=self.n_inputs,
                                hidden_size=self.n_hidden,
                                num_layers=self.n_layers,
                                dropout  = 0.5 if self.n_layers>1 else 0,
                                bidirectional=self.bidirectional,
                                batch_first=True)
    if (self.bidirectional):
        self.FC         = nn.Sequential(
                                        nn.Linear(self.n_hidden*2, self.n_outputs),
                                        nn.Dropout(p=0.5),
                                        nn.Sigmoid()
                                            )

    else:
        self.FC         = nn.Sequential(
                                        nn.Linear(self.n_hidden, self.n_outputs),
                                        # nn.Dropout(p=0.5),
                                        nn.Sigmoid()
                                        )

def init_hidden(self, batch_size, device=None): # input 4D tensor: (batch size, channels, width, height)
    # initialize the hidden and cell state to zero
    # vectors:(number of layer, batch size, number of hidden nodes)
    if (self.bidirectional):
        h0 = torch.zeros(2*self.n_layers,  batch_size, self.n_hidden)
        c0 = torch.zeros(2*self.n_layers,  batch_size, self.n_hidden)
    else:
        h0 = torch.zeros(self.n_layers,  batch_size, self.n_hidden)
        c0 = torch.zeros(self.n_layers,  batch_size, self.n_hidden)

    if device is not None:
        h0 = h0.to(device)
        c0 = c0.to(device)
    self.hidden = (h0,c0)

def forward(self, X): # X: tensor of shape (batch_size, channels, width, height)
    # forward propagate LSTM
    lstm_out, self.hidden = self.lstm(X, self.hidden) # lstm_out: tensor of shape (batch_size, seq_length, hidden_size)
    # Decode the hidden state of the last time step
    out = self.FC(lstm_out[:, -1, :])
    return out

Output:
please edit your post and add code. How did you initialize the hidden-state? What does you model look like.
hidden[0] is not your hidden-size, its the hidden-state of the lstm. The shape of the hidden-state has to be initialized like this:
hidden = ( torch.zeros((batch_size, layers, hidden_size)), torch.zeros((layers, batch_size, hidden_size)) )

You seem to have done this correctly. But the error tells you that you gave a batch of size 1 (because as you said you want to test with only one sample) but the hidden-state is initialized with batch-size=128.
So I guess (please add code) that you hard-coded that the batch-size = 128. Dont do that. Since you have to reinitialize the hidden-state every forward pass you can do this:
...

def forward(self, x):
    batch_size = x.shape[0]
    hidden = (torch.zeros(self.layers, batch_size, self.hidden_size).to(device=device), torch.zeros(self.layers, batch_size, self.hidden_size).to(device=device))

    output, hidden = lstm(x, hidden)

    # then do what every you want with the output

I guess that this is what causes this error but please post your code, too!


Instruction: Can´t install Pytorch on PyCharm: No matching distribution found for torch==1.7.0+cpu

Input:
I tried multiple times installing Pytorch on Pycharm. I used the code that the pytorch web site give you for a specific configuration. I use this one:
enter image description here
Then I copied this information on Pycharm Terminal and I get this message:
(venv) D:\Usuarios\AuCap\Documents\mnist>pip install torch==1.7.0+cpu torchvision==0.8.1+cpu torchaudio===0.7.0 -f https://download.pytorch.org/w
hl/torch_stable.html
Looking in links: https://download.pytorch.org/whl/torch_stable.html
ERROR: Could not find a version that satisfies the requirement torch==1.7.0+cpu (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)
ERROR: No matching distribution found for torch==1.7.0+cpu
I also tried installing it using Python interpreter in the Pycharm Settings and also didnd´t work.
Thanks for your help

Output:
Downgrade your Python version as python3.9 is not supported by PyTorch right now (python3.8 is fine though).
See this issue, it will be supported in subsequent releases.
You can build your own PyTorch from source if you wish though.


Instruction: object detection: is object in the photo, python

Input:
I am trying to detect plants in the photos, i've already labeled photos with plants (with labelImg), but i don't understand how to train model with only background photos, so that when there is no plant here model can tell me so.
Do I need to set labeled box as the size of image?
p.s. new to ml so don't be rude, please)

Output:
I recently had a problem where all my training images were zoomed in on the object. This meant that the training images all had very little background information. Since object detection models use space outside bounding boxes as negative examples of these objects, this meant that the model had no background knowledge. So the model knew what objects were, but didn't know what they were not.
So I disagree with @Rika, since sometimes background images are useful. With my example, it worked to introduce background images.
As I already said, object detection models use non-labeled space in an image as negative examples of a certain object. So you have to save annotation files without bounding boxes for background images. In the software you use here (labelImg), you can use verify image to say that it saves the annotation file of the image without boxes. So it saves a file that says it should be included in training, but has no bounding box information. The model uses this as negative examples.


Instruction: PyTorch dll issues for Caffe2

Input:
I am using a Windows 10 Machine and after re-installing Anaconda and all of the packages I had previously, including torchvision, torch and necessary dependencies, I am still getting this error:
OSError: [WinError 127] The specified procedure could not be found. Error loading "C:\Users\XXX\Anaconda3\envs\XXX\lib\site-packages\torch\lib\caffe2.dll" or one of its dependencies.

I am using python 3.7.9 and:

torchaudio=0.6.0=py37
torchvision=0.7.0=py37_cpu
tornado=6.0.4=py37he774522_1
traitlets=5.0.5=py_0

I've looked into it quite a bit but feel like this should be an easy solve...
I do not have CUDA and have used this:
conda install pytorch torchvision torchaudio cpuonly -c pytorch
as per instructed on the official website of pytorch

Output: After a long time trying many things with Anaconda I decided to use bare python instead and I installed Python 3.8.6 and installed PyTorch from the link you provided and it finally worked even with CUDA support. Make sure to completely remove all Anaconda/Other Python version scripts from your path to ensure only the 3.8.6 version is used by your prompt.


Instruction: Saving and reload huggingface fine-tuned transformer

Input:
I am trying to reload a fine-tuned DistilBertForTokenClassification model. I am using transformers 3.4.0 and pytorch version 1.6.0+cu101. After using the Trainer to train the downloaded model, I save the model with trainer.save_model() and in my trouble shooting I save in a different directory via model.save_pretrained(). I am using Google Colab and saving the model to my Google drive. After testing the model I also evaluated the model on my test getting great results, however, when I return to the notebook (or Factory restart the colab notebook) and try to reload the model, the predictions are terrible. Upon checking the directories, the config.json file is there as is the pytorch_mode.bin. Below is the full code.
from transformers import DistilBertForTokenClassification

# load the pretrained model from huggingface
#model = DistilBertForTokenClassification.from_pretrained('distilbert-base-cased', num_labels=len(uniq_labels))
model = DistilBertForTokenClassification.from_pretrained('distilbert-base-uncased', num_labels=len(uniq_labels))

model.to('cuda');

from transformers import Trainer, TrainingArguments

training_args = TrainingArguments(
    output_dir = model_dir +  'mitmovie_pt_distilbert_uncased/results',          # output directory
    #overwrite_output_dir = True,
    evaluation_strategy='epoch',
    num_train_epochs=3,              # total number of training epochs
    per_device_train_batch_size=16,  # batch size per device during training
    per_device_eval_batch_size=64,   # batch size for evaluation
    warmup_steps=500,                # number of warmup steps for learning rate scheduler
    weight_decay=0.01,               # strength of weight decay
    logging_dir = model_dir +  'mitmovie_pt_distilbert_uncased/logs',            # directory for storing logs
    logging_steps=10,
    load_best_model_at_end = True
)

trainer = Trainer(
    model = model,                         # the instantiated  Transformers model to be trained
    args = training_args,                  # training arguments, defined above
    train_dataset = train_dataset,         # training dataset
    eval_dataset = test_dataset             # evaluation dataset
)

trainer.train()

trainer.evaluate()

model_dir = '/content/drive/My Drive/Colab Notebooks/models/'
trainer.save_model(model_dir + 'mitmovie_pt_distilbert_uncased/model')

# alternative saving method and folder
model.save_pretrained(model_dir + 'distilbert_testing')

Coming back to the notebook after restarting...
from transformers import DistilBertForTokenClassification, DistilBertConfig, AutoModelForTokenClassification

# retreive the saved model
model = DistilBertForTokenClassification.from_pretrained(model_dir + 'mitmovie_pt_distilbert_uncased/model',
                                                        local_files_only=True)

model.to('cuda')

Model predictions are terrible now from either directory, however, the model does work and outputs the number of classes I would expect, it appears that the actual trained weights have not been saved or are somehow not getting loaded.

Output:
Do you tried loading the by the trainer saved model in the folder:
mitmovie_pt_distilbert_uncased/results

The Huggingface trainer saves the model directly to the defined output_dir.


Instruction: complex functions that already support autograd - Pytorch

Input:
I am using this customized function to reshape my tensors in the customized loss function.
def reshape_fortran(x, shape):
if len(x.shape) > 0:
    x = x.permute(*reversed(range(len(x.shape))))
return x.reshape(*reversed(shape)).permute(*reversed(range(len(shape))))

Though, I receive this error:
RuntimeError: _unsafe_view does not support automatic differentiation for outputs with complex dtype.
for reshape_fortran output.
Do you know what might be the problem? which function is not supported in Pytorch autograd for complex numbers?

Output: Complex Autograd was in Beta as of version 1.8, but is now stable and should fully support such operations as of 1.9.


Instruction: Unable to install PyTorch on Windows 10 (x86_64) with Cuda 11.0 using pip

Input:
I tried following the instructions on pytorch.org and ran the command provided by them for my configuration, but I get the following error
ERROR: Could not find a version that satisfies the requirement torch===1.7.0+cu110 (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)
ERROR: No matching distribution found for torch===1.7.0+cu110
A  similar error is being thrown for any other installation methods that I try, e.g. earlier cuda versions, CPU only etc. My Python version is 3.9 and was installed using the .exe installer provided at python.org, if that's relevant.

Output:
Just go to
https://pytorch.org/get-started/locally/
and get PyTorch for any platform using conda or pip.


Instruction: IndexError: index_select(): Index is supposed to be a vector

Input:
    for batch_id, (data, target) in enumerate(tqdm(train_loader)):
        print(target)
        print('Entered for loop')
        target = torch.sparse.torch.eye(10).index_select(dim=0, index=target)
        data, target = Variable(data), Variable(target)

The line which contains the index_select function gives this error and I am not able to find a solution to it anywhere. The target variable on printing looks like this:
tensor([[4],
[1],
[8],
[5],
[9],
[5],
[5],
[8],
[4],
[6]])

How do I convert the target variable into a vector? Isn’t it already a vector?

Output:
If you would look at the shape of your target variable, you would find that it is a 2D tensor of shape:
target.shape # torch.Size([10, 1])

Error message is a bit confusing, but in essence index should be a 1D tensor (vector). So using .squeeze method would make:
target.squeeze().shape # torch.Size([10])

and index_select method would not complain.


Instruction: PyTorch is tiling images when loaded with Dataloader

Input:
I am trying to load an Images Dataset using the PyTorch dataloader, but the resulting transformations are tiled, and don't have the original images cropped to the center as I am expecting them.
transform = transforms.Compose([transforms.Resize(224),
                             transforms.CenterCrop(224),
                             transforms.ToTensor()])

dataset = datasets.ImageFolder('ml-models/downloads/', transform=transform)
dataloader = torch.utils.data.DataLoader(dataset, batch_size=32, shuffle=True)

images, labels = next(iter(dataloader))
import matplotlib.pyplot as plt
plt.imshow(images[6].reshape(224, 224, 3))

The resulting image is tiled, and not center cropped.[![as shown in the Jupyter snapshot here][1]][1]
Is there something wrong in the provided transformation? (Image shown below on link: )
[1]: https://i.stack.imgur.com/HtrIa.png

Output:
Pytorch stores tensors in channel-first format, so a 3 channel image is a tensor of shape (3, H, W). Matplotlib expects data to be in channel-last format i.e. (H, W, 3). Reshaping does not rearrange the dimensions, for that you need Tensor.permute.
plt.imshow(images[6].permute(1, 2, 0))


Instruction: clever image augmentation - random zoom out

Input:
i'm building a CNN to identify facial keypoints. i want to make the net more robust, so i thought about applying some zoom-out transforms because most pictures have about the same location of keypoints, so the net doesn't learn much.
my approach:
i want augmented images to keep the original image size so apply MaxPool2d and then random (not equal) padding until the original size is reached.
first question
is it going to work with simple average padding or zero padding? i'm sure it would be even better if i made the padding appear more like a background but is there a simple way to do that?
second question
the keypoints are the target vector, they come as a row vector of 30. i'm getting confused with the logic needed to transform them to the smaller space.
generally if an original point was at (x=5,y=7) it transforms to (x=2,y=3)- i'm not sure about it but so far manually checked and it's correct. but what to do if to keypoints are in the same new pixel? i can't feed the network with less target values.
that's it. would be happy to hear your thoughts

Output: I suggest to use torchvision.transforms.RandomResizedCrop as a part of your Compose statement. which will give you random zooms AND resize the resulting the images to some standard size. This avoids issues in both your questions.


Instruction: Defining Loss function in pytorch

Input:
I have to define a huber loss function which is this:
This is my code
def huber(a, b):
   res = (((a-b)[abs(a-b) < 1]) ** 2 / 2).sum()
   res += ((abs(a-b)[abs(a-b) >= 1]) - 0.5).sum()
   res = res / torch.numel(a)
   return res

'''
yet, it is not working properly. Do you have any idea what is wrong?

Output:
Huber loss function already exists in PyTorch under the name of torch.nn.SmoothL1Loss.
Follow this link https://pytorch.org/docs/stable/generated/torch.nn.SmoothL1Loss.html for more!


Instruction: How to Use Class Weights with Focal Loss in PyTorch for Imbalanced dataset for MultiClass Classification

Input:
I am working on Multiclass Classification (4 classes) for Language Task and I am using the BERT model for classification task. I am following this blog as reference. My BERT Fine Tuned model returns nn.LogSoftmax(dim=1).
My data is pretty imbalanced so I used sklearn.utils.class_weight.compute_class_weight to compute weights of the classes and used the weights inside the Loss.
class_weights = compute_class_weight('balanced', np.unique(train_labels), train_labels)
weights= torch.tensor(class_weights,dtype=torch.float)
cross_entropy  = nn.NLLLoss(weight=weights)

My results were not so good so I thought of Experementing with  Focal Loss and have a code for Focal Loss.
class FocalLoss(nn.Module):
  def __init__(self, alpha=1, gamma=2, logits=False, reduce=True):
    super(FocalLoss, self).__init__()
    self.alpha = alpha
    self.gamma = gamma
    self.logits = logits
    self.reduce = reduce

  def forward(self, inputs, targets):
    BCE_loss = nn.CrossEntropyLoss()(inputs, targets)

    pt = torch.exp(-BCE_loss)
    F_loss = self.alpha * (1-pt)**self.gamma * BCE_loss

    if self.reduce:
      return torch.mean(F_loss)
    else:
      return F_loss

I have 3 questions now. First and the Most important is

Should I use Class Weight with Focal Loss?
If I have to Implement weights inside this Focal Loss, can I use weights parameters inside  nn.CrossEntropyLoss()
If this implement is incorrect, what should be the proper code for this one including the weights (if possible)

Output:
You may find answers to your questions as follows:

Focal loss automatically handles the class imbalance, hence weights are not required for the focal loss. The alpha and gamma factors handle the class imbalance in the focal loss equation.
No need of extra weights because focal loss handles them using alpha and gamma modulating factors
The implementation you mentioned is correct according to the focal loss formula but I had trouble in causing my model to converge with this version hence, I used the following implementation from mmdetection framework

    pred_sigmoid = pred.sigmoid()
    target = target.type_as(pred)
    pt = (1 - pred_sigmoid) * target + pred_sigmoid * (1 - target)
    focal_weight = (alpha * target + (1 - alpha) *
                    (1 - target)) * pt.pow(gamma)
    loss = F.binary_cross_entropy_with_logits(
        pred, target, reduction='none') * focal_weight
    loss = weight_reduce_loss(loss, weight, reduction, avg_factor)
    return loss

You can also experiment with another focal loss version available


Instruction: Pytorch installation could not find a version that satisfies the requirement

Input:
When i tried to install Pytorch in the way they suggest on their website:
pip install torch===1.7.0 torchvision===0.8.1 torchaudio===0.7.0 -f https://download.pytorch.org/whl/torch_stable.html

this is the error that appear:
ERROR: Could not find a version that satisfies the requirement torch===1.7.0 (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)
ERROR: No matching distribution found for torch===1.7.0

How can i solve it?

Output:
Just wanted to start out by letting all the mac, linux, and python 3.8.x- users here know that adding "https://" to the command does not solve the problem: Proof that it doesn't help or solve anything
Here's why: OP, you probably have python 3.9 installed on your machine. Unfortunately, Python 3.9 is not yet supported by Pytorch. Install python 3.8.6 instead, that's the latest version that's currently supported. You'll find similar problems trying to install packages like sklearn or tensorflow as well.
So here's the answer: either wait, or uninstall python and roll back to python 3.8.6
Sorry it couldn't be a better one.


Instruction: Pytorch : W ParallelNative.cpp:206

Input:
I'm trying to use a pre-trained template on my image set by following the tutorial right here :
https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html
Only I always get this "error" when I run my code and the console locks up :
[W ParallelNative.cpp:206] Warning: Cannot set number of intraop threads after parallel work has started or after set_num_threads call when using native parallel backend (function set_num_threads)
Thank you in advance for your help,

Output:
I have the same problem.
Mac. Python 3.6 (also reproduces on 3.8). Pytorch 1.7.
It seems that with this error dataloaders don't (or can't) use parallel computing.
You can remove the error (this will not fix the problem) in two ways.

If you can access your dataloaders, set num_workers=0 when creating a dataloader
Set environment variable export OMP_NUM_THREADS=1

Again, both solutions kill parallel computing and may slow down data loading (and therefore training). I look forward to efficient solutions or a patch in Pytorch 1.7


Instruction: Indexing list of tensors

Input:
I have two identical lists of tensors (with different sizes) except that for the first one all of the tensors are assigned to the cuda device. For example:
list1=[torch.tensor([0,1,2]).cuda(),torch.tensor([3,4,5,6]).cuda(),torch.tensor([7,8]).cuda()]
>>> list1
[tensor([0, 1, 2], device='cuda:0'), tensor([3, 4, 5, 6], device='cuda:0'), tensor([7, 8], device='cuda:0')]

list2=[torch.tensor([0,1,2]),torch.tensor([3,4,5,6]),torch.tensor([7,8])]
>>> list2
[tensor([0, 1, 2]), tensor([3, 4, 5, 6]), tensor([7, 8])]

I want to extract some tensors from the lists according to an array of indices such as:
ind=torch.tensor([0,2])
>>> ind
tensor([0, 2])

So my solution was to do something like that:
np.array(list1)[ind]

np.array(list2)[ind]

My question is why it works with the first list with the tensors defined on the cuda device and gives an error with the second list as shown below:
>>> np.array(list1)[ind]
array([tensor([0, 1, 2], device='cuda:0'),
       tensor([7, 8], device='cuda:0')], dtype=object)

>>> np.array(list2)[ind]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: only one element tensors can be converted to Python scalars

EDIT:
Just to clarify, the error isn't raised because the tensors have different shapes. The following examples illustrate this point:
list3=[torch.tensor([1,2,3]).cuda()]
list4=[torch.tensor([1,2,3]).cuda(),torch.tensor([4,5,6]).cuda()]
list5=[torch.tensor([1,2,3])]
list6=[torch.tensor([1,2,3]),torch.tensor([4,5,6])]

And the results are:
>>> np.array(list3)
array([tensor([1, 2, 3], device='cuda:0')], dtype=object)
>>> np.array(list4)
array([tensor([1, 2, 3], device='cuda:0'),
       tensor([4, 5, 6], device='cuda:0')], dtype=object)
>>> np.array(list5)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: only one element tensors can be converted to Python scalars
>>> np.array(list6)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: only one element tensors can be converted to Python scalars

Output:
np.array trys to convert each of the elements of a list into a numpy array. This is only supported for CPU tensors. The short answer is you can explicitly instruct numpy to create an array with dtype=object to make the CPU case works. To understand what exactly is happening lets take a closer look at both cases.
Case 1 (CUDA tensors)
First note that if you attempt to use np.array on a CUDA tensor you get the following error
np.array(torch.zeros(2).cuda())

TypeError: can't convert CUDA tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.

In your example, numpy tries to convert each element of list1 to a numpy array, however an exception is raised so it just settles on creating an array with dtype=object.
You end up with
np.array([torch.tensor([0,1,2]).cuda(), torch.tensor([3,4,5,6]).cuda(), torch.tensor([7,8]).cuda()])

being just a container pointing to different objects
array([tensor([0, 1, 2], device='cuda:0'),
       tensor([3, 4, 5, 6], device='cuda:0'),
       tensor([7, 8], device='cuda:0')], dtype=object)

Case 2 (CPU tensors)
For CPU tensors, PyTorch knows how to convert to numpy arrays. So when you run
np.array(torch.zeros(2))

you get a numpy array with dtype float32
array([0., 0.], dtype=float32)

The problem comes in your code when numpy successfully converts each element in list2 into a numpy array and then tries to stack them into a single multi-dimensional array. Numpy expects that each list entry represents one row of a multi-dimensional array, but in your case it finds that not all rows have the same shape, so doesn't know how to proceed and raises an exception.
One way to get around this is to explicitly specify that dtype should remain object. This basically tells numpy "don't try to convert the entries to numpy arrays first".
np.array([torch.tensor([0,1,2]), torch.tensor([3,4,5,6]), torch.tensor([7,8])], dtype=object)

which now gives a similar result to case 1
array([tensor([0, 1, 2]),
       tensor([3, 4, 5, 6]),
       tensor([7, 8])], dtype=object)


Instruction: Whats the equivalent of tf.keras.Input() in pytorch?

Input:
can someone tell me what the equivalent of tf.keras.Input() in pytorch is?
At the documentation it says, "Initiates a Keras Tensor", so does it just creates a new empty tensor?
Thanks

Output:
There's no equivalent in PyTorch to the Keras' Input. All you have to do is pass on the inputs as a tensor to the PyTorch model.
For eg: If you're working with a Conv net:

# Keras Code
input_image = Input(shape=(32,32,3)) # An input image of 32x32x3 (HxWxC)
feature = Conv2D(16, activation='relu', kernel_size=(3, 3))(input_image)

# PyTorch code
input_tensor = torch.randn(1, 3, 32, 32) # An input tensor of shape BatchSizexChannelsxHeightxWidth
feature = nn.Conv2d(in_channels=3, out_channels=16, kernel_size=(2, 2))(input_tensor)

If it is a normal Dense layer:
# Keras
dense_input = Input(shape=(1,))
features = Dense(100)(dense_input)

# PyTorch
input_tensor = torch.tensor([10]) # A 1x1 tensor with value 10
features = nn.Linear(1, 100)(input_tensor)


Instruction: Optuna Pytorch: returned value from the objective function cannot be cast to float

Input:
def autotune(trial):

      cfg= { 'device' : "cuda" if torch.cuda.is_available() else "cpu",
         #   'train_batch_size' : 64,
         #   'test_batch_size' : 1000,
         #   'n_epochs' : 1,
         #   'seed' : 0,
         #   'log_interval' : 100,
         #   'save_model' : False,
         #   'dropout_rate' : trial.suggest_uniform('dropout_rate',0,1.0),
            'lr' : trial.suggest_loguniform('lr', 1e-3, 1e-2),
            'momentum' : trial.suggest_uniform('momentum', 0.4, 0.99),
            'optimizer': trial.suggest_categorical('optimizer',[torch.optim.Adam,torch.optim.SGD, torch.optim.RMSprop, torch.optim.$
            'activation': F.tanh}
      optimizer = cfg['optimizer'](model.parameters(), lr=cfg['lr'])
      #optimizer = torch.optim.Adam(model.parameters(),lr=0.001

As u can see above , I am trying to run Optuna trials to search for the most optimal hyper parameters for my CNN model.
# Train the model
# use small epoch for large dataset
# An epoch is 1 run through all the training data
# losses = [] # use this array for plotting losses
      for _ in range(epochs):
    # using data_loader
         for i, (data, labels) in enumerate(trainloader):
        # Forward and get a prediction
        # x is the training data which is X_train
            if name.lower() == "rnn":
                model.hidden = (torch.zeros(1,1,model.hidden_sz),
                    torch.zeros(1,1,model.hidden_sz))

            y_pred = model.forward(data)

        # compute loss/error by comparing predicted out vs acutal labels
            loss = criterion(y_pred, labels)
        #losses.append(loss)

            if i%10==0:  # print out loss at every 10 epoch
                 print(f'epoch {i} and loss is: {loss}')

        #Backpropagation
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

study = optuna.create_study(sampler=optuna.samplers.TPESampler(), direction='minimize',pruner=optuna.pruners.SuccessiveHalvingPrune$
study.optimize(autotune, n_trials=1)

BUT , when I run the above code to tune and find out my most optimal parameters , the follow error occured, seems like the trial has failed even though I still get epoch losses and values. Please advise thanks!
[W 2020-11-11 13:59:48,000] Trial 0 failed, because the returned value from the objective function cannot be cast to float. Returned value is: None
Traceback (most recent call last):
  File "autotune2", line 481, in <module>
    n_instances, n_features, scores = run_analysis()
  File "autotune2", line 350, in run_analysis
    print(study.best_params)
  File "/home/shar/anaconda3/lib/python3.7/site-packages/optuna/study.py", line 67, in best_params
    return self.best_trial.params
  File "/home/shar/anaconda3/lib/python3.7/site-packages/optuna/study.py", line 92, in best_trial
    return copy.deepcopy(self._storage.get_best_trial(self._study_id))
  File "/home/shar/anaconda3/lib/python3.7/site-packages/optuna/storages/_in_memory.py", line 287, in get_best_trial
    raise ValueError("No trials are completed yet.")
ValueError: No trials are completed yet.

Output:
This exception is raised because the objetive function from your study must return a float.
In your case, the problem is in this line:
study.optimize(autotune, n_trials=1)

The autotune function you defined before does not return a value and cannot be used for optimization.
How to fix?
For hyperparameter search, the autotune function must return the either some metric you can get after some training - like the loss or cross-entropy.
A quick fix on your code could be something like this:
def autotune():
  cfg= { 'device' : "cuda" if torch.cuda.is_available() else "cpu"
        ...etc...
       }

  best_loss = 1e100;  # or larger

  # Train the model
  for _ in range(epochs):
     for i, (data, labels) in enumerate(trainloader):
        ... (train the model) ...
        # compute loss/error by comparing predicted out vs actual labels
        loss = criterion(y_pred, labels)
        best_loss = min(loss,best_loss)

  return best_loss

There is a good example with Pythorch in the Optuna repo that uses a pythoch callback to retrieve the accuracy (but can be changed easily to use the RMSE if needed). It also uses more than one experiment and takes the median for hyperparameters.


Instruction: How to implement Softmax regression with pytorch?

Input:
I am working on a uni assignment where I need to implement Softmax Regression with Pytorch. The assignment says:
Implement Softmax Regression as an nn.Module and pipe its output with its output with torch.nn.Softmax.

As I am new to pytorch, I am not sure how to do it exactly. So far I have tried:
class SoftmaxRegression(nn.Module):  # inheriting from nn.Module!
def __init__(self, num_labels, num_features):

    super(SoftmaxRegression, self).__init__()
    self.linear = torch.nn.Linear(num_labels, num_features)

def forward(self, x):
    # should return the probabilities for the classes, e.g.
    # tensor([[ 0.1757,  0.3948,  0.4295],
    #         [ 0.0777,  0.3502,  0.5721],
    #         ...

    # not sure what to do here

does anybody have any idea how I could go about it? I am not sure what should be written in the forward method. I appreciate any help !

Output:
As far as I understand, the assignment wants you to implement your own version of the Softmax function. But, I didn't get what do you mean by and pipe its output with torch.nn.Softmax. Are they asking you to return the output of your custom Softmax along with torch.nn.Softmax from your custom nn.Module? You could do this:
class SoftmaxRegression(nn.Module):
    def __init__(self, dim=0):
        super(SoftmaxRegression, self).__init__()
        self.dim = dim
    def forward(self, x):
        means = torch.mean(x, self.dim, keepdim=True)[0]
        exp_x= torch.exp(x-means)
        sum_exp_x = torch.sum(exp_x, self.dim, keepdim=True)
        value = exp_x/sum_exp_x
        return value


Instruction: Is NN just Bad at Solving this Simple Linear Problem, or is it because of Bad Training?

Input:
I was trying to train a very straightforward (I thought) NN model with PyTorch and skorch, but the bad performance really baffles me, so it would be great if you have any insight into this.

The problem is something like this: there are five objects, A, B, C,
D, E, (labeled by their fingerprint, e.g.(0, 0) is A, (0.2, 0.5) is B,
etc) each correspond to a number, and the problem is trying to find
what number does each correspond to. The training data is a list of
"collections" and the corresponding sum. for example: [A, A, A, B, B]
== [(0,0), (0,0), (0,0), (0.2,0.5), (0.2, 0.5)] --> 15, [B, C, D, E] == [(0.2,0.5), (0.5,0.8), (0.3,0.9), (1,1)] --> 30 .... Note that number of object in one collection is not constant

There is no noise or anything, so it's just a linear system that can be solved directly. So I would thought this would be very easy for a NN for find out. I'm actually using this example as a sanity check for a more complicated problem, but was surprised that NN couldn't even solve this.
Now I'm just trying to pinpoint exactly where it went wrong. The model definition seem to be right, the data input is right, is the bad performance due to bad training? or is NN just bad at these things?
here is the model definition:
class NN(nn.Module):
    def __init__(
        self,
        input_dim,
        num_nodes,
        num_layers,
        batchnorm=False,
        activation=Tanh,
    ):
        super(SingleNN, self).__init__()
        self.get_forces = get_forces
        self.activation_fn = activation

        self.model = MLP(
            n_input_nodes=input_dim,
            n_layers=num_layers,
            n_hidden_size=num_nodes,
            activation=activation,
            batchnorm=batchnorm,
        )

    def forward(self, batch):
        if isinstance(batch, list):
            batch = batch[0]
        with torch.enable_grad():
            fingerprints = batch.fingerprint.float()
            fingerprints.requires_grad = True
            #index of the current "collection" in the training list
            idx = batch.idx
            sorted_idx = torch.unique_consecutive(idx)
            o = self.model(fingerprints)
            total = scatter(o, idx, dim=0)[sorted_idx]

            return total

    @property
    def num_params(self):
        return sum(p.numel() for p in self.parameters())

class MLP(nn.Module):
    def __init__(
        self,
        n_input_nodes,
        n_layers,
        n_hidden_size,
        activation,
        batchnorm,
        n_output_nodes=1,
    ):
        super(MLP, self).__init__()
        if isinstance(n_hidden_size, int):
            n_hidden_size = [n_hidden_size] * (n_layers)
        self.n_neurons = [n_input_nodes] + n_hidden_size + [n_output_nodes]
        self.activation = activation
        layers = []
        for _ in range(n_layers - 1):
            layers.append(nn.Linear(self.n_neurons[_], self.n_neurons[_ + 1]))
            layers.append(activation())
            if batchnorm:
                layers.append(nn.BatchNorm1d(self.n_neurons[_ + 1]))
        layers.append(nn.Linear(self.n_neurons[-2], self.n_neurons[-1]))
        self.model_net = nn.Sequential(*layers)

    def forward(self, inputs):
        return self.model_net(inputs)

and the skorch part is straightforward
model = NN(2, 100, 2)
net = NeuralNetRegressor(
        module=model,
        ...
    )
net.fit(train_dataset, None)

For a test run, the dataset looks like the following (16 collections in total):
[[0.7484336 0.5656401]
 [0.        0.       ]
 [0.        0.       ]
 [0.        0.       ]]
[[1. 1.]
 [0. 0.]
 [0. 0.]]
[[0.51311415 0.67012525]
 [0.51311415 0.67012525]
 [0.         0.        ]
 [0.         0.        ]]
[[0.51311415 0.67012525]
 [0.7484336  0.5656401 ]
 [0.         0.        ]]
[[0.51311415 0.67012525]
 [1.         1.        ]
 [0.         0.        ]
 [0.         0.        ]]
[[0.51311415 0.67012525]
 [0.51311415 0.67012525]
 [0.         0.        ]
 [0.         0.        ]
 [0.         0.        ]
 [0.         0.        ]
 [0.         0.        ]
 [0.         0.        ]]
[[0.51311415 0.67012525]
 [1.         1.        ]
 [0.         0.        ]
 [0.         0.        ]
 [0.         0.        ]
 [0.         0.        ]]
....

with corresponding total:
[10, 11, 14, 14, 17, 18, ...]
It's easy to tell what are the objects/how many of them are in one collection just by eyeballing it
and the training process looks like:
 epoch    train_energy_mae    train_loss    cp     dur
-------  ------------------  ------------  ----  ------
      1              4.9852        0.5425     +  0.1486
      2             16.3659        4.2273        0.0382
      3              6.6945        0.7403        0.0025
      4              7.9199        1.2694        0.0024
      5             12.0389        2.4982        0.0024
      6              9.9942        1.8391        0.0024
      7              5.6733        0.7528        0.0024
      8              5.7007        0.5166        0.0024
      9              7.8929        1.0641        0.0024
     10              9.2560        1.4663        0.0024
     11              8.5545        1.2562        0.0024
     12              6.7690        0.7589        0.0024
     13              5.3769        0.4806        0.0024
     14              5.1117        0.6009        0.0024
     15              6.2685        0.8831        0.0024
....
    290              5.1899        0.4750        0.0024
    291              5.1899        0.4750        0.0024
    292              5.1899        0.4750        0.0024
    293              5.1899        0.4750        0.0024
    294              5.1899        0.4750        0.0025
    295              5.1899        0.4750        0.0025
    296              5.1899        0.4750        0.0025
    297              5.1899        0.4750        0.0025
    298              5.1899        0.4750        0.0025
    299              5.1899        0.4750        0.0025
    300              5.1899        0.4750        0.0025
    301              5.1899        0.4750        0.0024
    302              5.1899        0.4750        0.0025
    303              5.1899        0.4750        0.0024
    304              5.1899        0.4750        0.0024
    305              5.1899        0.4750        0.0025
    306              5.1899        0.4750        0.0024
    307              5.1899        0.4750        0.0025

You can see that it just stopped training after a while.
I can confirm that the NN does give different result for different fingerprint, but somehow the final predicted value is just never good enough.
I have tried different NN size, learning rate, batch size, activation function (tanh, relu, etc) and non of them seem to help. Do you have any insight into this? is there anything I did wrong/could try, or is NN just bad at this kind of task?

Output: First thing I've noticed: super(SingleNN, self).__init__() should be super(NN, self).__init__() instead. Change that and let me know if you still get any errors.


Instruction: Extracting feature vector for grey images via ResNet18: output with shape [1, 224, 224] doesn't match the broadcast shape [3, 224, 224]

Input:
I have 600x800 images that have only 1 channel. I am trying to use pre-trained ResNet18 to extract their features however the code expects 3 channel:
import torch
import torchvision
import torchvision.models as models
from PIL import Image

img = Image.open("labeled-data/train_moth/moth/frame163.png")

# Load the pretrained model
model = models.resnet18(pretrained=True)

# Use the model object to select the desired layer
layer = model._modules.get('avgpool')

# Set model to evaluation mode
model.eval()

transforms = torchvision.transforms.Compose([
    torchvision.transforms.Resize(256),
    torchvision.transforms.CenterCrop(224),
    torchvision.transforms.ToTensor(),
    torchvision.transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])

def get_vector(image):
    # Create a PyTorch tensor with the transformed image
    t_img = transforms(image)
    t_img = torch.cat((t_img, t_img, t_img), 0)
    # Create a vector of zeros that will hold our feature vector
    # The 'avgpool' layer has an output size of 512
    my_embedding = torch.zeros(512)

    # Define a function that will copy the output of a layer
    def copy_data(m, i, o):
        my_embedding.copy_(o.flatten())                 # <-- flatten

    # Attach that function to our selected layer
    h = layer.register_forward_hook(copy_data)
    # Run the model on our transformed image
    with torch.no_grad():                               # <-- no_grad context
        model(t_img.unsqueeze(0))                       # <-- unsqueeze
    # Detach our copy function from the layer
    h.remove()
    # Return the feature vector
    return my_embedding

Here's the error I am getting:
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-5-59ab45f8c1e6> in <module>
     42
     43
---> 44 pic_vector = get_vector(img)

<ipython-input-5-59ab45f8c1e6> in get_vector(image)
     21 def get_vector(image):
     22     # Create a PyTorch tensor with the transformed image
---> 23     t_img = transforms(image)
     24     t_img = torch.cat((t_img, t_img, t_img), 0)
     25     # Create a vector of zeros that will hold our feature vector

~/anaconda3/lib/python3.7/site-packages/torchvision/transforms/transforms.py in __call__(self, img)
     59     def __call__(self, img):
     60         for t in self.transforms:
---> 61             img = t(img)
     62         return img
     63

~/anaconda3/lib/python3.7/site-packages/torchvision/transforms/transforms.py in __call__(self, tensor)
    210             Tensor: Normalized Tensor image.
    211         """
--> 212         return F.normalize(tensor, self.mean, self.std, self.inplace)
    213
    214     def __repr__(self):

~/anaconda3/lib/python3.7/site-packages/torchvision/transforms/functional.py in normalize(tensor, mean, std, inplace)
    296     if std.ndim == 1:
    297         std = std[:, None, None]
--> 298     tensor.sub_(mean).div_(std)
    299     return tensor
    300

RuntimeError: output with shape [1, 224, 224] doesn't match the broadcast shape [3, 224, 224]

    pic_vector = get_vector(img)

Code is from: https://stackoverflow.com/a/63552285/2414957

I thought using
t_img = torch.cat((t_img, t_img, t_img), 0)

would be helpful but I was wrong.
Here's a bit about image:
$ identify frame163.png
frame163.png PNG 800x600 800x600+0+0 8-bit Gray 256c 175297B 0.000u 0:00.000

Output:
many models (almost all models) from torchvision module expects our input to be in 3 channel.
So when ever you are using pretrained model , just convert your image to RGB scale.
So if i see your code
just change this
img = Image.open("labeled-data/train_moth/moth/frame163.png")

to this
img = Image.open("labeled-data/train_moth/moth/frame163.png").convert('RGB')

The above line will just stack your gray scale image to have 3 channel
Second option what you have is Defining our model class...with single channel as input
model = models.resnet18(pretrained=True)
model.conv1 = nn.Conv2d(1, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)

Plz vote if you find this useful


Instruction: Build KNN graph over some subset of Node features

Input:
I have a point cloud that I want to use a graph neural network on. Each point in the point cloud is characterised by its positional coordinates as well as it's color. So a single node is (X, Y, Z, C).
Now I want to apply an Edge Convolution on this (as described in the DGL Edge-Conv example, and to do it I should build a Nearest Neighbors graph on (X, Y, Z) (And not on C), then use all the 4 properties as features for my neural network.
What would be a clean and efficient way to do this? (I have a lot of data so I want to batch and collate well)

Output:
Supposing you have a tensor pc of shape (NUM_POINTS, 4) where each row is (X, Y, Z, C), then you could use sklearn as follows:
from sklearn.neighbors import NearestNeighbors
import dgl

k = 3 # number of neighbours you want

neigh = NearestNeighbors(n_neighbors=k)
neigh.fit(pc[:, :3].numpy()) # selects only (X, Y, Z)

knn = neigh.kneighbors_graph()
graph = dgl.from_scipy(knn)
graph.ndata['x'] = pc

I would recommend saving these graphs to disk, so they are not computed each time you train etc.


Instruction: trying to build pytorch 1.0.0 cuda 10.2 with support for old gpu (3.0)

Input:
I'm playing with a couple of projects that explicitly require pytorch == 1.0.0, but I have an old graphics card that only supports cuda 3.0 so I'm using the cpu, which is very slow, being the graphics card a dual gpu I decided to give a try and build pytorch from the sources with support for 3.0 (I have planned to update the pc but is not gonna happen anytime soon).
I am using docker to do the build, in particular I tried to modify an existing Dockerfile from build-pytorch, on the host system I am using debian/sid and there is cuda 10.2 cudnn 7.6 installed, I'm not sure if I can downgrade cuda, and I don't know if the versions in the container must be exactly the same as the host (like for nvidia drivers).
Gist of the modified Dockerfile
The first thing I noticed when updating the versions is that package cuda-cublas-dev-10-2 was not found, the latest version was 10-0,
CUBLAS packaging changed in CUDA 10.1 to be outside of the toolkit installation path
If I install cublas version 10-0 or if I don't install it obviously no header files are found (error below), if I install the recommended libcublas-dev version the build continues for a while, with some warnings (below) , but then it stops with the error below.
I searched for the error online but I did not find anything specific, if I understand correctly there is a function declared more than once and when it is called the choice is ambiguous, but I have not yet investigated looking at the sources.
I would like to know if anyone has run into this error before and knows how to fix it.
libcublas-dev installed error:
[ 67%] Building NVCC (Device) object caffe2/CMakeFiles/caffe2_gpu.dir/__/aten/src/ATen/native/sparse/cuda/caffe2_gpu_generated_SparseCUDABlas.cu.o
/pytorch/aten/src/ATen/native/sparse/cuda/SparseCUDABlas.cu(58): error: more than one instance of function "at::native::sparse::cuda::cusparseGetErrorString" matches the argument list:
            function "cusparseGetErrorString(cusparseStatus_t)"
            function "at::native::sparse::cuda::cusparseGetErrorString(cusparseStatus_t)"
            argument types are: (cusparseStatus_t)

1 error detected in the compilation of "/tmp/tmpxft_00004ccc_00000000-6_SparseCUDABlas.cpp1.ii".
CMake Error at caffe2_gpu_generated_SparseCUDABlas.cu.o.Release.cmake:279 (message):
  Error generating file
  /pytorch/build/caffe2/CMakeFiles/caffe2_gpu.dir/__/aten/src/ATen/native/sparse/cuda/./caffe2_gpu_generated_SparseCUDABlas.cu.o

caffe2/CMakeFiles/caffe2_gpu.dir/build.make:1260: recipe for target 'caffe2/CMakeFiles/caffe2_gpu.dir/__/aten/src/ATen/native/sparse/cuda/caffe2_gpu_generated_SparseCUDABlas.cu.o' failed

warnings:
ptxas warning : Too big maxrregcount value specified 96, will be ignored

missing header error:
Scanning dependencies of target caffe2_pybind11_state
[ 59%] Building CXX object caffe2/CMakeFiles/caffe2_pybind11_state.dir/python/pybind_state.cc.o
In file included from /pytorch/aten/src/THC/THC.h:4:0,
                 from /pytorch/torch/lib/THD/../THD/base/TensorDescriptor.h:6,
                 from /pytorch/torch/lib/THD/../THD/base/TensorDescriptor.hpp:6,
                 from /pytorch/torch/lib/THD/../THD/THD.h:14,
                 from /pytorch/torch/lib/THD/base/DataChannelRequest.h:3,
                 from /pytorch/torch/lib/THD/base/DataChannelRequest.hpp:6,
                 from /pytorch/torch/lib/THD/base/DataChannelRequest.cpp:1:
/pytorch/build/caffe2/aten/src/THC/THCGeneral.h:17:23: fatal error: cublas_v2.h: No such file or directory
compilation terminated.
make[2]: *** [caffe2/torch/lib/THD/CMakeFiles/THD.dir/base/DataChannelRequest.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....

Output:
Apparently the problem was that both libcusparse and aten/src/ATen/native/sparse/cuda/SparseCUDABlas.cu implement cusparseGetErrorString() and for version >= 10.2 the one in the library should be used.
--- aten/src/ATen/native/sparse/cuda/SparseCUDABlas.cu.orig 2020-11-16 12:13:17.680023134 +0000
+++ aten/src/ATen/native/sparse/cuda/SparseCUDABlas.cu  2020-11-16 12:13:45.158407583 +0000
@@ -9,7 +9,7 @@

 namespace at { namespace native { namespace sparse { namespace cuda {

-
+#if 0
 std::string cusparseGetErrorString(cusparseStatus_t status) {
   switch(status)
   {
@@ -51,6 +51,7 @@
       }
   }
 }
+#endif

 inline void CUSPARSE_CHECK(cusparseStatus_t status)
 {

I haven't tried yet if it works at runtime but the build is successful.


Instruction: How to convert pip install to Poetry file?

Input:
With hours of research, I still can't figure out a way to convert this pip install cmd to pyproject.toml
file. I'm trying to install PyTorch.
pip install torch==1.7.0+cpu torchvision==0.8.1+cpu torchaudio===0.7.0 -f https://download.pytorch.org/whl/torch_stable.html
This is what I got at the moment (Completely Wrong!)
[tool.poetry]
name = "poetry-test"
version = "0.1.0"
description = ""
authors = ["Your Name <you@example.com>"]

[tool.poetry.dependencies]
python = "^3.8"
torch = "^1.7.0"
torchvision = "^0.8.1"
torchaudio = "^0.7.0"

[tool.poetry.dependencies.torch]
url = "https://download.pytorch.org/whl/torch_stable.html"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

Output:
Previous solution didnt work for me as mentioned in this poetry issue#2543. So what worked for me in the meantime was to upgrade to version 1.2(preview) which addresses that issue.
Install poetry 1.2
curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/install-poetry.py | python - --preview
Add below repo to your pyproject.toml as suggested here issue#4124, and you shouldnt get version mismatch.
[[tool.poetry.source]]
name = "torch_rep"
url = "https://eternalphane.github.io/pytorch-pypi"

After that, you can just proceed to install them
poetry add torch =1.7.0+cpu
poetry add torchvision=0.8.1+cpu
Alternatively, adding dependencies to pyproject.toml and doing poetry install should also work.


Instruction: LU is not same as A when I used torch.solver

Input:
let A=[[3,2],[1,-3]] and B=[[3],[-10]]
and solve equation AX=B by using torch.solve:
X, LU = torch.solve(B,A)

Then I got X=[[-1],[3]] and LU=[[3,2],[0.333,-3.666]].
According to definition of LU decompose, LU must be same as A, however they aren't same.
Can anyone explain this???
Thank you

Output:
The representation you got is a compact way of representing the lower trainagular matrix L and the upper trainagular matrix U. You can use torch.tril and torch.triu to get these matrices explicitly:
L = torch.tril(LU, -1) + torch.eye(LU.shape[-1])
U = torch.triu(LU)

verify:

In [*]: L
Out[*]:
tensor([[1.0000, 0.0000],
       [0.3333, 1.0000]])

In [*]: U
Out[*]:
tensor([[ 3.0000,  2.0000],
        [ 0.0000, -3.6667]])

And the product is indeed equal to A:

In [*]: torch.dist(L @ U , A)
Out[*]: tensor(0.)


Instruction: What is the correct input to LSTM?

Input:
I have tensors of varying length . These tensors are data for different time period. My aim is to get final output of the lstm.
torch.randn(4)-Time1
torch.randn(3,4)-Time2
torch.randn(4,4)-Time3

These are my  data, what is the input to LSTM from here?  , my aim is to get the final output from the lstm
For example, this is what I did
out_position = self.linear_out_position(features)
    _, (hn, _) = self.lstm(out_position)
        output = self.ffn(hn)

however i am getting result for each tensor, how can i only get the final result? Please I need your help

Output:
You can access the last hidden layer as hn[-1]
output = self.ffn(hn[-1])


Instruction: How to run a pytorch project with CPU?

Input: A Pytorch project is supposed to run on GPU. I want to run it on my laptop only with CPU. There are a lot of places calling .cuda() on models, tensors, etc., which fail to execute when cuda is not available. Is it possible to do it without changing the code everywhere?

Output:
Here's the simplest fix I can think of:

Put the following line near the top of your code:

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

Do a global replace. Change .cuda() to .to(device), where device is the variable set in step 1.


Instruction: How to write a contextmanager to throw and catch errors

Input:
I want to catch the runtime error CUDA out of memory on multiple occasions in my code. I do this to then rerun the whole training workflow with lower batch size. What is the best way to do that?
I am currently doing this:
try:
    result = model(input)
# if the GPU runs out of memory, start the experiment again with a smaller batch size
except RuntimeError as e:
    if str(e).startswith('CUDA out of memory.') and batch_size > 10:
        raise CudaOutOfMemory(e)
    else:
        raise e

I then catch the error CudaOutOfMemory outside my main function.
However, this is a pretty long piece of code that I need to repeat many times. Is there any way to make a context manager for this?
such that instead I can run:
with catch_cuda_out_of_mem_error:
  result = model(input)

Edit:
I want to create a context manager instead of a function because the functions I want to wrap the "try, except" around are not always the same. In my workflow, I have many functions that use a lot of GPU memory and I would like to catch this error in any of them.

Output:
Using a context manager is about properly acquiring and releasing a resource. Here you don't really have any resource that you are acquiring and releasing, so I don't think a context manager is appropriate. How about just using a function?
def try_compute_model(input):
    try:
        return model(input)
    # if the GPU runs out of memory, start the experiment again with a smaller batch size
    except RuntimeError as e:
        if str(e).startswith('CUDA out of memory.') and batch_size > 10:
            raise CudaOutOfMemory(e)
        else:
            raise e

Then use it like
result = try_compute_model(input)


Instruction: How to use fairseq interactive.py non-interactively?

Input:
I am trying to translate from English to Arabic using Fairseq. But the interactive.py script translate pieces of text fragment on-the-fly. But I need to use it as reading an input text file and writing output text file write. I referred this GitHub issue - https://github.com/pytorch/fairseq/issues/858 But it doesn't clearly explain on how to do it in general.
Any suggestions ?

Output:
fairseq-interactive can read lines from a file with the --input parameter, and it outputs translations to standard output.
So let's say I have this input text file source.txt (where every sentence to translate is on a separate line):
Hello world!
My name is John

You can run:
fairseq-interactive --input=source.txt [all-your-fairseq-parameters] > target.txt

Where > target.txt means "put in the target.txt file all (standard) output generated by fairseq-interactive". The file will be created if it doesn't exist yet.
With an English to French model it would generate a file target.txt that looks something like this (actual output may vary depending on your model, configuration and Fairseq version):
S-0     Hello world!
W-0     0.080   seconds
H-0     -0.43813419342041016    Bonj@@ our le monde !
D-0     -0.43813419342041016    Bonjour le monde !
P-0     -0.1532 -1.7157 -0.0805 -0.0838 -0.1575
S-1     My name is John
W-1     0.080   seconds
H-1     -0.3272092938423157     Je m' appelle John .
D-1     -0.3272092938423157     Je m'appelle John.
P-2     -0.3580 -0.2207 -0.0398 -0.1649 -1.0216 -0.1583

To keep only the translations (lines starting with D-), you would have to filter the content of this file. You could use this command for example:
grep -P "D-[0-9]+" target.txt | cut -f3 > only_translations.txt

but you can merge all commands in one line:
fairseq-interactive --input=source.txt [all-your-fairseq-parameters] | grep -P "D-[0-9]+" | cut -f3 > target.txt

(Actual command will depend on the actual structure of target.txt.)
Finally, know that you can use --input=- to read input from standard input.


Instruction: PyTorch: Vectorizing patch selection from a batch of images

Input:
Suppose I have a batch of images as a tensor, for example:
images = torch.zeros(64, 3, 1024, 1024)

Now, I want to select a patch from each of those images. All the patches are of the same size, but have different starting positions for each image in the batch.
size_x = 100
size_y = 100
start_x = torch.zeros(64)
start_y = torch.zeros(64)

I can achieve the desired result like that:
result = []
for i in range(arr.shape[0]):
    result.append(arr[i, :, start_x[i]:start_x[i]+size_x, start_y[i]:start_y[i]+size_y])
result = torch.stack(result, dim=0)

The question is -- is it possible to do the same thing faster, without a loop? Perhaps there is some form of advanced indexing, or a PyTorch function that can do this?

Output:
You can use torch.take to get rid of a for loop. But first, an array of indices should be created with this function
def convert_inds(img_a,img_b,patch_a,patch_b,start_x,start_y):

    all_patches = np.zeros((len(start_x),3,patch_a,patch_b))

    patch_src = np.zeros((patch_a,patch_b))
    inds_src = np.arange(patch_b)
    patch_src[:] = inds_src
    for ind,info in enumerate(zip(start_x,start_y)):

        x,y = info
        if x + patch_a + 1 > img_a: return False
        if y + patch_b + 1 > img_b: return False
        start_ind = img_b * x + y
        end_ind = img_b * (x + patch_a -1) + y
        col_src = np.linspace(start_ind,end_ind,patch_b)[:,None]
        all_patches[ind,:] = patch_src + col_src

    return all_patches.astype(np.int)

As you can see, this function essentially creates the indices for each patch you want to slice. With this function, the problem can be easily solved by
size_x = 100
size_y = 100
start_x = torch.zeros(64)
start_y = torch.zeros(64)

images = torch.zeros(64, 3, 1024, 1024)
selected_inds = convert_inds(1024,1024,100,100,start_x,start_y)
selected_inds = torch.tensor(selected_inds)
res = torch.take(images,selected_inds)

UPDATE
OP's observation is correct, the approach above is not faster than a naive approach. In order to avoid building indices every time, here is another solution based on unfold
First, build a tensor of all the possible patches
# create all possible patches
all_patches = images.unfold(2,size_x,1).unfold(3,size_y,1)

Then, slice the desired patches from all_patches
img_ind = torch.arange(images.shape[0])
selected_patches = all_patches[img_ind,:,start_x,start_y,:,:]


Instruction: Why we need a decoder_start_token_id during generation in HuggingFace BART?

Input:
During the generation phase in HuggingFace's code:
https://github.com/huggingface/transformers/blob/master/src/transformers/generation_utils.py#L88-L100
They pass in a decoder_start_token_id, I'm not sure why they need this. And in the BART config, the decoder_start_token_id is actually 2 (https://huggingface.co/facebook/bart-base/blob/main/config.json), which is the end of sentence token </s>.
And I tried a simple example:
from transformers import *

import torch
model = BartForConditionalGeneration.from_pretrained('facebook/bart-base')
tokenizer = BartTokenizer.from_pretrained('facebook/bart-base')
input_ids = torch.LongTensor([[0, 894, 213, 7, 334, 479, 2]])
res = model.generate(input_ids, num_beams=1, max_length=100)

print(res)

preds = [tokenizer.decode(g, skip_special_tokens=True, clean_up_tokenization_spaces=True).strip() for g in res]
print(preds)

The results I obtained:
tensor([[  2,   0, 894, 213,   7, 334, 479,   2]])
['He go to school.']

Though it does not affect the final "tokenization decoding" results. But it seems weird to me that the first token we generate is actually 2(</s>).

Output:
You can see in the code for encoder-decoder models that the input tokens for the decoder are right-shifted from the original (see function shift_tokens_right). This means that the first token to guess is always BOS (beginning of sentence). You can check that this is the case in your example.
For the decoder to understand this, we must choose a first token that is always followed by BOS, so which could it be? BOS? Obviously not because it must be followed by regular tokens. The padding token? Also not a good choice because it is followed by another padding token or by EOS (end of sentence). So, what about EOS then? Well, that makes sense because it is never followed by anything in the training set so there is no next token coming in conflict. And besides, isn't it natural that the beginning of sentence follows the end of another?


Instruction: how to add transformation in pytorch object detection

Input:
I'm new to PyTorch & going through the PyTorch object detection documentation tutorial pytorch docx.
At their collab version, I made the below changes to add some transformation techniques.

First change to the __getitem__ method of class PennFudanDataset(torch.utils.data.Dataset)

if self.transforms is not None:
   img = self.transforms(img)
   target = T.ToTensor()(target)
   return img, target

In actual documentation it is
if self.transforms is not None:
   img, target = self.transforms(img, target)

Secondly, at the get_transform(train)function.
def get_transform(train):
  if train:
    transformed = T.Compose([
           T.ToTensor(),
           T.GaussianBlur(kernel_size=5, sigma=(0.1, 2.0)),
          T.ColorJitter(brightness=[0.1, 0.2], contrast=[0.1, 0.2], saturation=[0, 0.2], hue=[0,0.5])
    ])
    return transformed

  else:
    return T.ToTensor()

**In the documentation it is-**
def get_transform(train):
    transforms = []
    transforms.append(T.ToTensor())
    if train:
        transforms.append(T.RandomHorizontalFlip(0.5))
    return T.Compose(transforms)

While implementing the code, I'm getting the below error. I'm not able to get what I',m doing wrong.
TypeError: Caught TypeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/worker.py", line 198, in _worker_loop
    data = fetcher.fetch(index)
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataset.py", line 272, in __getitem__
    return self.dataset[self.indices[idx]]
  File "<ipython-input-41-94e93ff7a132>", line 72, in __getitem__
    target = T.ToTensor()(target)
  File "/usr/local/lib/python3.6/dist-packages/torchvision/transforms/transforms.py", line 104, in __call__
    return F.to_tensor(pic)
  File "/usr/local/lib/python3.6/dist-packages/torchvision/transforms/functional.py", line 64, in to_tensor
    raise TypeError('pic should be PIL Image or ndarray. Got {}'.format(type(pic)))
TypeError: pic should be PIL Image or ndarray. Got <class 'dict'>

Output: I believe the Pytorch transforms only work on images (PIL images or np arrays in this case) and not labels (which are dicts according to the trace). As such, I don't think you need to "tensorify" the labels as in this line target = T.ToTensor()(target) in the __getitem__ function.


Instruction: PyTorch: RuntimeError: Input, output and indices must be on the current device

Input:
I am running a BERT model on torch. It's a multi-class sentiment classification task with about 30,000 rows. I have already put everything on cuda, but not sure why I'm getting the following run time error. Here is my code:
for epoch in tqdm(range(1, epochs+1)):

    model.train()

    loss_train_total = 0

    progress_bar = tqdm(dataloader_train, desc='Epoch {:1d}'.format(epoch), leave=False, disable=False)
    for batch in progress_bar:

        model.zero_grad()

        batch = tuple(b.to(device) for b in batch)

        inputs = {'input_ids':      batch[0],
                  'attention_mask': batch[1],
                  'labels':         batch[2],
                 }

        outputs = model(**inputs)

        loss = outputs[0]
        loss_train_total += loss.item()
        loss.backward()

        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)

        optimizer.step()
        scheduler.step()

        progress_bar.set_postfix({'training_loss': '{:.3f}'.format(loss.item()/len(batch))})

    torch.save(model.state_dict(), f'finetuned_BERT_epoch_{epoch}.model')

    tqdm.write(f'\nEpoch {epoch}')

    loss_train_avg = loss_train_total/len(dataloader_train)
    tqdm.write(f'Training loss: {loss_train_avg}')

    val_loss, predictions, true_vals = evaluate(dataloader_validation)
    val_f1 = f1_score_func(predictions, true_vals)
    tqdm.write(f'Validation loss: {val_loss}')
    tqdm.write(f'F1 Score (Weighted): {val_f1}')

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-67-9306225bb55a> in <module>()
     17                  }
     18
---> 19         outputs = model(**inputs)
     20
     21         loss = outputs[0]

8 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
   1850         # remove once script supports set_grad_enabled
   1851         _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> 1852     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
   1853
   1854

RuntimeError: Input, output and indices must be on the current device

Any suggestions would be appreciated. Thanks!

Output:
You should put your model on the device, which is probably cuda:
device = "cuda:0"
model = model.to(device)

Then make sure the inputs of the model(input) are on the same device as well:
input = input.to(device)

It should work!


Instruction: cuda is not available on my pytorch, but I can't find anything wrong with the version

Input:
for some reason I have to use cuda version10.0 instead of upgrading it
The version of DriverAPI is higher than RunTimeAPI but somebody told me thats OK
Others who asked the same question at last found their version was not match. emmmm not like me
details here
OS:Windows10
Python 3.7&3.8 both tried
result of 'nvcc -V':
nvcc: release 10.0, V10.0.130
nvidia-smi:
NVIDIA-SMI 451.67       Driver Version: 451.67       CUDA Version: 11.0
conda list: cudatoolkit  10.0.130  0
import torch
print(torch.version.cuda)-----None
torch.cuda.is_available()-----False

Output:
How did you install it ? I assume with pip. For pytorch I would recommend manually downloading the wheel from https://download.pytorch.org/whl/torch_stable.html and install it with:
    pip install torch-1.4.0+cu100-cp38-cp38-linux_x86_64.whl

Assuming python 3.8 and linux. If you use something different make sure to select the appropriate version for your OS, Cuda version and python interpreter.


Instruction: Pytorch: use pretrained vectors to initialize nn.Embedding, but this embedding layer is not updated during the training

Input:
I initialized nn.Embedding with some pretrain parameters (they are 128 dim vectors), the following code demonstrates how I do this:
self.myvectors = gensim.models.KeyedVectors.load_word2vec_format(cfg.vec_dir)
self.vec_weights = torch.FloatTensor(self.myvectors.vectors)
self.embeds = torch.nn.Embedding.from_pretrained(self.vec_weights)

cfg.vec_dir is a json file where vec_dir indicates the path of the pretrained 128 dim vectors I used to initialize this layer.
After the model is trained, I print out this embedding layer, and I found that the parameters are exactly the same as I initialized them, so clearly the parameters are not updated during the training. Why is this happening? What should I do in order to update these vectors?

Output:
The torch.nn.Embedding.from_pretrained classmethod by default freezes the parameters. If you want to train the parameters, you need to set the freeze keyword argument to False. See the documentation.
So you might try this instead:
self.embeds = torch.nn.Embedding.from_pretrained(self.vec_weights, freeze=False)


Instruction: Training Data Split across GPUs in DDP Pytorch Lightning

Input:
Goal: Train a model in Distriubted Data Parallel(DDP) setting using Pytorch Lightning Framework
Questions:

Training Data Partition: How is data partition across separate GPUs is handled with Pytorch Lightning? Am I supposed to manually partition the data or Pytorch lightning will take care of that?

Loss Averaging: Do I have to aggregate the losses myself or Pytorch Lightning is going automatically do that?

I have been spending time with the code-base of pytorch lightning, looking for how DDP sync is handled, but unable to find that exact code. Would appreciate a clarification on this.

Output: Lightning handles both of these scenarios for you out of the for you but it can be overridden. The code for this can be found in the official github here.


Instruction: The result is empty when prediction of Faster RCNN model (Pytorch)

Input:
I'm trying to train Faster RCNN model. After training, I try to predict the result of image but the result is empty.
My data is w: 1600, h: 800, c: 3, classes: 7, bounding boxes:(x1, y1, x2, y2)
My model is below.

My model
import torchvision
from torchvision.models.detection.faster_rcnn import FastRCNNPredictor
from torchvision.models.detection.mask_rcnn import MaskRCNNPredictor
from torchvision.models.detection import FasterRCNN
from torchvision.models.detection.rpn import AnchorGenerator

def get_instance_segmentation_model(num_classes):
    backbone = torchvision.models.vgg16(pretrained=True).features
    backbone.out_channels = 512

    anchor_generator = AnchorGenerator(sizes=((32, 64, 128, 256, 512),),
                                    aspect_ratios=((0.5, 1.0, 2.0),))

    roi_pooler = torchvision.ops.MultiScaleRoIAlign(featmap_names=['0'],
                                                    output_size=7,
                                                    sampling_ratio=2)
    model = FasterRCNN(backbone,
                    num_classes=2,
                    rpn_anchor_generator=anchor_generator,
                    box_roi_pool=roi_pooler)
    in_features = model.roi_heads.box_predictor.cls_score.in_features
    model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)

    return model

device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')

num_classes = 2

model = get_instance_segmentation_model(num_classes)

model.to(device)

params = [p for p in model.parameters() if p.requires_grad]
optimizer = torch.optim.SGD(params, lr=0.005,
                            momentum=0.9, weight_decay=0.0005)

lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer,
                                               step_size=3,
                                               gamma=0.1)

training
# let's train it for 10 epochs
num_epochs = 10

for epoch in range(num_epochs):
    # train for one epoch, printing every 10 iterations
    train_one_epoch(model, optimizer, train_data_loader, device, epoch, print_freq=10)
    # update the learning rate
    lr_scheduler.step()
    # evaluate on the test dataset
    evaluate(model, valid_data_loader, device=device)

prediction:
prediction
[{'boxes': tensor([], device='cuda:0', size=(0, 4)),
  'labels': tensor([], device='cuda:0', dtype=torch.int64),
  'scores': tensor([], device='cuda:0')}]

Output:
You should change the number of classes to
    model = FasterRCNN(backbone,
                    num_classes=YOUR_CLASSES+1,  # +1 is for the background
                    rpn_anchor_generator=anchor_generator,
                    box_roi_pool=roi_pooler)

Remember that the class 0 is reserved for the background, so your classes should start from 1.
Also, please make sure your network has converged during the training.


Instruction: What is the most efficient way to broadcast an operation on slices of PyTorch Tensors?

Input:
I have a tensor T of shape (b, r)
I want to do an operation for each (r), in a way that it gets parallelized by the GPU
The naive implementation, in numpy for simplicity, would look something like:
T_dash = np.array([(T[i] - np.max(T[i]) for i in range(T.size[0])])

What would be the best way to do this?

Output:
There's a new vmap function available (in the master branch at the time of writing, experimental) that will help do batch operations, where you define the operation to be performed for each element.
vmap can be helpful in hiding batch dimensions. In your case, it goes something like
def each_elem_fn(tensor):
  return tensor - np.max(tensor)

torch.vmap(each_elem_fn)(batch_of_elems)

Note: As @jodag mentioned in the comment, for simply subtracting max you can use
T = T - T.max(dim=1, keepdim=True)[0]

however, if you want to generalize it to any custom function you shall use vmap.


Instruction: DistributedSampler — Expected a ‘cuda’ device type for generator when generating indices

Input:
Performing distributed training, I have the following code like this:
training_sampler = DistributedSampler(training_set, num_replicas=2, rank=0)
training_generator = data.DataLoader(training_set, **params, sampler=training_sampler)
for x, y, z in training_generator:  # Error occurs here.
    ...

Overall, I get the following message:
-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
    fn(i, *args)
  File "/home/ubuntu/VC/ppg_training_extraction/ppg_training_scripts/train_ASR_trim_scp.py", line 336, in train
    for local_batch_src, local_batch_tgt, lengths in dataloaders[phase]:
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 352, in __iter__
    return self._get_iterator()
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 294, in _get_iterator
    return _MultiProcessingDataLoaderIter(self)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 827, in __init__
    self._reset(loader, first_iter=True)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 857, in _reset
    self._try_put_index()
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1091, in _try_put_index
    index = self._next_index()
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 427, in _next_index
    return next(self._sampler_iter)  # may raise StopIteration
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/torch/utils/data/sampler.py", line 227, in __iter__
    for idx in self.sampler:
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/torch/utils/data/distributed.py", line 97, in __iter__
    indices = torch.randperm(len(self.dataset), generator=g).tolist()  # type: ignore
RuntimeError: Expected a 'cuda' device type for generator but found 'cpu'

Now at that line, I ran the following instructions in pdb:
(Pdb) g = torch.Generator()
(Pdb) g.manual_seed(0)
<torch._C.Generator object at 0x7ff7f8143110>
(Pdb) indices = torch.randperm(4556, generator=g).tolist()
(Pdb) indices = torch.randperm(455604, generator=g).tolist()
*** RuntimeError: Expected a 'cuda' device type for generator but found 'cpu'

Why am I getting the runtime error when the upperbound integer is high, but not when it’s low enough?
Note, I ran on a clean Python session and found
>>> import torch
>>> g = torch.Generator()
>>> g.manual_seed(0)
<torch._C.Generator object at 0x7f9d2dfb39f0>
>>> indices = torch.randperm(455604, generator=g).tolist()

that this worked fine. Is it some configuration in how I’m handling distributed training among multiple GPUs? Any sort of insights would be appreciated!

Output:
I just met the same problem using dataloader and I found the following helps without removing torch.set_default_tensor_type('torch.cuda.FloatTensor')
data.DataLoader(..., generator=torch.Generator(device='cuda'))
since I don't want to manually add .to('cuda') for tons of tensors in my code


Instruction: Installing Python dependencies in Heroku

Input:
I wanted to make an Ai chatbot hosted on Heroku, but there are some problems with installing the requirements.
The chatbot needs the following packages:

discord
pytorch
spaCy

I already figured out putting discord into the requirements.txt, but I got no clue on how to do the other two dependencies.
The problem with them is that they aren't just installed with
pip install NAME_OF_THE_PACKAGE .
Pytorch for example needs:
pip install torch==1.7.0+cpu torchvision==0.8.1+cpu torchaudio==0.7.0 -f https://download.pytorch.org/whl/torch_stable.html

and spaCy needs
pip install -U spacy
pip install -U spacy-lookups-data
python -m spacy download en_core_web_sm.
I tried different variations of the commands above, but none worked.
Now I even reached the built limit on my account.
Please help me if you know how to solve the problem.

Output:
You can use the following to create a requirements.txt file having all the dependency installed in your current environment. This will install
pip freeze > requirements.txt

To download the Spacy Models, you can add this in the requirements.txt
https://github.com/explosion/spacy-models/releases/download/en_core_web_trf-3.0.0a0/en_core_web_trf-3.0.0a0.tar.gz

For more info, go through this issue on Github
https://github.com/explosion/spaCy/issues/1129


Instruction: Pytorch - AttributeError: 'tuple' object has no attribute 'dim'

Input:
I am trying to use this architecture:
class Net(BaseFeaturesExtractor):

    def __init__(self, observation_space: gym.spaces.Box, features_dim: int = 512):
        super(Net, self).__init__(observation_space, features_dim)

        self.conv1 = nn.Conv2d(1, 64, 3, stride=1, padding=1)
        self.conv2 = nn.Conv2d(64, 64, 3, stride=1, padding=1)
        self.conv3 = nn.Conv2d(64, 64, 3, stride=1)
        self.conv4 = nn.Conv2d(64, 64, 3, stride=1)

        self.bn1 = nn.BatchNorm2d(64)
        self.bn2 = nn.BatchNorm2d(64)
        self.bn3 = nn.BatchNorm2d(64)
        self.bn4 = nn.BatchNorm2d(64)

        self.fc1 = nn.Linear(64 * (7 - 4) * (6 - 4), 128)
        self.fc_bn1 = nn.BatchNorm1d(128)

        self.fc2 = nn.Linear(128, 64)
        self.fc_bn2 = nn.BatchNorm1d(64)

        self.fc3 = nn.Linear(64, 7)

        self.fc4 = nn.Linear(64, 1)

    def forward(self, s):
        #                                                            s: batch_size x board_x x board_y
        s = s.view(-1, 1, 7, 6)  # batch_size x 1 x board_x x board_y
        s = F.relu(self.bn1(self.conv1(s)))  # batch_size x num_channels x board_x x board_y
        s = F.relu(self.bn2(self.conv2(s)))  # batch_size x num_channels x board_x x board_y
        s = F.relu(self.bn3(self.conv3(s)))  # batch_size x num_channels x (board_x-2) x (board_y-2)
        s = F.relu(self.bn4(self.conv4(s)))  # batch_size x num_channels x (board_x-4) x (board_y-4)
        s = s.view(-1,64 * (7 - 4) * (6 - 4))

        s = F.dropout(
            F.relu(self.fc_bn1(self.fc1(s))),
            p=0.3,
            training=self.training)  # batch_size x 128
        s = F.dropout(
            F.relu(self.fc_bn2(self.fc2(s))),
            p=0.3,
            training=self.training)  # batch_size x 64

        pi = self.fc3(s)  # batch_size x action_size
        v = self.fc4(s)  # batch_size x 1

        return F.log_softmax(pi, dim=1), th.tanh(v)

When I am trying to use this architecture, I am getting following error:
Traceback (most recent call last):
  File "/Users/joe/Documents/JUPYTER/ConnectX/training3.py", line 130, in <module>
    learner.learn(total_timesteps=iterations, callback=eval_callback)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/stable_baselines3/ppo/ppo.py", line 264, in learn
    reset_num_timesteps=reset_num_timesteps,
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/stable_baselines3/common/on_policy_algorithm.py", line 222, in learn
    continue_training = self.collect_rollouts(self.env, callback, self.rollout_buffer, n_rollout_steps=self.n_steps)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/stable_baselines3/common/on_policy_algorithm.py", line 154, in collect_rollouts
    actions, values, log_probs = self.policy.forward(obs_tensor)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/stable_baselines3/common/policies.py", line 545, in forward
    latent_pi, latent_vf, latent_sde = self._get_latent(obs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/stable_baselines3/common/policies.py", line 564, in _get_latent
    latent_pi, latent_vf = self.mlp_extractor(features)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/stable_baselines3/common/torch_layers.py", line 220, in forward
    shared_latent = self.shared_net(features)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/nn/modules/container.py", line 117, in forward
    input = module(input)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/nn/modules/linear.py", line 93, in forward
    return F.linear(input, self.weight, self.bias)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/nn/functional.py", line 1688, in linear
    if input.dim() == 2 and bias is not None:
AttributeError: 'tuple' object has no attribute 'dim'

How this problem can be fixed?

Output:
I tried to reproduce a small working code based on the class definitions given by you and I was able to get the outputs from the model. Here is the following code:
# BaseFeaturesExtractor class
import gym
import torch as th
from torch import nn

class BaseFeaturesExtractor(nn.Module):
    """
    Base class that represents a features extractor.
    :param observation_space:
    :param features_dim: Number of features extracted.
    """

    def __init__(self, observation_space: gym.Space, features_dim: int = 0):
        super(BaseFeaturesExtractor, self).__init__()
        assert features_dim > 0
        self._observation_space = observation_space
        self._features_dim = features_dim

    @property
    def features_dim(self) -> int:
        return self._features_dim

    def forward(self, observations: th.Tensor) -> th.Tensor:
        raise NotImplementedError()

# Net class
class Net(BaseFeaturesExtractor):

    def __init__(self, observation_space: gym.spaces.Box, features_dim: int = 512):
        super(Net, self).__init__(observation_space, features_dim)

        self.conv1 = nn.Conv2d(1, 64, 3, stride=1, padding=1)
        self.conv2 = nn.Conv2d(64, 64, 3, stride=1, padding=1)
        self.conv3 = nn.Conv2d(64, 64, 3, stride=1)
        self.conv4 = nn.Conv2d(64, 64, 3, stride=1)

        self.bn1 = nn.BatchNorm2d(64)
        self.bn2 = nn.BatchNorm2d(64)
        self.bn3 = nn.BatchNorm2d(64)
        self.bn4 = nn.BatchNorm2d(64)

        self.fc1 = nn.Linear(64 * (7 - 4) * (6 - 4), 128)
        self.fc_bn1 = nn.BatchNorm1d(128)

        self.fc2 = nn.Linear(128, 64)
        self.fc_bn2 = nn.BatchNorm1d(64)

        self.fc3 = nn.Linear(64, 7)

        self.fc4 = nn.Linear(64, 1)

    def forward(self, s):
        #                                                            s: batch_size x board_x x board_y
        s = s.view(-1, 1, 7, 6)  # batch_size x 1 x board_x x board_y
        s = F.relu(self.bn1(self.conv1(s)))  # batch_size x num_channels x board_x x board_y
        s = F.relu(self.bn2(self.conv2(s)))  # batch_size x num_channels x board_x x board_y
        s = F.relu(self.bn3(self.conv3(s)))  # batch_size x num_channels x (board_x-2) x (board_y-2)
        s = F.relu(self.bn4(self.conv4(s)))  # batch_size x num_channels x (board_x-4) x (board_y-4)
        s = s.view(-1,64 * (7 - 4) * (6 - 4))

        s = F.dropout(
            F.relu(self.fc_bn1(self.fc1(s))),
            p=0.3,
            training=self.training)  # batch_size x 128
        s = F.dropout(
            F.relu(self.fc_bn2(self.fc2(s))),
            p=0.3,
            training=self.training)  # batch_size x 64

        pi = self.fc3(s)  # batch_size x action_size
        v = self.fc4(s)  # batch_size x 1

        return F.log_softmax(pi, dim=1), th.tanh(v)

# Minimal code to reproduce a forward pass

import numpy as np
import torch
import torch.nn.functional as F

params = gym.spaces.Box(np.array([-1,0,0]), np.array([+1,+1,+1]))
model = Net(params)
inputs = torch.randn(2, 1, 7, 6)
outputs = model(inputs)
print(outputs[0].shape, outputs[1].shape) # prints (torch.Size([2, 7]), torch.Size([2, 1]))


Instruction: Image augmentation in Pytorch

Input:
I like to augment image alternately.
I have pytorch transform code as follows.
import torchvision.transforms as tt
from torchvision.datasets import ImageFolder
#Data transform (normalization & data augmentation)
stats = ((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010))
train_tfms = tt.Compose([tt.RandomCrop(32, padding = 4, padding_mode = 'reflect'),
                         tt.RandomHorizontalFlip(),
                         tt.RandomAffine(degrees=(10, 30),
                                         translate=(0.1, 0.3),
                                         scale=(0.7, 1.3),
                                         shear=0.1,
                                         resample=Image.BICUBIC)
                         tt.ToTensor(),
                         tt.Normalize(*stats)])

When I create dataset as follow and do training, all images will be augmented.
train_ds = ImageFolder('content/train', train_tfms)

But I want alternately. First image, just train as original image. But the next image is augmented.
How can I do that?

Output:
From a single dataset you can create two datasets one with augmentation and the other without, and then concatenate them. The order is going to be kept since we are using the subdataset pytorch class which will handle this for us.
train_ds_no_aug = ImageFolder('content/train')
train_ds_aug = ImageFolder('content/train', train_tfms)

# Check that aug_idx and no_aug_idx are not overlapping
aug_idx = torch.arange(1, len(train_ds_no_aug), 2)
no_aug_idx = torch.arange(0, len(train_ds_no_aug), 2)

train_ds_no_aug = torch.utils.data.Subset(train_ds_no_aug, no_aug_idx)
train_ds_aug = torch.utils.data.Subset(train_ds_aug, aug_idx)

train_ds = torch.utils.data.ChainDataset([train_ds_no_aug, train_ds_aug])
# Done :=


Instruction: Multiplying two 3D Pytorch tensors iteratively

Input:
I have two 3 dimensional Pytorch tensors, one of dimension (8, 1, 1024) and the other has dimension (8, 59, 77). I wish to multiply these two tnesors.
I know they cannot be multiplied in their current state, so I want to multiply them iteratively and append into a single tensor.
The second tensor can be represented as (8, 59, 1) when we iterate over the 2nd dimension. In this state multiplying it with the first tensor of shape (8, 1, 1024), resulting in a tensor of shape (8, 59, 1024), and finally appending all these 77 outputs into one, resulting in the final shape of (8, 59, 1024, 77).
However I am having issues in it's implementation. Can someone help me here ?

Output:
If I didn't mess up the computation, it would be equivalent to:
import torch

x = torch.rand(8, 1, 1024)
y = torch.rand(8, 59, 77)

torch.matmul(
    y.unsqueeze(-1),   # shape = (8, 59, 77, 1)
    x.unsqueeze(1)     # shape = (8, 1, 1, 1024)
).permute(0, 1, 3, 2)  # output shape = (8, 59, 1024, 77)

Note that, in this case, matmul performs a batched matrix multiply.


Instruction: LSTM to Predict Pattern 010101... Understanding Hidden State

Input:
I did a quick experiment to see if I could understand what the hidden state in an LSTM does...
I tried to make an LSTM predict a sequence of [1,0,1,0,1...] based off an input sequence of X with X[0] = 1 and the remainder as random noise.
X = [1, randFloat, randFloat, randFloat...]
label = [1, 0, 1, 0...]

In my head, the model would understand:

The inputs X mean nothing, or at least very little (as it's noise) - so it'd discard these values for the most part
Solely the hidden state from the previous sequence/timestep n would be used to predict the next timestep n+1... [1, 0, 1, 0...]
I also set X[0] = 1 so the first initial in an attempt to guide the net to predicting 1 on the first item (which it does)

So, this didn't work. In theory, should it not? Can you someone explain?
It essentially never converges, and is on the cusp of guessing between 0 or 1

## Code
import os
import numpy as np
import torch

from torchvision import transforms
from torch import nn
from sklearn import preprocessing
from util import create_sequences
import torch.optim as optim

Create some fake data
sequence_1 = torch.tensor(np.random.uniform(size=50)).float().detach()
sequence_1[0] = 1
sequence_2 = torch.tensor(np.random.uniform(size=50)).float().detach()
sequence_2[0] = 1

labels_1 = np.zeros(50)
labels_1[::2] = 1
labels_1 = torch.tensor(labels_1, dtype=torch.long)
labels_2 = labels_1.clone()

training_data = [sequence_1, sequence_2]
label_data = [labels_1, labels_2]

Create simple LSTM Model
class LSTM(nn.Module):
    def __init__(self, input_dim, hidden_dim, output_dim):
        super(LSTM, self).__init__()
        self.lstm = nn.LSTM(input_dim, hidden_dim)
        self.fc = nn.Linear(hidden_dim, output_dim)

    def forward(self, seq):
        lstm_out, _ = self.lstm(seq.view(len(seq), 1, -1))
        out = self.fc(lstm_out.view(len(seq), -1))
        out = F.log_softmax(out, dim=1)
        return out

We try to overfit on the dataset
INPUT_DIM = 1
HIDDEN_DIM = 6
model = LSTM(INPUT_DIM, HIDDEN_DIM, 2)

loss_function = nn.NLLLoss()
optimizer = optim.SGD(model.parameters(), lr=0.1)

for epoch in range(500):
    for i, seq in enumerate(training_data):
        labels = label_data[i]
        model.zero_grad()
        scores = model(seq)
        loss = loss_function(scores, labels)
        loss.backward()
        print(loss)

        optimizer.step()

with torch.no_grad():
    seq_d = training_data[0]
    tag_scores = model(seq_d)
    for score in tag_scores:
        print(np.argmax(score))

Output:
I would say it's not meant to work.
The model would always try to make sense and find patterns in the data it's trained on i.e sequence_1 and to "verify" that it has "found" them, it uses labels_1. Since the data is random the model fails to find the pattern.
The pattern the model tries to find is not in the label but in the data, so it doesn't matter how the label is arranged. The label actually never passes through the model, so NO.
If perhaps, you trained it on a single example then definitely. The model will become overfit and give you your ones and zeros and fail miserably on other examples, otherwise it just won't be able to make sense of the random data no matter the size.
Hidden State

Solely the hidden state from the previous sequence/timestep n would be used to predict the next timestep n+1... [1, 0, 1, 0...]

Concerning Hidden state, NOTE that it is not a trainable parameter, it is the result of performing some operations on the data and parameters, meaning that the input data determines the Hidden state.
What the Hidden state does is to hold the information the model has extracted from the previous timesteps and passes it to the next timestep or as output. In the case of LSTM, it does some forgetting and updating before passing it.


Instruction: How to create and use PyTorch learnable scalar variables outside of nn.Module?

Input:
I am working on a multi-objective problem where I have multiple losses that I need to compute, and the total loss is just the sum of the losses. I want to have PyTorch learnable floating-point parameters alpha, and beta that act as coefficients to the individual losses. Note that the summation of losses occurs outside my NN model in the training loop:
optimizer = AdamW(model.parameters(), lr=2e-5, eps=1e-8)

for batch in dl:

    optimizer.zero_grad()

    result = model(batch)

    loss1 = loss_fn_1(result)
    loss2 = loss_fn_2(result)
    loss3 = loss_fn_3(result)

    loss = alpha*loss1 + beta*loss2 + (1-beta)*loss3 # How to optimize alpha, beta?

    loss.backward()
    optimizer.step()

How would I declare and use the learnable parameters alpha and beta?

Output:
You can put them into a list and add them to an optimizer, for example,
optimizer_for_my_params = torch.Adam([alpha, beta], lr=1e-3)

or separately,
optimizer_alpha = torch.Adam([alpha], lr=1e-3)
optimizer_beta = torch.Adam([beta], lr=1e-3)

and at each step, call zero_grad and step on all optimizers.
Or you can put them in an nn.Module and declare them as parameters:
class MyParams(nn.Module):
  def __init__(self):
    super(MyParams, self).__init__()

    self.alpha = nn.Parameter(torch.tensor(0.))
    self.beta = nn.Parameter(torch.tensor(0.))

  def forward(self, loss1, loss2, loss3):
    loss = self.alpha*loss1 + self.beta*loss2 + (1 - self.beta)*loss3
    return loss

When using it, defining a separate optimizer for the class object should do the job.

Update:
Here is a more comprehensive example for the first method.
import torch
import torch.optim as optim

alpha = torch.tensor(0.)
alpha.requires_grad = True
optimizer_alpha = optim.Adam([alpha], lr=1e-3)

print(optimizer_alpha)
# Adam (
# Parameter Group 0
#     amsgrad: False
#     betas: (0.9, 0.999)
#     eps: 1e-08
#     lr: 0.001
#     weight_decay: 0
# )

out = alpha + 1

# test backward()
optimizer_alpha.zero_grad()
out.backward()
print(alpha.grad)
# tensor(1.)

# test step()
optimizer_alpha.step()
print(alpha)
# tensor(-0.0010, requires_grad=True)


Instruction: Pytorch CNN:RuntimeError: Given groups=1, weight of size [16, 16, 3], expected input[500, 1, 19357] to have 16 channels, but got 1 channels instead

Input:
class ConvolutionalNetwork(nn.Module):
    def __init__(self, in_features, trial):
        super().__init__()
        self.in_features = in_features
        self.trial = trial
        # this computes num features outputted from the two conv layers
        c1 = int(((self.in_features - 2)) / 64)  # this is to account for the loss due to conversion to int type
        c2 = int((c1 - 2) / 64)
        self.n_conv = int(c2 * 16)
        # self.n_conv = int((( ( (self.in_features - 2)/4 ) - 2 )/4 ) * 16)
        self.conv1 = nn.Conv1d(16, 16, 3, 1)
        self.conv1_bn = nn.BatchNorm1d(16)
        self.conv2 = nn.Conv1d(16, 16, 3, 1)
        self.conv2_bn = nn.BatchNorm1d(16)
        # self.dp = nn.Dropout(trial.suggest_uniform('dropout_rate',0,1.0))
        self.dp = nn.Dropout(0.5)
        self.fc3 = nn.Linear(self.n_conv, 2)

    def forward(self, x):
        # shape x for conv 1d op
        x = x.view(-1, 1, self.in_features)
        x = self.conv1(x)
        x = F.tanh(x)
        x = F.max_pool1d(x, 64, 64)
        x = self.conv2(x)
        x = F.tanh(x)
        x = F.max_pool1d(x, 64, 64)
        x = x.view(-1, self.n_conv)

        x = self.dp(x)
        x = self.fc3(x)
        x = F.log_softmax(x, dim=1)

        return x

Ran the code above and this error popped up :
RuntimeError: Given groups=1, weight of size [16, 16, 3], expected input[500, 1, 19357] to have 16 channels, but got 1 channels instead.

Anyone able to advise on this? It says discrepancies in input , but the above codes works well earlier , unsure what happened after I re-arranged the code.

Output:
Well, just after entering in the forward method you are reshaping your input array so it has only a single channel:
x = x.view(-1, 1, self.in_features)

And at the same time at the model constructor you are specifying that conv1 has 16 channels as input:
self.conv1 = nn.Conv1d(16, 16, 3, 1)

Thus the error of expecting 16 channels but received 1.
There are two things to note here:

If you are used to tensorflow, maybe you are thinking that channels are the last dimension but in pytorch channels are located at the first dimension. Take a look at the Conv1d torch documentation. Take this into account when reshaping the data.
Conv1d are agnostic to the length of your input (I am telling you this just in case in_features represents the length)

I Cannot provide you with a concrete solution since I am not sure of what you are trying to do.


Instruction: How to get number of classes from .pth file without any model information in Python?

Input:
I have a .pth file and there is no model information available. How can I know the number of final output classes in .pth file.
For visualization, I can use Netron to see the number of classes. But, how can I get the same output number in Python.

Output:
You can load the checkpoint and inspect the shape of the weights of the last layer. Depending if the last layer has two weights (kernel and bias), you will have to inspect both.
I provide you with an example on how to inspect the last weight.
import torch

checkpoint = torch.load('_.pt')

last_key = list(checkpoint)[-1]
print(checkpoint[last_key].size())