Instruction: Assigning weights to the feature maps of a convolutional layer

Input:
I'm using a class SE_Block (squeeze and excitation block) and give N feature maps (channels) of a convolutional layer as the input to this SE_Block. My goal is that after using the SE_Block, each of its input feature maps obtain their own weight. In other words, the SE_Block aims to assign a weight for each of the feature maps.
But I face the following error:

TypeError: init() missing 1 required positional argument: 'c'

class SE_Block(nn.Module):
"credits: https://github.com/moskomule/senet.pytorch/blob/master/senet/se_module.py#L4"
def __init__(self, c, r=16):
    super().__init__()
    self.squeeze = nn.AdaptiveAvgPool2d(1)
    self.excitation = nn.Sequential(
        nn.Linear(c, c // r, bias=False),
        nn.ReLU(inplace=True),
        nn.Linear(c // r, c, bias=False),
        nn.Sigmoid()
    )

def forward(self, x):
    bs, c, _, _ = x.shape
    y = self.squeeze(x).view(bs, c)
    y = self.excitation(y).view(bs, c, 1, 1)
    return x * y.expand_as(x)

My code:
class myclass(nn.Module):
def __init__(self, in_channel=1024, out_channel=[512], out_sigmoid=False):
    super(myclass, self).__init__()

    self.out_sigmoid=out_sigmoid

    self.SEBlock = SE_Block()

    self.deconvlayer = self._make_deconv(in_channel, out_channel[0], num_conv=3)
    self.upsample=Upsample(scale_factor=2, mode='bilinear')

def _make_deconv(self, in_channel, out_channel, num_conv=2, kernel_size=3, stride=1, padding=1):
    layers=[]
    layers.append(BasicConv2d(in_channel, out_channel ,kernel_size=kernel_size, stride=stride, padding=padding))
    for i in range(1, num_conv):
        layers.append(_SepConv2d(out_channel, out_channel,kernel_size=kernel_size, stride=stride, padding=padding))

    return nn.Sequential(*layers)

def forward(self, x):
    x=self.deconvlayer(x)
    x = self.upsample(x)
    w = self.SEBlock(x)

    return x, w

Output:
When you're creating the SE_block you're not passing the c (channel) argument.
You need to add:
class myclass(nn.Module):
    def __init__(self, in_channel=1024, out_channel=512, out_sigmoid=False):

        ...

        self.SEBlock = SE_Block(out_channel)   # Adding argument here

        ...

You also have some errors with the forward part in your class implementation, it should be rewritten like:
class myclass(nn.Module):

    ...

    def forward(self, x):
        x = self.deconvlayer(x)  # Remove the 5_5 part
        x = self.upsample(x)     # Remove the 5_5 part
        w = self.SEBlock(x)

        return x, w


Instruction: How define the number of class in Detectron2 bounding box predict Pytorch?

Input:
Where should i define the number os classes ?
ROI HEAD or RETINANET ?
Or both should have the same value ?
cfg.MODEL.RETINANET.NUM_CLASSES =int( len(Classe_list)-1)

cfg.MODEL.ROI_HEADS.NUM_CLASSES=int( len(Classe_list)-1)

Output:
It depends on the network architecture you choose to use. If you use the "MaskRCNN", then you should set the cfg.MDOEL.ROI_HEADS.NUM_CLASSES.
The deep reason is that ROI_HEAD is the component used by MaskRCNN. If you use different network, you may need to change different things dependent on their implementation


Instruction: Automatically check available GPU on Google Colab

Input:
Is there an automatic way to check which GPU is currently available on Google Colab (Pro).
Say I would like to use a Tesla P100 instead of the Tesla T4 to train my model, is there a way to periodically check with a python script in Colab whether the P100 is available?
I have tried eliminate the kernel periodically but it won't restart again automatically after shutting down:
import os

def restart_runtime():
   os.kill(os.getpid(), 9)

Thank you

Output:
There is no way to check what GPU is available. Add the line:
!nvidia-smi

to the beginning of your code and then keep on disconnecting and reconnecting the runtime until you get the GPU that you want.


Instruction: Curious loss in a CNN

Input:
in my CNN for image classification, I get a curious loss and I don't know what's wrong. I'm lucky, if you help me to find the failure.
Here is a cutout of my print output and at the end there is my code:
Train Epoch: 1 [0/2048 (0%)]    Loss: 0.654869
Train Epoch: 1 [64/2048 (3%)]   Loss: 0.271722
Train Epoch: 1 [128/2048 (6%)]  Loss: 0.001958
Train Epoch: 1 [192/2048 (9%)]  Loss: 0.003399
Train Epoch: 1 [256/2048 (12%)] Loss: 0.000000
Train Epoch: 1 [320/2048 (16%)] Loss: 0.006664
Train Epoch: 1 [384/2048 (19%)] Loss: 0.000000
Train Epoch: 1 [448/2048 (22%)] Loss: 0.000000
Train Epoch: 1 [512/2048 (25%)] Loss: 0.000000
Train Epoch: 1 [576/2048 (28%)] Loss: 0.000000

Train Epoch: 2 [0/2048 (0%)]    Loss: 173505.656250
Train Epoch: 2 [64/2048 (3%)]   Loss: 0.000000
Train Epoch: 2 [128/2048 (6%)]  Loss: 0.000000
Train Epoch: 2 [192/2048 (9%)]  Loss: 33394.285156
Train Epoch: 2 [256/2048 (12%)] Loss: 0.000000
Train Epoch: 2 [320/2048 (16%)] Loss: 0.000000

Train Epoch: 2 [960/2048 (47%)] Loss: 0.000000
Train Epoch: 2 [1024/2048 (50%)]        Loss: 636908.437500
Train Epoch: 2 [1088/2048 (53%)]        Loss: 32862667387437056.000000
Train Epoch: 2 [1152/2048 (56%)]        Loss: 15723443952412777718762887446528.000000
Train Epoch: 2 [1216/2048 (59%)]        Loss: nan
Train Epoch: 2 [1280/2048 (62%)]        Loss: nan
Train Epoch: 2 [1344/2048 (66%)]        Loss: nan
Train Epoch: 2 [1408/2048 (69%)]        Loss: nan

Here, you see code for the training.
def trainM(epoch):
    model.train()
    for batch_id, (data, target) in enumerate(net.train_data):
        target = torch.LongTensor(target[64*batch_id:64*(batch_id+1)])
        data = Variable(data)
        target = Variable(target)
        optimizer.zero_grad()

        out = model(data)
        criterion = F.nll_loss

        loss = criterion(out,target)
        loss.backward()
        optimizer.step()

        print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(epoch,batch_id*len(data), len(net.train_data)*64, 100*batch_id/len(net.train_data), loss.item()))

for item in range(1,10):
    trainM(item)

That's the code for neural network and the end there is the dataPrep method for data preparation.
train_data = []
target_list = []
class Netz(nn.Module):
    def __init__(self):
        super(Netz, self).__init__()
        self.conv1 = nn.Conv2d(1, 10,kernel_size=5)
        self.conv2 = nn.Conv2d(10,20, kernel_size = 5)
        self.conv_dropout = nn.Dropout2d()
        self.fc1 = nn.Linear(1050,60)
        self.fc2 = nn.Linear(60,2)
        self.fce = nn.Linear(20,1)

    def forward(self,x):
        x = self.conv1(x)
        x = F.max_pool2d(x, 2)
        x = F.relu(x)
        x = self.conv2(x)
        x = self.conv_dropout(x)
        x = F.max_pool2d(x,2)
        x = F.relu(x)
        x = x.reshape(x.shape[0], x.shape[1], -1)
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        x = self.fce(x.permute(0,2,1)).squeeze(-1)
        return F.log_softmax(x, -1)

def dataPrep(list_of_data, data_path, category, quantity):
    global train_data
    global target_list
    train_data_list = []

    transform = transforms.Compose([
    transforms.ToTensor(),
        ])

    len_data = len(train_data)
    for item in list_of_data:
        f = random.choice(list_of_data)
        list_of_data.remove(f)
        try:
            img = Image.open(data_path +f)
        except:
            continue
        img_crop = img.crop((310,60,425,240))
        img_tensor = transform(img_crop)
        train_data_list.append(img_tensor)

        if category == True:
            target = 1
        else:
            target = 0
        target_list.append(target)

        if len(train_data_list) >=64:
            train_data.append((torch.stack(train_data_list), target_list))
            train_data_list = []

        if (len_data*64 + quantity) <= len(train_data)*64:
            break
    return list_of_data

Output: I might also suggest that the network needs to be initialized with random parameters for the convolutional layer weights. By default these weights are 0, which probably means that you end up predicting all one class. This might explain the very low (0) or very high losses (based on the makeup of the particular batch).


Instruction: Fine tune/train a pre-trained BERT on data with no sentences but only words (bank transactions)

Input:
I have a lot of bank-transactions which I want to classify into different categories. The issue is that the text is not a sentence as such but consists only of words e.g "private withdrawal", "payment invoice 19234", "taxes" etc.
Since the domain is so specific, I think we might get a better performance by fine-tune a already pre-trained BERT, compared to just use the pre-trained BERT right away, but how do we do that when we don't have any sentences? I.e how would the "guess next sentence" part be created? Or can we skip it?

Output:
Your problem is a sequence classification problem. If you want to use a pre-trained model, you want to do transfer learning. Basically you want to use the Bert base model and add a layer of classification.
You can check huggingface for that https://huggingface.co/docs/transformers/model_doc/bert#transformers.BertForSequenceClassification
My answer isn't very specific, feel free to add details to your question


Instruction: Training multiple pytorch models on GPUs

Input:
I'm trying to implement something with pytorch.
I have 2 GPUs and I want to train 2 models as below:
model0 = Mymodel().to('cuda:0')
model1 = Mymodel().to('cuda:1')
opt0 = torch.optim.Adam(model0.parameters(), lr=0.01)
opt1 = torch.optim.Adam(model0.parameters(), lr=0.01)

# 1.Forward data into model0 on GPU0
out = model0(x.to('cuda:0'))

# 2.Calculate the loss on model0, update model0's parameters
model0.loss.backward()
opt0.step()
opt0.zero_grad()

# 3.Use model0's output as input of model1 on GPU1
out = model1(out.to('cuda:1'))

# 4.Calculate the loss on model1, update model1's parameters
model1.loss.backward()
opt1.step()
opt1.zero_grad()

I want to train them simultaneously to speed up the whole procedure, but I think the code now will wait step 2(or 4) finished and finally do step 3(or 1). How can I implement my idea? Or which technique is I need(e.g. model parel, thread, multiprocessing...)?
I've consider some article like this, but I think there is some worng with the result, and I think it actually doesn't train models simultaneously.

Output:
You have a strong dependency between the 2 models, the 2nd one always needs the output from the previous one, so that part of the code will always be sequential.
I think you might need some sort of multiprocessing (take a look at torch.multiprocessing) or some kind of queue, where you can store the output from the first model.


Instruction: While training BERT variant, getting IndexError: index out of range in self

Input:
While training XLMRobertaForSequenceClassification:
xlm_r_model(input_ids = X_train_batch_input_ids
            , attention_mask = X_train_batch_attention_mask
            , return_dict = False
           )

I faced following error:
Traceback (most recent call last):
  File "<string>", line 3, in <module>
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/transformers/models/roberta/modeling_roberta.py", line 1218, in forward
    return_dict=return_dict,
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/transformers/models/roberta/modeling_roberta.py", line 849, in forward
    past_key_values_length=past_key_values_length,
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/transformers/models/roberta/modeling_roberta.py", line 132, in forward
    inputs_embeds = self.word_embeddings(input_ids)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/sparse.py", line 160, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py", line 2044, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
IndexError: index out of range in self

Below are details:

Creating model
config = XLMRobertaConfig()
config.output_hidden_states = False
xlm_r_model = XLMRobertaForSequenceClassification(config=config)
xlm_r_model.to(device) # device is device(type='cpu')

Tokenizer
xlmr_tokenizer = XLMRobertaTokenizer.from_pretrained('xlm-roberta-large')

MAX_TWEET_LEN = 402

>>> df_1000.info() # describing a data frame I have pre populated
<class 'pandas.core.frame.DataFrame'>
Int64Index: 1000 entries, 29639 to 44633
Data columns (total 2 columns):
#    Column  Non-Null Count  Dtype
---  ------  --------------  -----
0    text    1000 non-null   object
1    class   1000 non-null   int64
dtypes: int64(1), object(1)
memory usage: 55.7+ KB

X_train = xlmr_tokenizer(list(df_1000[:800].text), padding=True, max_length=MAX_TWEET_LEN+5, truncation=True) # +5: a head room for special tokens / separators
>>> list(map(len,X_train['input_ids']))  # why its 105? shouldn't it be MAX_TWEET_LEN+5 = 407?
[105, 105, 105, 105, 105, 105, 105, 105, 105, 105, 105, 105, 105, 105, ...]

>>> type(train_index) # describing (for clarity) training fold indices I pre populated
<class 'numpy.ndarray'>

>>> train_index.size
640

X_train_fold_input_ids = np.array(X_train['input_ids'])[train_index]
X_train_fold_attention_mask = np.array(X_train['attention_mask'])[train_index]

>>> i # batch id
0
>>> batch_size
16

X_train_batch_input_ids = X_train_fold_input_ids[i:i+batch_size]
X_train_batch_input_ids = torch.tensor(X_train_batch_input_ids,dtype=torch.long).to(device)

X_train_batch_attention_mask = X_train_fold_attention_mask[i:i+batch_size]
X_train_batch_attention_mask = torch.tensor(X_train_batch_attention_mask,dtype=torch.long).to(device)

>>> X_train_batch_input_ids.size()
torch.Size([16, 105]) # why 105? Shouldnt this be MAX_TWEET_LEN+5 = 407?

>>> X_train_batch_attention_mask.size()
torch.Size([16, 105]) # why 105? Shouldnt this be MAX_TWEET_LEN+5 = 407?

After this I make the call xlm_r_model(...) as stated at the beginning of this question and ending up with the specified error.
Noticing all these details, I am still not able to get why I am getting the specified error. Where I am doing it wrong?

Output:
As per this post on github, there can be possibly many reasons for this. Below is the list of reasons summmarised from that post (as of April 24, 2022, note that 2nd and 3rd reasons are not tested):

Mismatching vocabulary size of tokenizer and bert model. This will cause the tokenizer to generate IDs that the model cannot understand. ref
Model and data to exist on different devices (CPUs, GPUs, TPUs)  ref
Sequences of length more than 512 (which is max for BERT-like models) ref

In my case, it was the first reason, mismatching vocab size and I have fixed this as follows:
Here is how I fixed this:
xlmr_tokenizer = XLMRobertaTokenizer.from_pretrained('xlm-roberta-large')
config = XLMRobertaConfig()
config.vocab_size = xlmr_tokenizer.vocab_size  # setting both to have same vocab size


Instruction: Resize to 224×224 directly or resize to 256 ×256 then crop 224×224？

Input:
My images in training set are leaves like this.
enter image description here
Its size is 572*108 and my resnet network need 224×224 images as input.
I found most of the codes process the images with the second way(resize to 256 ×256 then crop 224×224) so I did that.
As a result,parts of my leaves were cut which may influence the effect of leaves classification.Like this:
enter image description here
The first way (resize to 256×256 directly) may keep more complete structure.Like this:
enter image description here
I am worrying that if choose the first way I will lose many training images since random crop 224×224 may generate more samples for training.
Which should I choose？

Output:
Running two experiments and compare their evalution results is the simplest solution.
A complete view is not neccessary for model to classify images, so as it to human. On the contrary, learning from cropped image normally can improve the generalization capacity of a model.


Instruction: Is there a multi images to 1 image deep learning method? (pix2pix?)

Input:
I'm trying to build a video stabilization deep learning model.
I want to make the model predict how the frame should be stabilized depending on the last 10 frames
I have tried pix2pix, which is image to image, but I didn't get a good result
so, I want the same as pix2pix but multi images to 1 image
is there a method or can I do it using pix2pix?

Output:
So, I do not know if you actually need to build this video stabilization using deep learning or if you just want on off-the-shelves solution.
For the on-the-shelves solution, you can look into vidgear that has an awesome stabilisation system built-in: https://abhitronix.github.io/vidgear/latest/gears/stabilizer/overview/
If you want a more advanced solution and architecture, you could take a look at his thread of paper with code: https://paperswithcode.com/task/video-stabilization
Given the current architecture of pix2pix, I do not see how multi-images will provide some stabilisation since, it is just as you said, pix2pix does not consider its previous output nor the flow of images to generate its prediction.
I hope that it helps ^^


Instruction: Splitting a Tensor channelwise

Input: I am dumping a tensor of size [1,3,224,224] to a file and would like to split into 3 tensors of size [1,1,224,224], one for each RGB channel and dump them into 3 separate files. How do I implement this?

Output:
I think the simplest way is by a loop:
for c in range(x.shape[1]):
  torch.save(x[:, c:c+1, ...], f'channel{c}.pth')

Note the indexing of the second (channel) dimension: you want the saved tensor to have a singleton channel dimension. If you were to index it using x[:, c, ...] you will get a tensor of shape [1, 224, 224] (no channel dimension at all).


Instruction: Unable to convert the pytorch model to the TorchScript format

Input:
Loaded the pretrained PyTorch model file, and when I try to run it with torch.jit.script I get the below error, When I try to run the inbuilt pretrained model from pytorch.org it works perfectly fine. (Ex. Link to example code) but throws error for custom built pretrained model (Git repo containing the pretrained model weights), (pretrained weight)
encoder = enCoder()
encoder = torch.nn.DataParallel(encoder)
encoder.load_state_dict(weights['state_dict'])
encoder.eval()

torchscript_model = torch.jit.script(encoder)

# Error
---------------------------------------------------------------------------
NotSupportedError                         Traceback (most recent call last)
[<ipython-input-30-1d9f30e14902>](https://localhost:8080/#) in <module>()
      1 # torch.quantization.convert(encoder, inplace=True)
      2
----> 3 torchscript_model = torch.jit.script(encoder)

8 frames
[/usr/local/lib/python3.7/dist-packages/torch/jit/_script.py](https://localhost:8080/#) in script(obj, optimize, _frames_up, _rcb, example_inputs)
   1256         obj = call_prepare_scriptable_func(obj)
   1257         return torch.jit._recursive.create_script_module(
-> 1258             obj, torch.jit._recursive.infer_methods_to_compile
   1259         )
   1260

[/usr/local/lib/python3.7/dist-packages/torch/jit/_recursive.py](https://localhost:8080/#) in create_script_module(nn_module, stubs_fn, share_types, is_tracing)
    449     if not is_tracing:
    450         AttributeTypeIsSupportedChecker().check(nn_module)
--> 451     return create_script_module_impl(nn_module, concrete_type, stubs_fn)
    452
    453 def create_script_module_impl(nn_module, concrete_type, stubs_fn):

[/usr/local/lib/python3.7/dist-packages/torch/jit/_recursive.py](https://localhost:8080/#) in create_script_module_impl(nn_module, concrete_type, stubs_fn)
    461     """
    462     cpp_module = torch._C._create_module_with_type(concrete_type.jit_type)
--> 463     method_stubs = stubs_fn(nn_module)
    464     property_stubs = get_property_stubs(nn_module)
    465     hook_stubs, pre_hook_stubs = get_hook_stubs(nn_module)

[/usr/local/lib/python3.7/dist-packages/torch/jit/_recursive.py](https://localhost:8080/#) in infer_methods_to_compile(nn_module)
    730     stubs = []
    731     for method in uniqued_methods:
--> 732         stubs.append(make_stub_from_method(nn_module, method))
    733     return overload_stubs + stubs
    734

[/usr/local/lib/python3.7/dist-packages/torch/jit/_recursive.py](https://localhost:8080/#) in make_stub_from_method(nn_module, method_name)
     64     # In this case, the actual function object will have the name `_forward`,
     65     # even though we requested a stub for `forward`.
---> 66     return make_stub(func, method_name)
     67
     68

[/usr/local/lib/python3.7/dist-packages/torch/jit/_recursive.py](https://localhost:8080/#) in make_stub(func, name)
     49 def make_stub(func, name):
     50     rcb = _jit_internal.createResolutionCallbackFromClosure(func)
---> 51     ast = get_jit_def(func, name, self_name="RecursiveScriptModule")
     52     return ScriptMethodStub(rcb, ast, func)
     53

[/usr/local/lib/python3.7/dist-packages/torch/jit/frontend.py](https://localhost:8080/#) in get_jit_def(fn, def_name, self_name, is_classmethod)
    262         pdt_arg_types = type_trace_db.get_args_types(qualname)
    263
--> 264     return build_def(parsed_def.ctx, fn_def, type_line, def_name, self_name=self_name, pdt_arg_types=pdt_arg_types)
    265
    266 # TODO: more robust handling of recognizing ignore context manager

[/usr/local/lib/python3.7/dist-packages/torch/jit/frontend.py](https://localhost:8080/#) in build_def(ctx, py_def, type_line, def_name, self_name, pdt_arg_types)
    300                        py_def.col_offset + len("def"))
    301
--> 302     param_list = build_param_list(ctx, py_def.args, self_name, pdt_arg_types)
    303     return_type = None
    304     if getattr(py_def, 'returns', None) is not None:

[/usr/local/lib/python3.7/dist-packages/torch/jit/frontend.py](https://localhost:8080/#) in build_param_list(ctx, py_args, self_name, pdt_arg_types)
    324         expr = py_args.kwarg
    325         ctx_range = ctx.make_range(expr.lineno, expr.col_offset - 1, expr.col_offset + len(expr.arg))
--> 326         raise NotSupportedError(ctx_range, _vararg_kwarg_err)
    327     if py_args.vararg is not None:
    328         expr = py_args.vararg

NotSupportedError: Compiled functions can't take variable number of arguments or use keyword-only arguments with defaults:
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/parallel/data_parallel.py", line 147
    def forward(self, *inputs, **kwargs):
                                ~~~~~~~ <--- HERE
        with torch.autograd.profiler.record_function("DataParallel.forward"):
            if not self.device_ids:
`

### Versions

Collecting environment information...
PyTorch version: 1.10.0+cu111
Is debug build: False
CUDA used to build PyTorch: 11.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
CMake version: version 3.12.0
Libc version: glibc-2.26

Python version: 3.7.13 (default, Mar 16 2022, 17:37:17)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
Is CUDA available: False
CUDA runtime version: 11.1.105
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.0.5
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.21.6
[pip3] torch==1.10.0+cu111
[pip3] torchaudio==0.10.0+cu111
[pip3] torchsummary==1.5.1
[pip3] torchtext==0.11.0
[pip3] torchvision==0.11.1+cu111
[conda] Could not collect

Any help is appreciated.

Output:
torch.jit.script create a ScriptFunction(a Function with Graph) by parsing the python source code from module.forward().
If your module contains some grammar cannot support by the python parser, it will failed. Especially for the object not contains a static type.
Using torch.jit.trace is able to avoid such problems. It creates Graph in the op call process (c++ way). It will never failed, but cannot handle if-else branch cases. If you have branches, you should trace it every iteration which leading to 2 forward 1 backward in each training process. With no-brach model, you can reuse the traced ScriptFunction.


Instruction: Cannot import name 'functional_datapipe' from 'torch.utils.data'

Input:
When I am running datasets_utils.py from '/usr/local/lib/python3.7/dist-packages/torchtext/data/datasets_utils.py' in Google Colab, the following error occurs even with the most updated versions of Python packages:
ImportError: cannot import name 'functional_datapipe' from 'torch.utils.data' (/usr/local/lib/python3.7/dist-packages/torch/utils/data/init.py)
Are there any solutions to solve such errors, as I could not find functional_datapipe even in the official torch.utils.data documentation? The following is excerpt from datasets._utils.py in the Google Colab environment
import functools
import inspect
import os
import io
import torch
from torchtext.utils import (
    validate_file,
    download_from_url,
    extract_archive,
)
from torch.utils.data import functional_datapipe, IterDataPipe
from torch.utils.data.datapipes.utils.common import StreamWrapper
import codecs

Output: It might be available only on torchdata.datapipes


Instruction: What does Union from typing module in Python do?

Input:
I was looking the implementation of ResNet deep learning architecture in PyTorch from git-hub.
At line 167, inside the initializer of another class definition which defines ResNet and is also named ResNet, I saw the code below:
block: Type[Union[BasicBlock, Bottleneck]],

BasicBlock and Bottleneck are two classes defined before line 167. When I looked up for what Type and Union do, as far as I understood it says that block could be either of BasicBlock or Bottleneck. block itself was passed to a function named _make_layer which is defined at line 223 and is a method of ResNet class and I wonder how _make_layer know how the passed argument is either BasicBlock or Bottleneck?
When someone is making an instance of ResNet class, should they pass an object of BasicBlock or Bottleneck? and is this how _make_layer knows what it got as its argument? Then why do we need to use Union?

Output:
To reinforce the previous answer.  Python couldn't care less about types.  They are ignored completely.  Their sole purpose is for linters.
IDE's like PyCharm also have linters built into them.  PyCharm has caught numerous bugs for me:  "you said that function was supposed to take a two strings, but you're passing it an integer and a string".  Python itself just doesn't care.


Instruction: Troubles while compiling C++ program with PyTorch, HElib and OpenCV

Input:
I'm trying to compile my C++ program that uses the libraries HElib, OpenCV and PyTorch. I'm on Ubuntu 20.04. The entire code is:
#include <iostream>
#include <fstream>

#include <string>
#include <vector>
#include <cstdint>
#include <memory>
#include <stdio.h>
#include <opencv2/core.hpp>
#include <opencv2/imgcodecs.hpp>
#include <opencv2/highgui.hpp>
#include <helib/helib.h>
#include <torch/torch.h>
#include "include/mnist/mnist_reader.hpp"

using namespace cv;
using namespace torch;
using namespace std;
using namespace mnist;
using namespace helib;

int main(int argc, char* argv[]) {
  Tensor tensor = torch::rand({2, 3});
  cout << tensor << endl;

  Mat imageMat = Mat(image, true).reshape(0, 1).t();
  return 0;
}

(where image is a 28x28 matrix).
I'm compiling it with the command (I know I should be using cmake but I'm new to C++ and for now I'd like to learn how to link libraries properly just from the command line):
g++ -g -O2 -std=c++17 -pthread -march=native prova.cpp -lopencv_core -lopencv_highgui -lopencv_imgcodecs -o prova -I/home/lulu/helib_install/helib_pack/include -I/usr/include/opencv4 -I/home/lulu/libtorch/include -I/home/lulu/libtorch/include/torch/csrc/api/include -I/home/lulu/libtorch/include/torch -L/home/lulu/helib_install/helib_pack/lib -L/usr/include/opencv4 -L/home/lulu/libtorch/lib -lhelib -lntl -lgmp -lm -ltorch -ltorch_cpu -lc10 -D_GLIBCXX_USE_CXX11_ABI=0

The error I get is the following:
/usr/bin/ld: /tmp/cc3mP2mc.o: in function `cv::Mat::Mat(int, int, int, void*, unsigned long)':
/usr/include/opencv4/opencv2/core/mat.inl.hpp:548: undefined reference to `cv::error(int, std::string const&, char const*, char const*, int)'
collect2: error: ld returned 1 exit status

Deleting the flag -D_GLIBCXX_USE_CXX11_ABI=0 doesn't help, I tried.
I also tried setting the variable LD_LIBRARY_PATH to /home/lulu/libtorch/lib, but neither that helps.
I think I'm linking all the libraries I need, what am I missing?
Thanks in advance for the help.

Output:
I found the answer, but I can't really explain with my little experience what I've done, I'll just illustrate the passages.
I've re-downloaded PyTorch from its website, selecting the libtorch-cxx11-abi-shared-with-deps version (the one compiled with -D_GLIBCXX_USE_CXX11_ABI=1).
Then I had to add to the compilation command the flag -Wl,-rpath,/path/to/pytorch/lib, because for some reason the compiler didn't find libc10 and libtorch_cpu, so the final command was:
g++ -g -O2 -std=c++17 \
-pthread \
-march=native \
-I/home/lulu/helib_install/helib_pack/include \
-I/usr/include/opencv4 \
-I/home/lulu/libtorch/include \
-I/home/lulu/libtorch/include/torch/csrc/api/include \
-/home/lulu/libtorch/include/torch \
-L/home/lulu/helib_install/helib_pack/lib \
-L/usr/include/opencv4 \
-L/home/lulu/libtorch/lib \
-Wl,-rpath,/home/lulu/libtorch/lib \
prova.cpp \
-lopencv_core -lopencv_highgui -lopencv_imgcodecs \
-lhelib -lntl -lgmp -lm \
-ltorch -ltorch_cpu -lc10 \
-o prova


Instruction: Pytorch: Having trouble understanding the inline replacement happening

Input:
This seems to be a common error people get, but i can't really understand the real cause.
I am having trouble figuring out where the inline replacement is happening.
My forward function:
def forward(self, input, hidden=None):
    if hidden is None :
        hidden = self.init_hidden(input.size(0))
    out, hidden = self.lstm(input, hidden)
    out = self.linear(out)
    return out, hidden

The training loop
def training(dataloader, iterations,  device):
    torch.autograd.set_detect_anomaly(True)
    model = NModel(662, 322, 2, 1)
    hidden = None
    model.train()
    loss_fn = F.MSELoss()
    optimizer = torch.optim.SGD(model.parameters(), lr=0.001, momentum=0.9)
    running_loss =  []
    last_loss = 0
    for i, (feature, label) in tqdm(enumerate(dataloader)):
        optimizer.zero_grad()
        outputs, hidden = model(feature, hidden)
        loss =  loss_fn(outputs, label)
        print("loss item" , loss.item())
        running_loss.append(loss.item())
        loss.backward(retain_graph=True)
        optimizer.step()
        if i%1000 == 0:
            last_loss = len(running_loss) /1000
    return last_loss

The error's stack trace
Traceback (most recent call last):
  File "main.py", line 18, in <module>
    main()
  File "main.py", line 14, in main
    training(dataloader=training_loader, iterations=3, device=0)
  File "/home//gitclones/feature-extraction/training.py", line 30, in training
    loss.backward(retain_graph=True)
  File "/home/miniconda3/envs/pytorch-openpose/lib/python3.7/site-packages/torch/_tensor.py", line 307, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/home//miniconda3/envs/pytorch-openpose/lib/python3.7/site-packages/torch/autograd/__init__.py", line 156, in backward
    allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [322, 1288]], which is output 0 of AsStridedBackward0, is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

when i remove optimizer.step() the code runs but i think there is no backpropagation happening.
[EDIT]
Strange how now it works when i don't pass the hidden state as input in forward pass
def forward(self, input, hidden=None):
    if hidden is None :
        hidden = self.init_hidden(input.size(0))
    out, hidden = self.lstm(input)
    out = self.linear(out)
    return out, hidden

Output: Adding hidden = tuple([each.data for each in hidden]) after your optimizer.step() fix the error, but zeros the gradient on the hidden value. You can achieve the same effect with hidden = tuple([each.detach() for each in hidden])


Instruction: Terminate called after throwing an instance of 'std::bad_alloc' from importing torch_geometric

Input:
I am writing in python and getting the error:
"terminate called after throwing an instance of 'std::bad_alloc'.
what():  std::bad_alloc.
Aborted (core dumped)"
After lots of debugging, I found out the source of the issue is:
import torch_geometric

I even created a file with just this line of code, and I still get the error.
I am running in a conda environment (4.10.3)
I made sure that I installed torch_geometric while I was in the conda environment. I tried deleting and reinstalling, but this did not work.
I also tried deleting and reinstalling torch/cuda.
I googled the error, but only seemed to come up with issues in data allocation, but I'm not sure how this would be an issue, since I am just importing torch_geometric.
Any ideas?

Output:
This problem is because of mismatched versions of pytorch.
The current pytorch being used is 1.11.0, but when scatter and sparse were installed installed scatter and sparse, 1.10.1 were used:

pip install torch-scatter -f https://data.pyg.org/whl/torch-1.10.1+cu113.html.
pip install torch-sparse -f https://data.pyg.org/whl/torch-1.10.1+cu113.html

So,torch-1.10.1 was used to install scatter and sparse, but torch-1.11.0 was the true version.
Simply doing:

pip uninstall torch-scatter
pip uninstall torch-sparse
pip install torch-scatter -f https://data.pyg.org/whl/torch-1.11.0+cu113.html.
pip install torch-sparse -f https://data.pyg.org/whl/torch-1.11.0+cu113.html

Resolves the issue.


Instruction: Pytorch Temporal Fusion Transformer - TimeSeriesDataSet TypeError: '<' not supported between instances of 'int' and 'str'

Input:
I'm following Temporal-Fusion-Transformer (TFT) tutorial in the PytorchForecasting (https://pytorch-forecasting.readthedocs.io/en/stable/tutorials/stallion.html#Demand-forecasting-with-the-Temporal-Fusion-Transformer) to train TFT model with custom dataset to predict "booking" value based on several static/time-varying features for each Sold TO Party in each Region and Sub-Region.
When converting the dataframe into a PyTorch Forecasting TimeSeriesDataSet, I encountered the error: "TypeError: '<' not supported between instances of 'int' and 'str'." Does anyone know what's potential issue for this error?
The code is as follows:
max_prediction_length = 2
max_encoder_length = 8
training_cutoff = ctmdata["time_idx"].max() - max_prediction_length
target_value = "Booking"
key_idx = ["Region","Sub-Region","Sold TO Party Code"]

training = TimeSeriesDataSet(
    ctmdata[lambda x: x.time_idx <= training_cutoff],
    time_idx="time_idx",
    target=target_value,
    group_ids=key_idx,
    min_encoder_length=max_encoder_length // 2,  # keep encoder length long (as it is in the validation set)
    max_encoder_length=max_encoder_length,
    min_prediction_length=1,
    max_prediction_length=max_prediction_length,
    static_categoricals=["Region","Sub-Region","Sold TO Party Code","Customer Type","Customer Segment L1","Customer Segment L2"],
    static_reals=[],
    time_varying_known_categoricals=["Supply Chain Customer","Quarter"],
    variable_groups={},  # group of categorical variables can be treated as one variable
    time_varying_known_reals=["time_idx"],
    time_varying_unknown_categoricals=["DW Customer"],
    time_varying_unknown_reals=[
        "Booking","yeojohnson_Booking","avg_booking_bySubRegion","avg_booking_byCTMSegL1",
        "Billing","yeojohnson_Billing","avg_billing_bySubRegion","avg_billing_byCTMSegL1",
        "MGP","MGP%","BB Ratio"
    ],
    target_normalizer=GroupNormalizer(
        groups=key_idx, transformation="softplus"
    )  # use softplus and normalize by group
)

The error message shows:
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-119-9cc2c7eb9aa5> in <module>()
     38     ],
     39     target_normalizer=GroupNormalizer(
---> 40         groups=key_idx, transformation="softplus"
     41     )  # use softplus and normalize by group
     42 )

4 frames
/usr/local/lib/python3.7/dist-packages/pytorch_forecasting/data/timeseries.py in __init__(self, data, time_idx, target, group_ids, weight, max_encoder_length, min_encoder_length, min_prediction_idx, min_prediction_length, max_prediction_length, static_categoricals, static_reals, time_varying_known_categoricals, time_varying_known_reals, time_varying_unknown_categoricals, time_varying_unknown_reals, variable_groups, constant_fill_strategy, allow_missing_timesteps, lags, add_relative_time_idx, add_target_scales, add_encoder_length, target_normalizer, categorical_encoders, scalers, randomize_length, predict_mode)
    432
    433         # preprocess data
--> 434         data = self._preprocess_data(data)
    435         for target in self.target_names:
    436             assert target not in self.scalers, "Target normalizer is separate and not in scalers."

/usr/local/lib/python3.7/dist-packages/pytorch_forecasting/data/timeseries.py in _preprocess_data(self, data)
    651             # use existing encoder - but a copy of it not too loose current encodings
    652             encoder = deepcopy(self.categorical_encoders.get(group_name, NaNLabelEncoder()))
--> 653             self.categorical_encoders[group_name] = encoder.fit(data[name].to_numpy().reshape(-1), overwrite=False)
    654             data[group_name] = self.transform_values(name, data[name], inverse=False, group_id=True)
    655

/usr/local/lib/python3.7/dist-packages/pytorch_forecasting/data/encoders.py in fit(self, y, overwrite)
     88
     89         idx += offset
---> 90         for val in np.unique(y):
     91             if val not in self.classes_:
     92                 self.classes_[val] = idx

<__array_function__ internals> in unique(*args, **kwargs)

/usr/local/lib/python3.7/dist-packages/numpy/lib/arraysetops.py in unique(ar, return_index, return_inverse, return_counts, axis)
    270     ar = np.asanyarray(ar)
    271     if axis is None:
--> 272         ret = _unique1d(ar, return_index, return_inverse, return_counts)
    273         return _unpack_tuple(ret)
    274

/usr/local/lib/python3.7/dist-packages/numpy/lib/arraysetops.py in _unique1d(ar, return_index, return_inverse, return_counts)
    331         aux = ar[perm]
    332     else:
--> 333         ar.sort()
    334         aux = ar
    335     mask = np.empty(aux.shape, dtype=np.bool_)

TypeError: '<' not supported between instances of 'int' and 'str'

Thanks for help!

Output: You can check the values in the static_categoricals=["Region","Sub-Region","Sold TO Party Code","Customer Type","Customer Segment L1","Customer Segment L2"], maybe you are passing a value that have a str and this is trying to be compare with a int


Instruction: Torchvision transforms.ToTensor show different range result. Dose not scale array to [0,1] as document says

Input:
I am converting the numpy array to tensor with the following code：
self.transform_1 = transforms.Compose([transforms.ToTensor()])
source_parsing_np = cv2.imread(source_parsing_path, cv2.IMREAD_GRAYSCALE) #The range is integer in the range [0,14]
source_parsing_tensor = self.transform_1(source_parsing_tensor)

As the documentation says, the data will be scaled to  [0.0, 1.0]. But in my environment, two different results appeared at different times.
Specifically, in the previous training and testing code and the jupyter notebook code I'm testing now, the tensor results are still integer in the range [0,14]. wrong range result
When I use the same code in test phase again, the data truly be scaled to [0.0, 1.0], which is different from previous train phase. And another numpy array data is with same transform has not been changed(still integer in [0,24]). same transform different result
Because of the above problem I cannot reproduce my model test results. I would be very thankful for any information on this.

Output: I found the reason: the conversion occurs only when the numpy.ndarray has dtype = np.uint8, but my dytpe is np.long, sorry for my carelessness.


Instruction: Converting PyTorch Boolean target to regression target

Input:
Question
I have code that is based on Part 2, Chapter 11 of Deep Learning with PyTorch, by Luca Pietro Giovanni Antiga, Thomas Viehmann, and Eli Stevens.  It's working just fine.  It predicts the value of a Boolean variable.  I want to convert this so that it predicts the value of a real number variable that happens to be always between 0 and 34.
There are two parts that I don't know how to convert.  First, this part:
    pos_t = torch.tensor([
            not candidateInfo_tup.isNodule_bool,
            candidateInfo_tup.isNodule_bool
        ],
        dtype=torch.long,
    )

(Why are two values passed in here when one is completely determined by the other?)
and then this part:
    self.head_linear = nn.Linear(1152, 2)
    self.head_softmax = nn.Softmax(dim=1)

How do I do this?
Guess
I don't want people to think I haven't thought about this at all, so here is my guess:
First part:
    age_t = torch.tensor(candidateInfo_tup.age_int, dtype=torch.double)

Second part:
    self.head_linear = nn.Linear(299520, 1)
    self.head_relu = nn.ReLU()

I'm also guessing that I need to change this:
    loss_func = nn.CrossEntropyLoss(reduction='none')

to something like this:
    loss_func = nn.L1Loss()

My guesses are based on this article by Christian Versloot.

Output:
The example from the book is working but it has some redundant elements which confuse you.
Normally output size of 1 is enough for a binary classification problem. To bring it  to 0 or 1, one may use sigmoid and then rounding, like in the example here: PyTorch Binary Classification - same network structure, 'simpler' data, but worse performance?
Or just put after single output neuron this:
y_pred_binary = torch.round(torch.sigmoid(y_pred))

The book example uses output size of 2 and then applies softmax to get to 0 and 1. This works, but such technique is typically used in multi-class classification.
For prediction of 0-34 variable:

if these are discrete variables - it is called "multiclass classification" as Ken indicated. Use size 35 output and softmax in this case. Search for "pytorch multiclass classification" for examples.
if this is a regression - than your changes seem in the right direction, except 'Second part'. instead of RELU - clip output at both ends at [0, 34]. Also 299520 - is too much for the previous layer. Use whatever input size there was before. Search for "pytorch regression" for examples.


Instruction: Which method should I use to sample from a normal distribution?

Input:
I'm trying to sample batch_size points from an N-dim standard Gaussian distribution. But I noticed there are two similar functions I can use, and I want to know which one is correct or both.
Assume I want to sample 8 points from a 2-dim standard Gaussian.

torch.distributions.MultivariateNormal(torch.zeros(2), torch.eye(2)).sample([8]), it will return a tensor of size [8,2]
torch.randn(8,2)

They return similar outputs, but I'm wondering whether they're the same.

Output:
torch.randn gives you samples from a univariate standard normal distribution and reshapes them to the desired shape. So the mean of all the samples is 0 with unit variance.
x = torch.randn(1000000,2).numpy()
assert np.isclose(np.mean(x.flatten()), 0, atol=0.01)
plt.hist(x.flatten())

MultivariateNormal generates samples from a multivariate normal distribution. It can be parameterized by a mean vector and a covariance matrix.
x = torch.distributions.MultivariateNormal(
    torch.zeros(2), torch.eye(2)).sample([10000]).numpy()
assert np.isclose(np.mean(x.flatten()), 0, atol=0.01)
plt.hist(x.flatten())

The above usage is sort of a hack; we are creating two standard normals in two different dimensions and since the distributions (mean and variance) are the same we can consider them to be as well from a single distribution and hence the flattened array can be considered to be from a univariate standard normal.

The real purpose of MultivariateNormal:
x = torch.distributions.MultivariateNormal(
    torch.tensor([-2.0, 2.0]), torch.eye(2)).sample([10000]).numpy()
plt.hist(x[:, 0])
plt.hist(x[:, 1])


Instruction: How to fix "initial_lr not specified when resuming optimizer" error for scheduler?

Input:
In PyTorch I have configured SGD like this:
sgd_config = {
    'params' : net.parameters(),
    'lr' : 1e-7,
    'weight_decay' : 5e-4,
    'momentum' : 0.9
}
optimizer = SGD(**sgd_config)

My requirements are:

Total epochs are 100
Every 30 epochs learning rate is decreased by a factor of 10
Decreasing learning rate will stop at 60 epochs

So for 100 epochs I will get two times a decrease of 0.1 of my learning rate.
I read about learning rate scheduler, available in torch.optim.lr_scheduler so I decided to try using that instead of manually adjusting the learning rate:
scheduler = lr_scheduler.StepLR(optimizer, step_size=30, last_epoch=60, gamma=0.1)

However I am getting
Traceback (most recent call last):
  File "D:\Projects\network\network_full.py", line 370, in <module>
    scheduler = lr_scheduler.StepLR(optimizer, step_size=30, last_epoch=90, gamma=0.1)
  File "D:\env\test\lib\site-packages\torch\optim\lr_scheduler.py", line 367, in __init__
    super(StepLR, self).__init__(optimizer, last_epoch, verbose)
  File "D:\env\test\lib\site-packages\torch\optim\lr_scheduler.py", line 39, in __init__
    raise KeyError("param 'initial_lr' is not specified "
KeyError: "param 'initial_lr' is not specified in param_groups[0] when resuming an optimizer"

I read a post here and I still don't get how I would use the scheduler for my scenario. Maybe I am just not understanding the definition of last_epoch given that the documentation is very brief on this parameter:

last_epoch (int) – The index of last epoch. Default: -1.

Since the argument is made available to the user and there is no explicit prohibition on using a scheduler for less epochs than the optimizer itself, I am starting to think it's a bug.

Output:
You have misunderstood the last_epoch argument and you are not using the correct learning rate scheduler for your requirements.
This should work:
optim.lr_scheduler.MultiStepLR(optimizer, [0, 30, 60], gamma=0.1, last_epoch=args.current_epoch - 1)

The last_epoch argument makes sure to use the correct LR when resuming training. It defaults to -1, so the epoch before epoch 0.


Instruction: UserWarning: Using a target size (torch.Size([1])) that is different to the input size (torch.Size([1, 1]))

Input:
I have this code:
    actual_loes_score_g = actual_loes_score_t.to(self.device, non_blocking=True)

    predicted_loes_score_g = self.model(input_g)

    loss_func = nn.L1Loss()
    loss_g = loss_func(
        predicted_loes_score_g,
        actual_loes_score_g,
    )

where predicted_loes_score_g is tensor([[-24.9374]], grad_fn=<AddmmBackward0>) and actual_loes_score_g is tensor([20.], dtype=torch.float64).  (I am using a batch size of 1 for debugging purposes.)
I am getting this warning:
torch/nn/modules/loss.py:96: UserWarning: Using a target size (torch.Size([1])) that is
different to the input size (torch.Size([1, 1])). This will likely lead to incorrect
results due to broadcasting. Please ensure they have the same size.

How do I correctly ensure they have the same size?
I thought this might be the answer:
    predicted_loes_score = predicted_loes_score_g.detach()[0]
    loss_g = loss_func(
        predicted_loes_score,
        actual_loes_score_g,
    )

but then I get this error later:
torch/autograd/__init__.py", line 154, in backward
    Variable._execution_engine.run_backward(
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

Output:
predicted_loes_score_g = tensor([[-24.9374]], grad_fn=<AddmmBackward0>)

which is size [1,1]
actual_loes_score_g = tensor([20.], dtype=torch.float64)

which is size [1]
You need to either remove a dimension from your prediction or add a dimension to your target. I would recommend the latter because that extra dimension corresponds to your batch size. Try this:
actual_loes_score_g =  actual_loes_score_g.unsqueeze(1)


Instruction: Overlap two tensors of different size based on an offset in PyTorch

Input:
I have the following structure:
torch.Size([channels, width, height])

Let's say I have a tensor a
torch.Size([4, 512, 512])

And tensor b
torch.Size([4, 100, 100])

What I would like to do is to create a tensor c that is the result of "placing" tensor b on an arbitrary (width, height) coordinate offset of tensor a. For example, let's say I would like to place tensor b on (300,100) of tensor a
So for tensor a's width between the 300-400 position, the values on tensor a should be replaced by the 100 values of tensor b width.
For tensor a's height between the 100-200 position, the values of tensor a should be replaced by the 100 values of tensor b height.
I would also like to choose for which channels I want to do this substitution and for which channels I would keep tensor a's value
(PS: The image is just an easy to illustrate example, but I would like to do it in a more generalisable way, so I'm no interested in converting to PIL, using PIL.paste and back to tensor but I would like to do all operations directly with)

Output:
This function would be fragile without a bunch of pre-conditions to catch for size mismatches, but I think this is basically what you're describing:
def place(a: torch.Tensor, b: torch.Tensor,
          height: int, width: int,
          channels: list[int]) -> torch.Tensor:
    """create a tensor ``c`` that is the result of
    "placing" tensor ``b`` on an arbitrary (height, width)
    coordinate offset of tensor ``a``,
    for only the specified ``channels``
    """
    channels_b, height_b, width_b  = b.size()
    c = a.clone().detach()
    for channel in channels:
        c[channel,
          height:height + height_b,
          width:width + width_b] = b[channel]
    return c


Instruction: torch_optimizer has no attribute 'SGD'

Input:
I am trying to import torch_optimizer on Google Colab. I have successfully !pip installed torch_optimizer and then imported it. However, every attribute I call with torch_optimizer gives an attribute error:
AttributeError: module 'torch_optimizer' has no attribute 'SGD'
This holds true for SGD, Adam, etc.
Here is a photo of my code. Thanks!
Pytorch Optimizer

Output: I have checked the documentation of pytorch-optimizer. The vanilla SGD is not there. There are small modifications of SGD as AccSGD, SGDW, SGDP etc. You can use the simple pytorch optimizer torch.optim.SGD. Check this visualization script where they are comparing the baseline SGD to other methods implemented by this library. Same thing holds for Adam as well. Pretty clear from the script.


Instruction: pytorch layer input, output shape calculation

Input: Can anyone help me to understand when I use conv1d and then a linear layer, What will be the inputs of the linear layer? How to calculate how many input features should I have to pass in pytorch

Output:
In Pytorch, Linear layers operate using only the last dimension of the input tensor: [*features_in] -> [*,features_out].
However, Conv1D layers consider the last 2 dimensions of the input tensor: [batches,channels_in, length_in] -> [batches,channels_out, length_out].
Therefore, if no pre-processing is used, Linear layers will only work with the signals defined for every channel, i.e., [batches,channels_in,features_in] -> [batches,channels_in,features_out]. This behavior is rarely desired, so people usually flatten tensors before passing them to a Linear layer. For example, it's common to use Linear(x.view(n_batches,-1)).
The behavior you need depends on the details of your application. Good luck,
Sources:
https://pytorch.org/docs/stable/generated/torch.nn.Conv1d.html (Conv1d)
https://pytorch.org/docs/stable/generated/torch.nn.Linear.html (Linear)


Instruction: RuntimeError: output with shape [1] doesn't match the broadcast shape [10]

Input:
Hi I trying to make RBM Model code using pytorch module but got a issue in visible layer to hidden layer. Here is the problem part code.
        h_bias = (self.h_bias.clone()).expand(10)
        v = v.clone().expand(10)

        p_h = F.sigmoid(
            F.linear(v, self.W, bias=h_bias)
        )

        sample_h = self.sample_from_p(p_h)
        return p_h, sample_h

and each parameters size is here.
h_bias           v                self.W
torch.Size([10]) torch.Size([10]) torch.Size([1, 10])
1                1                2

Traceback (most recent call last):
  File "/Users/bahk_insung/Documents/Github/ecg-dbn/model.py", line 68, in <module>
    v, v1 = rbm(sample_data)
  File "/Users/bahk_insung/miniforge3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/Users/bahk_insung/Documents/Github/ecg-dbn/RBM.py", line 54, in forward
    pre_h1, h1 = self.v_to_h(v)
  File "/Users/bahk_insung/Documents/Github/ecg-dbn/RBM.py", line 36, in v_to_h
    F.linear(v, self.W, bias=h_bias)
  File "/Users/bahk_insung/miniforge3/lib/python3.9/site-packages/torch/nn/functional.py", line 1849, in linear
    return torch._C._nn.linear(input, weight, bias)
RuntimeError: output with shape [1] doesn't match the broadcast shape [10]

I think dimension and size are not matched thats reason why happened. But I can't get any solutions. Please help me guys. Thank you.

Output:
If you look at the pytorch functional.linear documentation it shows the weight parameter can be either 1D or 2D: "Weight: (out_features, in_features) or (in_features)". Since your weight is 2D ([1, 10]) it indicates that you are trying to create an output of size "1" with an input size of "10". The linear transform does not know how to change your inputs of size 10 into an output of size 1. If your weight will always be [1, N] then you can use squeeze to change it to 1D like so:
F.linear(v, self.W.squeeze(), bias=h_bias)

This would create an output of size 10.


Instruction: How to install fastai on Mac m1

Input:
I am trying to install fastai (version 1.0.61) on my new Mac m1.
I first tried:
pip install fastai==1.0.61

This gave me an error that I didn't have cmake, so I installed cmake successfully with brew install cmake.
Then, rerunning the fastai install command, I get this error:
Collecting fastai==1.0.61
  Using cached fastai-1.0.61-py3-none-any.whl (239 kB)
Collecting fastprogress>=0.2.1
  Using cached fastprogress-1.0.2-py3-none-any.whl (12 kB)
Collecting numexpr
  Using cached numexpr-2.8.1-cp310-cp310-macosx_11_0_arm64.whl
Collecting scipy
  Using cached scipy-1.8.0-cp310-cp310-macosx_12_0_arm64.whl (28.7 MB)
Collecting pyyaml
  Using cached PyYAML-6.0-cp310-cp310-macosx_11_0_arm64.whl (173 kB)
Collecting bottleneck
  Using cached Bottleneck-1.3.4-cp310-cp310-macosx_11_0_arm64.whl
Collecting matplotlib
  Using cached matplotlib-3.5.1-cp310-cp310-macosx_11_0_arm64.whl (7.2 MB)
Collecting pynvx>=1.0.0
  Using cached pynvx-1.0.0.tar.gz (150 kB)
  Preparing metadata (setup.py) ... done
Collecting beautifulsoup4
  Using cached beautifulsoup4-4.11.1-py3-none-any.whl (128 kB)
Collecting nvidia-ml-py3
  Using cached nvidia_ml_py3-7.352.0-py3-none-any.whl
Requirement already satisfied: requests in /Users/connoreaton/miniforge3/envs/spaff_nlp/lib/python3.10/site-packages (from fastai==1.0.61) (2.27.1)
Requirement already satisfied: Pillow in /Users/connoreaton/miniforge3/envs/spaff_nlp/lib/python3.10/site-packages (from fastai==1.0.61) (9.1.0)
Requirement already satisfied: pandas in /Users/connoreaton/miniforge3/envs/spaff_nlp/lib/python3.10/site-packages (from fastai==1.0.61) (1.4.2)
Collecting packaging
  Using cached packaging-21.3-py3-none-any.whl (40 kB)
Requirement already satisfied: torch>=1.0.0 in /Users/connoreaton/miniforge3/envs/spaff_nlp/lib/python3.10/site-packages (from fastai==1.0.61) (1.11.0)
Requirement already satisfied: torchvision in /Users/connoreaton/miniforge3/envs/spaff_nlp/lib/python3.10/site-packages (from fastai==1.0.61) (0.12.0)
Requirement already satisfied: numpy>=1.15 in /Users/connoreaton/miniforge3/envs/spaff_nlp/lib/python3.10/site-packages (from fastai==1.0.61) (1.22.3)
Requirement already satisfied: typing-extensions in /Users/connoreaton/miniforge3/envs/spaff_nlp/lib/python3.10/site-packages (from torch>=1.0.0->fastai==1.0.61) (4.2.0)
Collecting soupsieve>1.2
  Using cached soupsieve-2.3.2.post1-py3-none-any.whl (37 kB)
Collecting pyparsing>=2.2.1
  Using cached pyparsing-3.0.8-py3-none-any.whl (98 kB)
Collecting kiwisolver>=1.0.1
  Using cached kiwisolver-1.4.2-cp310-cp310-macosx_11_0_arm64.whl (63 kB)
Requirement already satisfied: python-dateutil>=2.7 in /Users/connoreaton/miniforge3/envs/spaff_nlp/lib/python3.10/site-packages (from matplotlib->fastai==1.0.61) (2.8.2)
Collecting cycler>=0.10
  Using cached cycler-0.11.0-py3-none-any.whl (6.4 kB)
Collecting fonttools>=4.22.0
  Using cached fonttools-4.33.3-py3-none-any.whl (930 kB)
Requirement already satisfied: pytz>=2020.1 in /Users/connoreaton/miniforge3/envs/spaff_nlp/lib/python3.10/site-packages (from pandas->fastai==1.0.61) (2022.1)
Requirement already satisfied: certifi>=2017.4.17 in /Users/connoreaton/miniforge3/envs/spaff_nlp/lib/python3.10/site-packages (from requests->fastai==1.0.61) (2021.10.8)
Requirement already satisfied: idna<4,>=2.5 in /Users/connoreaton/miniforge3/envs/spaff_nlp/lib/python3.10/site-packages (from requests->fastai==1.0.61) (3.3)
Requirement already satisfied: charset-normalizer~=2.0.0 in /Users/connoreaton/miniforge3/envs/spaff_nlp/lib/python3.10/site-packages (from requests->fastai==1.0.61) (2.0.12)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /Users/connoreaton/miniforge3/envs/spaff_nlp/lib/python3.10/site-packages (from requests->fastai==1.0.61) (1.26.9)
Requirement already satisfied: six>=1.5 in /Users/connoreaton/miniforge3/envs/spaff_nlp/lib/python3.10/site-packages (from python-dateutil>=2.7->matplotlib->fastai==1.0.61) (1.16.0)
Building wheels for collected packages: pynvx
  Building wheel for pynvx (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [78 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.macosx-11.0-arm64-cpython-310
      creating build/lib.macosx-11.0-arm64-cpython-310/pynvx
      copying pynvx/pynvml.py -> build/lib.macosx-11.0-arm64-cpython-310/pynvx
      copying pynvx/__init__.py -> build/lib.macosx-11.0-arm64-cpython-310/pynvx
      running build_ext
      -- The C compiler identification is AppleClang 13.1.6.13160021
      -- The CXX compiler identification is AppleClang 13.1.6.13160021
      -- Detecting C compiler ABI info
      -- Detecting C compiler ABI info - done
      -- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc - skipped
      -- Detecting C compile features
      -- Detecting C compile features - done
      -- Detecting CXX compiler ABI info
      -- Detecting CXX compiler ABI info - done
      -- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ - skipped
      -- Detecting CXX compile features
      -- Detecting CXX compile features - done
      -- Found PythonInterp: /Users/connoreaton/miniforge3/envs/spaff_nlp/bin/python3.10 (found version "3.10.4")
      -- Found PythonLibs: /Users/connoreaton/miniforge3/envs/spaff_nlp/lib/libpython3.10.dylib
      -- Performing Test HAS_CPP14_FLAG
      -- Performing Test HAS_CPP14_FLAG - Success
      -- pybind11 v2.3.dev0
      -- Performing Test HAS_FLTO
      -- Performing Test HAS_FLTO - Success
      -- LTO enabled
      CMake Error at /opt/homebrew/Cellar/cmake/3.23.1/share/cmake/Modules/FindCUDA.cmake:859 (message):
        Specify CUDA_TOOLKIT_ROOT_DIR
      Call Stack (most recent call first):
        CMakeLists.txt:8 (find_package)

      -- Configuring incomplete, errors occurred!
      See also "/private/var/folders/fn/lgvtrxd90cl3nqryfdfcsld80000gn/T/pip-install-81yvkyvs/pynvx_19ff0317a3854976bb7ea683521d385f/build/temp.macosx-11.0-arm64-cpython-310/CMakeFiles/CMakeOutput.log".
      See also "/private/var/folders/fn/lgvtrxd90cl3nqryfdfcsld80000gn/T/pip-install-81yvkyvs/pynvx_19ff0317a3854976bb7ea683521d385f/build/temp.macosx-11.0-arm64-cpython-310/CMakeFiles/CMakeError.log".
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/private/var/folders/fn/lgvtrxd90cl3nqryfdfcsld80000gn/T/pip-install-81yvkyvs/pynvx_19ff0317a3854976bb7ea683521d385f/setup.py", line 68, in <module>
          setup(
        File "/Users/connoreaton/miniforge3/envs/spaff_nlp/lib/python3.10/site-packages/setuptools/__init__.py", line 87, in setup
          return distutils.core.setup(**attrs)
        File "/Users/connoreaton/miniforge3/envs/spaff_nlp/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 148, in setup
          return run_commands(dist)
        File "/Users/connoreaton/miniforge3/envs/spaff_nlp/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 163, in run_commands
          dist.run_commands()
        File "/Users/connoreaton/miniforge3/envs/spaff_nlp/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 967, in run_commands
          self.run_command(cmd)
        File "/Users/connoreaton/miniforge3/envs/spaff_nlp/lib/python3.10/site-packages/setuptools/dist.py", line 1214, in run_command
          super().run_command(command)
        File "/Users/connoreaton/miniforge3/envs/spaff_nlp/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
          cmd_obj.run()
        File "/Users/connoreaton/miniforge3/envs/spaff_nlp/lib/python3.10/site-packages/wheel/bdist_wheel.py", line 299, in run
          self.run_command('build')
        File "/Users/connoreaton/miniforge3/envs/spaff_nlp/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/Users/connoreaton/miniforge3/envs/spaff_nlp/lib/python3.10/site-packages/setuptools/dist.py", line 1214, in run_command
          super().run_command(command)
        File "/Users/connoreaton/miniforge3/envs/spaff_nlp/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
          cmd_obj.run()
        File "/Users/connoreaton/miniforge3/envs/spaff_nlp/lib/python3.10/site-packages/setuptools/_distutils/command/build.py", line 136, in run
          self.run_command(cmd_name)
        File "/Users/connoreaton/miniforge3/envs/spaff_nlp/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/Users/connoreaton/miniforge3/envs/spaff_nlp/lib/python3.10/site-packages/setuptools/dist.py", line 1214, in run_command
          super().run_command(command)
        File "/Users/connoreaton/miniforge3/envs/spaff_nlp/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
          cmd_obj.run()
        File "/private/var/folders/fn/lgvtrxd90cl3nqryfdfcsld80000gn/T/pip-install-81yvkyvs/pynvx_19ff0317a3854976bb7ea683521d385f/setup.py", line 32, in run
          self.build_extension(ext)
        File "/private/var/folders/fn/lgvtrxd90cl3nqryfdfcsld80000gn/T/pip-install-81yvkyvs/pynvx_19ff0317a3854976bb7ea683521d385f/setup.py", line 56, in build_extension
          subprocess.check_call(['cmake', ext.sourcedir] + cmake_args, cwd=self.build_temp, env=env)
        File "/Users/connoreaton/miniforge3/envs/spaff_nlp/lib/python3.10/subprocess.py", line 369, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['cmake', '/private/var/folders/fn/lgvtrxd90cl3nqryfdfcsld80000gn/T/pip-install-81yvkyvs/pynvx_19ff0317a3854976bb7ea683521d385f', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/private/var/folders/fn/lgvtrxd90cl3nqryfdfcsld80000gn/T/pip-install-81yvkyvs/pynvx_19ff0317a3854976bb7ea683521d385f/build/lib.macosx-11.0-arm64-cpython-310', '-DPYTHON_EXECUTABLE=/Users/connoreaton/miniforge3/envs/spaff_nlp/bin/python3.10', '-DCMAKE_BUILD_TYPE=Release']' returned non-zero exit status 1.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for pynvx
  Running setup.py clean for pynvx
Failed to build pynvx
Installing collected packages: pynvx, nvidia-ml-py3, soupsieve, scipy, pyyaml, pyparsing, kiwisolver, fonttools, fastprogress, cycler, bottleneck, packaging, beautifulsoup4, numexpr, matplotlib, fastai
  Running setup.py install for pynvx ... error
  error: subprocess-exited-with-error

  × Running setup.py install for pynvx did not run successfully.
  │ exit code: 1
  ╰─> [82 lines of output]
      running install
      /Users/connoreaton/miniforge3/envs/spaff_nlp/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        warnings.warn(
      running build
      running build_py
      creating build
      creating build/lib.macosx-11.0-arm64-cpython-310
      creating build/lib.macosx-11.0-arm64-cpython-310/pynvx
      copying pynvx/pynvml.py -> build/lib.macosx-11.0-arm64-cpython-310/pynvx
      copying pynvx/__init__.py -> build/lib.macosx-11.0-arm64-cpython-310/pynvx
      running build_ext
      -- The C compiler identification is AppleClang 13.1.6.13160021
      -- The CXX compiler identification is AppleClang 13.1.6.13160021
      -- Detecting C compiler ABI info
      -- Detecting C compiler ABI info - done
      -- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc - skipped
      -- Detecting C compile features
      -- Detecting C compile features - done
      -- Detecting CXX compiler ABI info
      -- Detecting CXX compiler ABI info - done
      -- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ - skipped
      -- Detecting CXX compile features
      -- Detecting CXX compile features - done
      -- Found PythonInterp: /Users/connoreaton/miniforge3/envs/spaff_nlp/bin/python3.10 (found version "3.10.4")
      -- Found PythonLibs: /Users/connoreaton/miniforge3/envs/spaff_nlp/lib/libpython3.10.dylib
      -- Performing Test HAS_CPP14_FLAG
      -- Performing Test HAS_CPP14_FLAG - Success
      -- pybind11 v2.3.dev0
      -- Performing Test HAS_FLTO
      -- Performing Test HAS_FLTO - Success
      -- LTO enabled
      CMake Error at /opt/homebrew/Cellar/cmake/3.23.1/share/cmake/Modules/FindCUDA.cmake:859 (message):
        Specify CUDA_TOOLKIT_ROOT_DIR
      Call Stack (most recent call first):
        CMakeLists.txt:8 (find_package)

      -- Configuring incomplete, errors occurred!
      See also "/private/var/folders/fn/lgvtrxd90cl3nqryfdfcsld80000gn/T/pip-install-81yvkyvs/pynvx_19ff0317a3854976bb7ea683521d385f/build/temp.macosx-11.0-arm64-cpython-310/CMakeFiles/CMakeOutput.log".
      See also "/private/var/folders/fn/lgvtrxd90cl3nqryfdfcsld80000gn/T/pip-install-81yvkyvs/pynvx_19ff0317a3854976bb7ea683521d385f/build/temp.macosx-11.0-arm64-cpython-310/CMakeFiles/CMakeError.log".
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/private/var/folders/fn/lgvtrxd90cl3nqryfdfcsld80000gn/T/pip-install-81yvkyvs/pynvx_19ff0317a3854976bb7ea683521d385f/setup.py", line 68, in <module>
          setup(
        File "/Users/connoreaton/miniforge3/envs/spaff_nlp/lib/python3.10/site-packages/setuptools/__init__.py", line 87, in setup
          return distutils.core.setup(**attrs)
        File "/Users/connoreaton/miniforge3/envs/spaff_nlp/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 148, in setup
          return run_commands(dist)
        File "/Users/connoreaton/miniforge3/envs/spaff_nlp/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 163, in run_commands
          dist.run_commands()
        File "/Users/connoreaton/miniforge3/envs/spaff_nlp/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 967, in run_commands
          self.run_command(cmd)
        File "/Users/connoreaton/miniforge3/envs/spaff_nlp/lib/python3.10/site-packages/setuptools/dist.py", line 1214, in run_command
          super().run_command(command)
        File "/Users/connoreaton/miniforge3/envs/spaff_nlp/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
          cmd_obj.run()
        File "/Users/connoreaton/miniforge3/envs/spaff_nlp/lib/python3.10/site-packages/setuptools/command/install.py", line 68, in run
          return orig.install.run(self)
        File "/Users/connoreaton/miniforge3/envs/spaff_nlp/lib/python3.10/site-packages/setuptools/_distutils/command/install.py", line 670, in run
          self.run_command('build')
        File "/Users/connoreaton/miniforge3/envs/spaff_nlp/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/Users/connoreaton/miniforge3/envs/spaff_nlp/lib/python3.10/site-packages/setuptools/dist.py", line 1214, in run_command
          super().run_command(command)
        File "/Users/connoreaton/miniforge3/envs/spaff_nlp/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
          cmd_obj.run()
        File "/Users/connoreaton/miniforge3/envs/spaff_nlp/lib/python3.10/site-packages/setuptools/_distutils/command/build.py", line 136, in run
          self.run_command(cmd_name)
        File "/Users/connoreaton/miniforge3/envs/spaff_nlp/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/Users/connoreaton/miniforge3/envs/spaff_nlp/lib/python3.10/site-packages/setuptools/dist.py", line 1214, in run_command
          super().run_command(command)
        File "/Users/connoreaton/miniforge3/envs/spaff_nlp/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
          cmd_obj.run()
        File "/private/var/folders/fn/lgvtrxd90cl3nqryfdfcsld80000gn/T/pip-install-81yvkyvs/pynvx_19ff0317a3854976bb7ea683521d385f/setup.py", line 32, in run
          self.build_extension(ext)
        File "/private/var/folders/fn/lgvtrxd90cl3nqryfdfcsld80000gn/T/pip-install-81yvkyvs/pynvx_19ff0317a3854976bb7ea683521d385f/setup.py", line 56, in build_extension
          subprocess.check_call(['cmake', ext.sourcedir] + cmake_args, cwd=self.build_temp, env=env)
        File "/Users/connoreaton/miniforge3/envs/spaff_nlp/lib/python3.10/subprocess.py", line 369, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['cmake', '/private/var/folders/fn/lgvtrxd90cl3nqryfdfcsld80000gn/T/pip-install-81yvkyvs/pynvx_19ff0317a3854976bb7ea683521d385f', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/private/var/folders/fn/lgvtrxd90cl3nqryfdfcsld80000gn/T/pip-install-81yvkyvs/pynvx_19ff0317a3854976bb7ea683521d385f/build/lib.macosx-11.0-arm64-cpython-310', '-DPYTHON_EXECUTABLE=/Users/connoreaton/miniforge3/envs/spaff_nlp/bin/python3.10', '-DCMAKE_BUILD_TYPE=Release']' returned non-zero exit status 1.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> pynvx

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

I am clearly unable to install pynvx. This makes sense because pynvx deals with CUDA, which apparently Mac M1 cannot support. I tried the CPU only build install documentation from fastai here:
https://fastai1.fast.ai/install.html
But I get the same error.
What can I do? If I can't train models locally that's okay, but I want to at least be able to load models and predict with them.

Output:
fastai seems to need pyenv which needs CUDA to work. CUDA is available only on Nvidia GPUs and the MAC M1 has a completly differente SOC with no Nvidia GPU
You can read the actual error
CMake Error at /opt/homebrew/Cellar/cmake/3.23.1/share/cmake/Modules/FindCUDA.cmake:859


Instruction: Access all batch outputs at the end of epoch in callback with pytorch lightning

Input:
The documentation for the on_train_epoch_end, https://pytorch-lightning.readthedocs.io/en/stable/extensions/callbacks.html#on-train-epoch-end, states:

To access all batch outputs at the end of the epoch, either:

Implement training_epoch_end in the LightningModule and access outputs via the module OR
Cache data across train batch hooks inside the callback implementation to post-process in this hook.

I am trying to use the first alternative with the following LightningModule and Callback setup:
import pytorch_lightning as pl
from pytorch_lightning import Callback

class LightningModule(pl.LightningModule):
    def __init__(self, *args):
        super().__init__()
        self.automatic_optimization = False

    def training_step(self, batch, batch_idx):
        return {'batch': batch}

    def training_epoch_end(self, training_step_outputs):
        # training_step_outputs has all my batches
        return

class MyCallback(Callback):
    def on_train_epoch_end(self, trainer, pl_module):
        # pl_module.batch ???
        return

How do I access the outputs via the pl_module in the callback? What is the recommended way of getting access to training_step_outputs in my callback?

Output:
You can store the outputs of each training batch in a state and access it at the end of the training epoch. Here is an example -
from pytorch_lightning import Callback

class MyCallback(Callback):
    def __init__(self):
        super().__init__()
        self.state = []

    def on_train_batch_end(self, trainer, pl_module, outputs, batch, batch_idx, unused=0):
        self.state.append(outputs)

    def on_train_epoch_end(self, trainer, pl_module):
        # access output using state
        all_outputs = self.state

Hope this helps you!


Instruction: Improve Python (.exe) startup time

Input:
I created an exe with the PyInstaller. As soon as I enable the --onefile flag. The exe needs several minutes to start. When I build the application with the --onedir flag, the exe launches immediately after launch. In order to distribute the application better, it is important for me that the exe is created with the --onefile flag.
My problem can be reproduced with the following two scripts.
In main.py I only import the torch module, because the problems only occur with this module.
#main.py
print("Test1")
import torch
print("Test2")
print(torch.cuda.is_available())
print("Test3")

I generate the exe with the setup.py file.
#setup.py
import PyInstaller.__main__
import pyinstaller_versionfile

PyInstaller.__main__.run([
    'main.py',
    '--onefile',
    '--name=Object Detection',
    '--console',
])

Alternatively, the exe can also be generated using the Powershell command.
pyinstaller --noconfirm --onefile --console --name "Object Detection"

So my question is what can be done to make the application start faster? What is the specific cause of the problem?
By excluding the module --exclude-module=torch the application also starts immediately,
but that's not my goal.

Output: Pyinstaller --onefile mode has to unpack all libraries before starting in a temporary directory, when you use --onedir they are already there. The problem is noticeable with big libraries like PyTorch ...


Instruction: PyTorch distributed dataLoader

Input: Any recommended ways to make PyTorch DataLoader (torch.utils.data.DataLoader) work in distributed environment, single machine and multiple machines? Can it be done without DistributedDataParallel?

Output:
Maybe you need to make your question clear. DistributedDataParallel is abbreviated as DDP, you need to train a model with DDP in a distributed environment. This question seems to ask how to arrange the dataset loading process for distributed training.
First of all,
data.Dataloader is proper for both dist and non-dist training, usually, there is no need to do something on that.
But the sampling strategy varies in this two modes, you need to specify a sampler for the dataloader(the sampler arg in data.Dataloader), adopting  torch.utils.data.distributed.DistributedSampler  is the simplest way.


Instruction: torch.nn.BCEloss() and torch.nn.functional.binary_cross_entropy

Input: What is the basic difference between these two loss functions? I have already tried using both the loss functions.

Output:
The difference is that nn.BCEloss and F.binary_cross_entropy are two PyTorch interfaces to the same operations.

The former, torch.nn.BCELoss, is a class and inherits from nn.Module which makes it handy to be used in a two-step fashion, as you would always do in OOP (Object Oriented Programming): initialize then use. Initialization handles parameters and attributes initialization as the name implies which is quite useful when using stateful operators such as parametrized layers and the kind. This is the way to go when implementing classes of your own, for example:
class Trainer():
    def __init__(self, model):
        self.model = model
        self.loss = nn.BCEloss()

    def __call__(self, x, y)
        y_hat = self.model(x)
        loss = self.loss(y_hat, y)
        return loss

On the other hand, the later, torch.nn.functional.binary_cross_entropy, is the functional interface. It is actually the underlying operator used by nn.BCELoss, as you can see at this line. You can use this interface but this can become cumbersome when using stateful operators. In this particular case, the binary cross-entropy loss does not have parameters (in the most general case), so you could do:
class Trainer():
    def __init__(self, model):
        self.model = model

    def __call__(self, x, y)
        y_hat = self.model(x)
        loss = F.binary_cross_entropy(y_hat, y)
        return loss


Instruction: Which nvidia drivers version do I need?

Input:
I'm running on Ubuntu 20.04.4 LTS
I'm going to work with cuda 11.3 and torch 1.11 python 3.8
Which nvidia driver (version) do I need to install ?
How can I do it ?

Output:
Download and install the latest version and you will be ok:
https://www.nvidia.com/download/index.aspx


Instruction: my code couldn't find instance that was created initialized using pytorch

Input:
I implemented dataset class to use model, and When i strated train i get error
Traceback (most recent call last):
  File "model.py", line 146, in <module>
    train = Train()
  File "model.py", line 70, in __init__
    self.dataset.get_label()
  File "model.py", line 61, in get_label
    return self.label
AttributeError: 'MaskDataset' object has no attribute 'label'

and bellow code is maked error. but I never know why it's make problem. I check 'self.imgs' and 'self.label' used print(self.imgs) and print(self.label). and it was perfecte.
So I Mean, I don't know why python interpreter couldn't find instance that created initialized.
class MaskDataset(object):
def __init__(self, transforms,path):
    self.data = data.Data()
    self.transform = transforms
    self.path = path

    if 'Validation' in self.path :
        self.img_path = "/home/ubuntu/lecttue-diagonosis/YangDongJae/ai/data/Validation/images/"
        self.lab_path = "/home/ubuntu/lecttue-diagonosis/YangDongJae/ai/data/Validation/annotations/"
        self.label = list(sorted(os.listdir(self.lab_path)))
        self.imgs = list(sorted(os.listdir(self.img_path)))

    elif 'train' in self.path:
        self.img_path = "/home/ubuntu/lecttue-diagonosis/YangDongJae/ai/data/Training/images/"
        self.lab_path = "/home/ubuntu/lecttue-diagonosis/YangDongJae/ai/data/Training/annotations/"
        self.label = list(sorted(os.listdir(self.lab_path)))
        self.imgs = list(sorted(os.listdir(self.img_path)))

    def __getitem__(self,idx):
        file_image = self.imgs[idx]
        file_label = self.label[idx]
        img_path = self.img_path+file_image
        label_path = self.lab_path + file_label

        img = Image.open(img_path).convert("RGB")
        target = self.data.generate_target(label_path)

        if self.transform is not None:
            img = self.transform(img)

        return img, target

class Train(MaskDataset):
    def __init__(self,epochs = 100, lr = 0.005, momentum = 0.9, weight_decay = 0.0005):
        self.data_transform = transforms.Compose([  # transforms.Compose : list 내의 작업을 연달아 할 수 있게 호출하는 클래스
                transforms.ToTensor() # ToTensor : numpy 이미지에서 torch 이미지로 변경
            ])

        self.dataset = MaskDataset(self.data_transform,'/home/ubuntu/lecttue-diagonosis/YangDongJae/ai/data/Training/')
        self.val_dataset = MaskDataset(self.data_transform, '/home/ubuntu/lecttue-diagonosis/YangDongJae/ai/data/Validation/')
        self.data_loader = torch.utils.data.DataLoader(self.dataset, batch_size = 10, collate_fn = self.collate_fn)
        self.val_data_loader = torch.utils.data.DataLoader(self.val_dataset, batch_size = 10,collate_fn = self.collate_fn)
        self.num_classes = 8
        self.epochs = epochs
        self.momentum = momentum
        self.lr = 0.005
        self.weight_decay = weight_decay

Output:
This is happening because elif condition is not True during self.dataset object creation. Note that the self.path has a Train sub-string staring with an uppercase T, while elif is comparing it with lower-case train, which evaluates to False. This can be fixed by changing the elif as:
elif 'train'.lower() in self.path.lower():
    self.img_path = "/home/ubuntu/lecttue-diagonosis/YangDongJae/ai/data/Training/images/"
    self.lab_path = "/home/ubuntu/lecttue-diagonosis/YangDongJae/ai/data/Training/annotations/"
    self.label = list(sorted(os.listdir(self.lab_path)))
    self.imgs = list(sorted(os.listdir(self.img_path)

You may also change the if statement for validation case similarly.


Instruction: Databricks notebook hanging with pytorch

Input:
We have a Databricks notebooks issue. One of our notebook cells seems to be hanging, while the driver logs do show that the notebook cell has been executed. Does anyone know why our notebook cell keeps hanging, and does not complete? See below the details.
Situation

We are training a ML model with pytorch in the Databricks notebook UI
The training uses mlflow to register a model
At the end of the cell we print a statement "Done with training"
We are using a single node cluster with

Databricks Runtime: 10.4 LTS ML (includes Apache Spark 3.2.1, GPU, Scala 2.12)
Node type: Standard_NC6s_v3

Observations

In the Databricks notebook UI we see the cell running pytorch training and showing the intermediate logs of the training
After awhile the model is registered in mlflow but we don't see this log in the Databricks notebook UI
We can also see the print statement "Done with training" in the driver logs. We don't see this statement in the Databricks notebook UI

Code
from pytorch_lightning import Trainer
from pytorch_lightning.callbacks.early_stopping import EarlyStopping

trainer = Trainer(gpus=-1, num_sanity_val_steps=0, logger = logger, callbacks=[EarlyStopping(monitor="test_loss", patience = 2, mode = "min", verbose=True)])

with mlflow.start_run(experiment_id = experiment_id) as run:
  trainer.fit(model, train_loader, val_loader)
  mlflow.log_param("param1", param1)
  mlflow.log_param("param2", param2)
  mlflow.pytorch.log_model(model._image_model, artifact_path="model", registered_model_name="image_model")
  mlflow.pytorch.log_state_dict(model._image_model.state_dict(), "model")

print("Done with training")

Packages
mlflow-skinny==1.25.1
torch==1.10.2+cu111
torchvision==0.11.3+cu111

Solutions that I tried that did not work

Tried adding cache deletion, but that did not work

# Cleaning up to avoid any open processes...
del trainer
torch.cuda.empty_cache()
# force garbage collection
gc.collect()

Tried forcing exiting the notebook, but also did not work

parameters = json.dumps({"Status": "SUCCESS", "Message": "DONE"})
dbutils.notebook.exit(parameters)

Output:
I figured out the issue. To solve this, adjust the parameters for the torch.utils.data.DataLoader

Disable pin_memory
Set num_workers to 30% of total vCPU (e.g. 1 or 2 for Standard_NC6s_v3)

For example:
train_loader = DataLoader(
    train_dataset,
    batch_size=32,
    num_workers=1,
    pin_memory=False,
    shuffle=True,
)

This issue seems to be related to PyTorch and is due to a deadlock issue. See the details here.


Instruction: Normalization required for pre-trained model (PyTorch)

Input:
I am using a pre-trained model from pytorch:
model = models.resnet50(pretrained=True).to(device)
for param in model.parameters():
    param.requires_grad = False
model.fc = Identity()

Should I normalize the data using my data mean and std or use the values used by the model creators?
class customDataset(torch.utils.data.Dataset):
    'Characterizes a dataset for PyTorch'
    def __init__(self, X, y):
        'Initialization'
        self.normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
        self.X = X
        self.y = torch.tensor(y, dtype=torch.float32)

    def __len__(self):
        return len(self.X)

    def __getitem__(self, idx):
        X = self.X[idx]
        X = ToTensor()(X).type(torch.float32)[:3,:]
        X = self.normalize(X)
        return X, self.y[idx]

Output:
You must use the normalization mean and std that was used during training. Based on the training data normalization, the model was optimized. In order for the model to work as expected the same data distribution has to be used.
If you train a model from scratch, you can use your dataset specific normalization parameters.


Instruction: Exception when converting Unet from pytorch to onnx

Input:
I'm trying to convert a Unet model from PyTorch to ONNX.
Running the following code:
import torch
from unets import Unet, thin_setup

net = Unet(in_features=3, down=[16, 32, 64, 64, 64], up=[64, 64, 64, 128 + 1],
           setup={**thin_setup, 'bias': True, 'padding': True})
net.eval()

inputs = torch.randn((1, 3, 768, 768))
outputs = net(inputs)
torch.onnx.export(net, inputs, "unet.onnx", opset_version=12)

a  RuntimeError: Unsupported: ONNX export of instance_norm for unknown channel size. exception is raised.
How am I solving it?
remark: I suspect that this is due to a node of a upsample layer that has no output shape:
%196 : Float(*, *, *, *, strides=[589824, 9216, 96, 1], requires_grad=1, device=cpu) = onnx::Resize[coordinate_transformation_mode="pytorch_half_pixel", cubic_coeff_a=-0.75, mode="linear", nearest_mode="floor"](%169, %194, %195, %193) # ~/miniconda/envs/my_env/lib/python3.7/site-packages/torch/nn/functional.py:3709:0
environment: python 3.7 / torch 1.9.1+cu102 / onnx 1.10.2

Output:
The problem is due to ONNX not having an implementation of the PyTorch 2D Instane Normalization layer.
The solution was to copy the relevant UNet code and implement the layer myself:
class InstanceNormAlternative(nn.InstanceNorm2d):

    def forward(self, inp: Tensor) -> Tensor:
        self._check_input_dim(inp)

        desc = 1 / (input.var(axis=[2, 3], keepdim=True, unbiased=False) + self.eps) ** 0.5
        retval = (input - input.mean(axis=[2, 3], keepdim=True)) * desc
        return retval

Make sure to use unbiased variance if you wish to be as similar as possible to  PyTorch.
NOTE: CoreML tools cannot convert the variance operator from PyTorch to CoreML. Make sure to use PyTorch's nn.InstanceNorm2d layer (and not the above alternative) when converting to CoreML.
FREE TIP: If converting PyTorch UNets to TF, you are also going to encounter the following error:
RuntimeError: Resize coordinate_transformation_mode=pytorch_half_pixel is not supported in Tensorflow
The remedy is to change the interpolation parameters in TrivialUpsample.forward to align_corners=True. In my experience, the effect of the change on the network output was minor.
This answer was written with the help of Michał Tyszkiewicz.


Instruction: Extracting feature maps from ResNet

Input:
TLDR - what is considered best practice when extracting feature maps from ResNet?

I'm trying to feed the entire CIFAR10 dataset through ResNet18, to extract a new dataset that consists of some non-output activation of every sample in CIFAR10. I have implemented a code that generates this dataset, but the running time takes too long (exceeds Google Colab free RAM access, which is quite some RAM). The code I've implemented is based on a blog post called Intermediate Activations — the forward hook.
activation = {}

def get_activation(name):
    """
    when given as input to register_forward_hook, this function is implicitly called when model.forward() is performed
    and saves the output of layer 'name' in the dictionary described above.
    :param name:
    :return:
    """

    def hook(model, input, output):
        activation[name] = output.detach()

    return hook

the get_activation helper function is used inside the activation_maps function which takes the feature map provided from the 4th layer, 2nd BasicBlock,  conv1 layer (batch-size, 3,224,224) -> (batch-size,512,7,7) of ResNet18
(PS - this layer was arbitrarily chosen - is there a known layer from which the activations are better?)
ResNet18 = torch.hub.load('pytorch/vision:v0.10.0', 'resnet18', pretrained=True)

def activation_maps(name='conv1'):
    """
    This function takes a batch and returns some non - last activation alongside the true labels
    :return: train_activations_and_true_labels: array of tuples (Activation,True_labels) as train data

    """
    non_output_activation_map = ResNet18.layer4[1].register_forward_hook(get_activation(name))
    # now we create a list of activations and true labels  for every sample.
    # This means that if we looped over (X,y) in a dataloader, we can now loop (activation,y) which is
    # an element in the arrays below, like a regular dataloader.
    train_activations_and_true_labels = []
    for i, (X_train, y_train) in enumerate(train_dataloader):
        out = ResNet18(X_train)
        train_activations_and_true_labels.append((activation[name], y_train))
        print(f"Training data [{i}/{len(train_dataloader)}]", end='\r')
    non_output_activation_map.remove()  # detaching hooks
    return train_activations_and_true_labels

Now, this code runs - but exceeds the memory capacity of my PyCharm/Google-Colab Am I missing something? what is the best approach when extracting feature maps?

Output:
What batch size are you using, and how much RAM do you have available? Resnet is a somewhat large model, and the layer you're extracting is quite large as well so storing all that in memory might be causing issues.
Try reducing your batch size, or storing intermediary results to disk and clearing them from memory.
You might also consider turning off the gradient computation when calling the ResNet18 model, this would save a good bit of memory. Putting the @torch.no_grad() decorator on activation_maps(name='conv1') might work.


Instruction: I was struggling build a lstm model during the model evaluation stage, bug shows different device, can anybody help me out

Input:
for the model building stage

    import torch
from torch import nn
import torch.nn.functional as F
from torch import utils

class LSTM(nn.Module):
    def __init__(self, vocab_size, embed_dim):
        super().__init__()
        self.embedding = nn.EmbeddingBag(vocab_size, embed_dim) # embedding layer
        self.lstm = nn.LSTM(embed_dim,hidden_dim, vocab_size,n_class,bidirectional=True, batch_first=True, dropout=0.3)
        self.fc1 = nn.Linear(hidden_dim * n_class,hidden_dim)

    def forward(self, text):

        text_embeded = self.embedding(text)

        output, (h_n, c_n) = self.lstm(text_embeded)
        out = torch.cat([h_n[-1, :, :], h_n[-2, :, :]], dim=-1)

        out_fc1 = self.fc1(output)
        out_fc1_relu = F.relu(out_fc1)
        return out_fc1_relu

# model initalization
# hyperparameter
embed_dim = 32
hidden_dim = 16
n_class = 5

model = LSTM(vocab_size, embed_dim)

loss_fn = torch.nn.BCEWithLogitsLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)

for the evaluation stage

def device():

  if torch.cuda.is_available():
    return torch.device('cuda')
  else:
    return torch.device('cpu')
from sklearn.metrics import accuracy_score
loss_list_train= []
loss_list_test = []
epochs =  30
for epoch in range(epochs):
  loss_now = 0.0
  correct = 0

  for sentence, targets in train_loader:
    sentence = sentence.to(device())
    targets = targets.to(device())

    temp_batch_size = sentence.shape[0]

    model.train()
    optimizer.zero_grad()
    pred = model(sentence)
    # loss = loss_function(pred.view(-1, pred.shape[-1]), targets.view(-1))
    loss = F.nll_loss(pred, targets)
    loss.backward()
    optimizer.step()

    loss_now += loss.item() * temp_batch_size
    predicted = torch.argmax(pred,-1)
    correct += accuracy_score(predicted.view(-1).cpu().numpy(),targets.view(-1).cpu()).numpy() * temp_batch_size

    model.eval()
    train_loss = loss_now/len(train_data)
    outputs = model(torch.from_numpy(sentence_padding_train).to(device())) # training set raw outputs
    predicted = torch.argmax(outputs, 1) # training set predictions
    f1 = f1_score(label_train, predicted.cpu().numpy().reshape(-1), average='weighted')
# training set f1-score

    outputs_test = model(torch.from_numpy(sentence_padding_val). to(device())) # testing set raw outputs
    loss_test = loss_function(outputs_test, torch.from_numpy(label_val).view(-1).to(device())) # testing set average loss
    predicted_test = torch.argmax(outputs_test, 1) # testing set predictions
    f1_test =f1_score(label_val, predicted_test.cpu().numpy(), average-'weighted' )
# testing set f1-score

    loss_list_train.append(train_loss)
    loss_list_test.append(loss_test.item())
    print('Epoch {}: {:.4f} (train loss), {:.4f} (val loss), {:.4f} (train f1-score), {:.4f} (test f1-score), {:.4f}'.format(epoch + 1, train_loss, loss_test.item,f1,f1_test))

the bug shows below, says "Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument indices in method wrapper___embedding_bag", I've already define the device, how can I solve this problem?
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-99-2fb3a5047897> in <module>()
     22     model.train()
     23     optimizer.zero_grad()
---> 24     pred = model(sentence)
     25     # loss = loss_function(pred.view(-1, pred.shape[-1]), targets.view(-1))
     26     loss = F.nll_loss(pred, targets)

4 frames
/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1108         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1109                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1110             return forward_call(*input, **kwargs)
   1111         # Do not call functions when jit is used
   1112         full_backward_hooks, non_full_backward_hooks = [], []

<ipython-input-94-7b5f176858ed> in forward(self, text)
     14     def forward(self, text):
     15
---> 16         text_embeded = self.embedding(text)
     17
     18         output, (h_n, c_n) = self.lstm(text_embeded)

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1108         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1109                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1110             return forward_call(*input, **kwargs)
   1111         # Do not call functions when jit is used
   1112         full_backward_hooks, non_full_backward_hooks = [], []

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/sparse.py in forward(self, input, offsets, per_sample_weights)
    385                                self.scale_grad_by_freq, self.mode, self.sparse,
    386                                per_sample_weights, self.include_last_offset,
--> 387                                self.padding_idx)
    388
    389     def extra_repr(self) -> str:

/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py in embedding_bag(input, weight, offsets, max_norm, norm_type, scale_grad_by_freq, mode, sparse, per_sample_weights, include_last_offset, padding_idx)
   2363
   2364     ret, _, _, _ = torch.embedding_bag(
-> 2365         weight, input, offsets, scale_grad_by_freq, mode_enum, sparse, per_sample_weights, include_last_offset, padding_idx
   2366     )
   2367     return ret

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument indices in method wrapper___embedding_bag)

Output:
The error indicates that two tensors are on different devices (clear to understand).
Since you'd moved your input tensor to your GPU via sentence = sentence.to(device()) but not your model, the model's parameters are on the CPU which causes this error. Just add model.to(device()) before your training procedure starts.