(with cpu)Pytorch: IndexError: index out of range in self. (with cuda)Assertion `srcIndex < srcSelectDimSize` failed. How to solve?
Today I get the following error when I use BERT with Pytorch and cuda: /pytorch/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [234,0,0], thread: [0,0,0] Assertion srcIndex &lt; srcSelectDimSize failed.
Epoch [1/100]
Iter:      0,  Train Loss:   1.1,  Train Acc: 39.06%,  Val Loss:   1.0,  Val Acc: 51.90%,  Time: 0:00:04 *
Iter:     10,  Train Loss:  0.99,  Train Acc: 57.81%,  Val Loss:   1.0,  Val Acc: 52.01%,  Time: 0:00:11 *
Iter:     20,  Train Loss:   1.0,  Train Acc: 42.19%,  Val Loss:  0.99,  Val Acc: 52.01%,  Time: 0:00:17 *
Iter:     30,  Train Loss:   1.0,  Train Acc: 40.62%,  Val Loss:  0.99,  Val Acc: 52.12%,  Time: 0:00:23 *
Iter:     40,  Train Loss:   1.0,  Train Acc: 50.00%,  Val Loss:  0.98,  Val Acc: 52.12%,  Time: 0:00:29 *
Iter:     50,  Train Loss:   1.1,  Train Acc: 43.75%,  Val Loss:  0.98,  Val Acc: 52.12%,  Time: 0:00:35 *
Traceback (most recent call last):
  File &quot;/content/drive/MyDrive/Prediction/run.py&quot;, line 38, in &lt;module&gt;
    train(config, model, train_iter, dev_iter, test_iter)
  File &quot;/content/drive/MyDrive/Prediction/train_eval.py&quot;, line 50, in train
    outputs = model(trains)
  File &quot;/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py&quot;, line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File &quot;/content/drive/MyDrive/Prediction/models/BERT+Covid.py&quot;, line 68, in forward
    output  = self.bert(context, attention_mask=mask)
  File &quot;/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py&quot;, line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File &quot;/usr/local/lib/python3.7/dist-packages/transformers/models/bert/modeling_bert.py&quot;, line 1005, in forward
    return_dict=return_dict,
  File &quot;/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py&quot;, line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File &quot;/usr/local/lib/python3.7/dist-packages/transformers/models/bert/modeling_bert.py&quot;, line 589, in forward
    output_attentions,
  File &quot;/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py&quot;, line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File &quot;/usr/local/lib/python3.7/dist-packages/transformers/models/bert/modeling_bert.py&quot;, line 475, in forward
    past_key_value=self_attn_past_key_value,
  File &quot;/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py&quot;, line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File &quot;/usr/local/lib/python3.7/dist-packages/transformers/models/bert/modeling_bert.py&quot;, line 408, in forward
    output_attentions,
  File &quot;/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py&quot;, line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File &quot;/usr/local/lib/python3.7/dist-packages/transformers/models/bert/modeling_bert.py&quot;, line 323, in forward
    attention_scores = attention_scores / math.sqrt(self.attention_head_size)
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [234,0,0], thread: [0,0,0] Assertion `srcIndex &lt; srcSelectDimSize` failed.

 #......I SKIPPED SEVERAL LINES DUE TO THE CHARACTER LIMITATION

/pytorch/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [235,0,0], thread: [127,0,0] Assertion `srcIndex &lt; srcSelectDimSize` failed

to find where exactly went wrong, I run my code again with CPU, and I got this error: IndexError: index out of range in self.
traceback (most recent call last):
  File &quot;/content/drive/MyDrive/Prediction/run.py&quot;, line 37, in &lt;module&gt;
    train(config, model, train_iter, dev_iter, test_iter)
  File &quot;/content/drive/MyDrive/Prediction/train_eval.py&quot;, line 49, in train
    outputs = model(trains)
  File &quot;/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py&quot;, line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File &quot;/content/drive/MyDrive/Prediction/models/BERT+Covid.py&quot;, line 66, in forward
    output  = self.bert(context, attention_mask=mask, )
  File &quot;/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py&quot;, line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File &quot;/usr/local/lib/python3.7/dist-packages/transformers/models/bert/modeling_bert.py&quot;, line 993, in forward
    past_key_values_length=past_key_values_length,
  File &quot;/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py&quot;, line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File &quot;/usr/local/lib/python3.7/dist-packages/transformers/models/bert/modeling_bert.py&quot;, line 215, in forward
    inputs_embeds = self.word_embeddings(input_ids)
  File &quot;/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py&quot;, line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File &quot;/usr/local/lib/python3.7/dist-packages/torch/nn/modules/sparse.py&quot;, line 160, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File &quot;/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py&quot;, line 2043, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
IndexError: index out of range in self

According to the guidance I found online, I have ensured the following issue:

the input length didn't exceed the maximum length in the model (the pad size I set is 98, and I had tried to print out the shape of the input before the line went wrong. it did be (batch_size, pad_size)).

len(tokenizer)==model.config.vocab_size, so this is not the problem


I have no idea now what could be the problem, could anybody help me?
My model structure is:
class Model(nn.Module):

    def __init__(self, config):
        super(Model, self).__init__()
        self.modelConfig = BertConfig.from_pretrained('./bert_pretrain/config.json')
        self.bert = BertModel.from_pretrained(config.bert_path,config=self.modelConfig)
        for param in self.bert.parameters():
            param.requires_grad = False
        self.cls_fc_layer = FCLayer(config.hidden_size, config.word_size, config.dropout_rate)
        self.label_classifier = FCLayer(
            config.word_size+config.numerical_size,
            config.num_classes,
            config.dropout_rate,
            use_activation=False,
        )

    def forward(self, x):
        context = x[0]  # input token ids
        mask = x[2]  # mask
        numerical=x[3] #size(batch_size,18)
        
        output  = self.bert(context, attention_mask=mask)
        pooled_output=output[1]
        ##size(batch_size,768)
        pooled_output = self.cls_fc_layer(pooled_output)
        ##size(batch_size,18)
        concat_h = torch.cat([pooled_output, numerical], dim=-1)
        ##size(batch_size,36)
        logits = self.label_classifier(concat_h)
        return logits


I have resolved it!!!
By printing out the  maximum input_ids each batch
for i, (trains, labels) in enumerate(train_iter):
            print(&quot;train max input:&quot;, torch.max(trains[0]))
            print(&quot;train min input:&quot;, torch.min(trains[0]))
            print(&quot;train max label:&quot;, torch.max(labels))
            print(&quot;train min label:&quot;, torch.min(labels))

I got the following output, the max input_id == 21128, while the length of my tokenizer == 21128, which means the maximum input_id should be 21127, this is where the index is out of range!
train max input: tensor(21128, device='cuda:0')
train min input: tensor(0, device='cuda:0')
train max label: tensor(2, device='cuda:0')
train min label: tensor(0, device='cuda:0')

The reason why this error occurs might be because I manually changed the vocab.txt file of the bert model (Sorry I was new to that...), and I resolved this problem by reloading the original BERT model/vocab and config
