I am trying to use pytorch's implementation of XLNet and got 'Trying to create tensor with negative dimension -1: [-1, 768]' when loading XLNet
I started working on this about two months ago on Google Colab for a midterm project and everything worked perfectly. Now I am modifying it for a final project and keep getting the error 'RuntimeError: Trying to create tensor with negative dimension -1: [-1, 768]'. It looks like pytorch recently pushed a new version 1.5, so I downgraded to version 1.4 and still got the same error. Same with 1.3, and I know I wasn't using anything lower since that came out last year. I checked it with my midterm code and still got the same error, so I don't know what's going on. Here is the chunk of code related to downloading and using the model.

    train_inputs, validation_inputs, train_labels, validation_labels = train_test_split(inputIds, 
                                                                                        labels, 
                                                                                        random_state=2020, 
                                                                                        test_size=0.2)
    train_masks, validation_masks, _, _ = train_test_split(attention_masks, inputIds, random_state=2020, 
                                                           test_size=0.2)
    # Turn data into torch tensors
    train_inputs = torch.tensor(train_inputs)
    validation_inputs = torch.tensor(validation_inputs)
    train_labels = torch.tensor(train_labels)
    validation_labels = torch.tensor(validation_labels)
    train_masks = torch.tensor(train_masks)
    validation_masks = torch.tensor(validation_masks)

    # Create Iterators of the datasets
    train_data = TensorDataset(train_inputs, train_masks, train_labels)
    train_sampler = RandomSampler(train_data)
    train_dataloader = DataLoader(train_data, sampler=train_sampler, batch_size=batch_size)
    validation_data = TensorDataset(validation_inputs, validation_masks, validation_labels)
    validation_sampler = SequentialSampler(validation_data)
    validation_dataloader = DataLoader(validation_data, sampler=validation_sampler, batch_size=batch_size)

    model = XLNetForSequenceClassification.from_pretrained('xlnet-base-cased', num_labels=2)
    # Loads model into GPU memory
    model.cuda()

    param_optimizer = list(model.named_parameters())
    no_decay = ['bias','gamma','beta']
    optimizer_grouped_parameters = [
        {'params':[p for n, p in param_optimizer if not any(nd in n for nd in no_decay)],
         'weight_decay_rate':0.01},
        {'params':[p for n, p in param_optimizer if any(nd in n for nd in no_decay)],
         'weight_decay_rate':0.0}
    ]
    optimizer = AdamW(optimizer_grouped_parameters, lr=2e-5)


The error happens on the line model = XLNetForSequenceClassification.from_pretrained('xlnet-base-cased', num_labels=2). The packages I am using:

from pandas import to_datetime
import torch
from torch.utils.data import TensorDataset, DataLoader, RandomSampler, SequentialSampler
from keras.preprocessing.sequence import pad_sequences
from sklearn.model_selection import train_test_split
# MUST INSTALL PYTORCH-TRANSFORMERS
from pytorch_transformers import XLNetTokenizer, XLNetForSequenceClassification, AdamW
from tqdm import trange
from numpy import argmax, sum 
import nltk
nltk.download('punkt')


Thank you to anyone who tries to help.

You can try transformers instead of pytorch_transformers.

! pip install transformers (Google Colab)

In terminal,

pip install transformers

import torch

from transformers import XLNetForSequenceClassification

model = XLNetForSequenceClassification.from_pretrained('xlnet-base-cased', num_labels=2)

model.cuda()

param_optimizer = list(model.named_parameters())
no_decay = ['bias','gamma','beta']
optimizer_grouped_parameters = [
    {'params':[p for n, p in param_optimizer if not any(nd in n for nd in no_decay)],
      'weight_decay_rate':0.01},
    {'params':[p for n, p in param_optimizer if any(nd in n for nd in no_decay)],
      'weight_decay_rate':0.0}
]


Here's the code without any error in google colab: https://colab.research.google.com/drive/1A8edGYyFuE7d1Z-ZJusz5zU_7yCevF2L
