Applying pre-trained BERT model to make predictions
I have recently been given a BERT model that has been pre-trained with a mental health dataset that I have. Now all I have to do is apply the model to a larger dataset to test its performance. I am absolutely new to machine learning and am stuck in this step. I am using PyTorch and would like to continue using it. Here is what I have tried so far:

    df = pd.read_csv('C:/Users/X/Downloads/number_users_09_05.csv',  encoding='unicode- 
    escape')
    
    REPLACE_BY_SPACE_RE = re.compile('[/(){}\[\]\|@,;*]')
    BAD_SYMBOLS_RE = re.compile('[^0-9a-z #+_]')
    STOPWORDS = set(stopwords.words('english'))
    
    # function to clean text
    def clean_text(text):
        text = text.lower() 
        text = REPLACE_BY_SPACE_RE.sub(' ', text) 
        text = BAD_SYMBOLS_RE.sub(' ', text) 
        text = ' '.join(word for word in text.split() if word not in STOPWORDS) 
        return text
    
    df['cleaned_text'] = df['post'].apply(clean_text)
    
    One-Hot Encoding
    
    df['Categories'] = df['Categories'].astype(str)
    
    from sklearn.preprocessing import MultiLabelBinarizer
    df['Categories'] = df['Categories'].str.split(&quot;,&quot;).tolist()
    multilabel_binarizer = MultiLabelBinarizer()
    y = multilabel_binarizer.fit_transform(df['Categories'])
    print(multilabel_binarizer.classes_)
    
    for idx, Categories in enumerate(multilabel_binarizer.classes_):
        df[Categories] = y[:,idx]
    X = df['cleaned_text'].values
    
    from sklearn.model_selection import train_test_split
    x_tr,x_val,y_tr,y_val=train_test_split(X, y, test_size=0.2, random_state=0,shuffle=True)
    
    input_dir = 'C:/Users/X/Documents/ppd_bert_model/'
    
    from transformers import AutoModelForMaskedLM
    model = AutoModelForMaskedLM.from_pretrained(input_dir)
    
    from transformers import AutoTokenizer
    tokenizer = AutoTokenizer.from_pretrained(input_dir)
    
    model.load_weights('C:/Users/X/Documents/ppd_bert_model/')


At this step I get the following error:
AttributeError                            Traceback (most recent call last)
Input In [30], in &lt;cell line: 1&gt;()
----&gt; 1 model.load_weights('C:/Users/X/Documents/ppd_bert_model/')
      2 pred_prob = model.predict(x_val_seq)

File ~\Anaconda3\lib\site-packages\torch\nn\modules\module.py:1207, in Module.__getattr__(self, name)
   1205     if name in modules:
   1206         return modules[name]
-&gt; 1207 raise AttributeError(&quot;'{}' object has no attribute '{}'&quot;.format(
   1208     type(self).__name__, name))

AttributeError: 'BertForMaskedLM' object has no attribute 'load_weights'

Then after this I tried:
    pred_prob = model.predict(df['cleaned_text'])

And got this error:
AttributeError                            Traceback (most recent call last)
Input In [28], in &lt;cell line: 1&gt;()
----&gt; 1 pred_prob = model.predict(df['cleaned_text'])

File ~\Anaconda3\lib\site-packages\torch\nn\modules\module.py:1207, in Module.__getattr__(self, name)
   1205     if name in modules:
   1206         return modules[name]
-&gt; 1207 raise AttributeError(&quot;'{}' object has no attribute '{}'&quot;.format(
   1208     type(self).__name__, name))

AttributeError: 'BertForMaskedLM' object has no attribute 'predict'

I have the pretrained model saved in a folder at 'C:/Users/X/Documents/ppd_bert_model/', and the model has three files: config.json, pytorch_model.bin, and vocab.txt. If anyone has any advice on how I should be calling and using the model to make predictions and test performance, please do let me know.
Thank you in advance!

From the HuggingFace documentation here, the AutoModelForMaskedLM class uses the from_pretrained() method to instantiate one of the model classes of the library (with a masked language modeling head) from a pretrained model. This is done either by using the model_type property in the config object (either passed as an argument, or loaded from pretained_model_name_or_path (in your case &quot;C:/Users/X/Documents/ppd_bert_model/&quot;), or if the model_type property is missing, it will fall back to pattern matching on pretrained_model_name_or_path.
Based on your first error AttributeError: 'BertForMaskedLM' object has no attribute 'load_weights', the BertForMaskedLM is being automatically loaded and an associated class instantiated by the from_pretrained() method call. If you follow the documentation I've hyperlinked, and look at the BertForMaskedLM class, there is no method load_weights(), hence the first error you encountered with your method call. You'll also see in the documentation that there is no predict() method, hence the second error you encountered. At the bottom of that subsection you'll find an example code snippet that shows you how to use the BertForMaskedLM model for masked token prediction. I'll reproduce the snippet here for your convenience and future reference, should their documentation change or the link break:
&gt;&gt;&gt; from transformers import BertTokenizer, BertForMaskedLM
&gt;&gt;&gt; import torch

&gt;&gt;&gt; tokenizer = BertTokenizer.from_pretrained(&quot;bert-base-uncased&quot;)
&gt;&gt;&gt; model = BertForMaskedLM.from_pretrained(&quot;bert-base-uncased&quot;)

&gt;&gt;&gt; inputs = tokenizer(&quot;The capital of France is [MASK].&quot;, return_tensors=&quot;pt&quot;)

&gt;&gt;&gt; with torch.no_grad():
...    logits = model(**inputs).logits

&gt;&gt;&gt; # retrieve index of [MASK]
&gt;&gt;&gt; mask_token_index = (inputs.input_ids == tokenizer.mask_token_id)[0].nonzero(as_tuple=True)[0]

&gt;&gt;&gt; predicted_token_id = logits[0, mask_token_index].argmax(axis=-1)

&gt;&gt;&gt; tokenizer.decode(predicted_token_id)
'paris'

Hopefully this helps get you on the right track towards making predictions and testing performance.
