title: How to apply random forests to the output produced by Bert?
body: <p>I'm trying to get the output embeddings of a RoBERTa model, so I can train a random forests classifier on it for text classification (sentiment analysis). The original dataset this is based on is 500 news articles that each have a left/center/right bias rating. 80% of this dataset is training data, the other 20% is test data.</p>
  <p>I run the following code for my training set:</p>
  <pre><code># Tokenize sentences van trainingset 
  encoded_input = tokenizer(X_train, padding=True, truncation=True, return_tensors='pt')
  
  # Compute token embeddings
  with torch.no_grad():
      model_output = model(**encoded_input)
  
  # Perform pooling 
  sentence_embeddings = mean_pooling(model_output,encoded_input['attention_mask'])
  
  ## Normalize embeddings
  sentence_embeddings = F.normalize(sentence_embeddings, p=3, dim=1)
  
  start = True
  numpy_emb = [] 
  
  if not start:
      np_emb = sentence_embeddings.cpu().detach().numpy()
      numpy_emb = np.vstack([numpy_emb, np_emb]) 
  else:
      start = False
      numpy_emb = np_emb = sentence_embeddings.cpu().detach().numpy()
  </code></pre>
  <p>Which gives me numpy_emb. Which I think are the embeddings that the RoBERTa model outputs.
  When I print it, it gives me:</p>
  <pre><code>tensor([[ 0.5329, -0.1224],
          [ 0.5409, -0.0730],
          [ 0.4594, -0.1282],
          [ 0.5116, -0.0769],
          [ 0.4861, -0.0212],
          [ 0.5246, -0.0560],
          [ 0.5555, -0.0962],
          [ 0.4779, -0.0551],
          [ 0.5428, -0.0904],
          [ 0.5939, -0.0504],
          [ 0.5219, -0.1342],
          [ 0.4672, -0.0936],
          [ 0.5051, -0.0518],
          [ 0.5536, -0.1016],
          [ 0.4761, -0.0736],
          [ 0.4754, -0.0991],
          [ 0.5613, -0.0541],
          [ 0.5155,  0.0303],
          [ 0.6053,  0.0214],
          [ 0.4766, -0.1019],
          [ 0.4262, -0.0869],
          [ 0.3871, -0.0756],
          [ 0.5048, -0.0067],
          [ 0.5425, -0.1303],
          [ 0.5020, -0.0715],
      ...
          [ 0.5462, -0.0686],
          [ 0.5476, -0.1465],
          [ 0.4968, -0.0354],
          [ 0.5586, -0.1234],
          [ 0.5725, -0.0685]])
  </code></pre>
  <p>I then repeat this process for my test set as well, giving me another set of embeddings.</p>
  <p>Then I try to train a random forests classifier using the embeddings given by the training set. But when I try to predict using the embeddings from my test set, I get very random results. Accuracy goes as low as 24% and as high as 58%. Is this because of the small amount of data that I have? Or is there something else I'm doing wrong?</p>
  <p>I also have the suspicion that I can't properly link the output embedding to their respective label. Which would also explain the random results I get.</p>
  <p>Code for random forests that I used:</p>
  <pre><code>from sklearn.ensemble import RandomForestClassifier
  text_classifier = RandomForestClassifier(n_estimators=100, random_state=0)
  text_classifier.fit(numpy_emb, y_train)
  
  predictions = text_classifier.predict(numpy_emb_test)
  
  #confusion matrix
  from sklearn.metrics import classification_report, confusion_matrix, accuracy_score
  print(confusion_matrix(y_test,predictions))
  print(classification_report(y_test,predictions))
  </code></pre>