title: Pseudo Labelling on Text Classification Python
body: <p>I'm not good at machine learning. Can someone tell me how to doing text classification with pseudo labeling in python? I never know the right implementation, I have searched everywhere in internet, but I give up as found anything :'( I just found the implementation for numeric datasets, but I found no implementation for text classification (vectorized text).. So I wrote this syntax, but I don't know whether my code is correct or not. Am I doing wrong? Please help me guys, I really need your help.. :'(</p>
  <p>This is my <a href="https://drive.google.com/file/d/1V4_wl2tl5hpVVJDRWgz6g-buKyQhAMba/view?usp=sharing" rel="nofollow noreferrer">datasets</a> if you wanna try. I want to classify 'Label' from 'Content'</p>
  <p>My steps are:</p>
  <ol>
  <li>Split data 0.75 unlabeled, 0.25 labeled</li>
  <li>From 0.25 labeld I split: 0.75 train labeled, and 0.25 test labeled</li>
  <li>Make vectorizer for train, test and unlabeled datasets</li>
  <li>Build first model from train labeled, then labelling the unlabeled datasets</li>
  <li>Concatting train labeled data with prediction of unlabeled that have &gt;0.99 (pseudolabeled), and make the second model</li>
  <li>Remove pseudolabeled from unabeled datasets</li>
  <li>Predict the remaining unlabeled from second model, then iterate step 3 until the probability of predicted pseudolabeled &lt;0.99.</li>
  </ol>
  <p>This is my code:</p>
  <h2>Performing pseudo labelling on text classification</h2>
  <pre><code>from sklearn.naive_bayes import MultinomialNB
  
  # Initiate iteration counter
  iterations = 0
  
  # Containers to hold f1_scores and # of pseudo-labels
  train_f1s = []
  test_f1s = []
  pseudo_labels = []
  
  # Assign value to initiate while loop
  high_prob = [1] 
  
  # Loop will run until there are no more high-probability pseudo-labels
  while len(high_prob) &gt; 0:
      
      # Set the vector transformer (from data train)
      columnTransformer = ColumnTransformer([
      ('tfidf',TfidfVectorizer(stop_words=None, max_features=100000),
       'Content')
      ],remainder='drop')
  
      def transforms(series):
          before_vect = pd.DataFrame({'Content':series})
          vector_transformer = columnTransformer.fit(pd.DataFrame({'Content':X_train}))
          return vector_transformer.transform(before_vect)
  
      X_train_df = transforms(X_train);
      X_test_df = transforms(X_test);
      X_unlabeled_df = transforms(X_unlabeled)
      
      # Fit classifier and make train/test predictions
      nb = MultinomialNB()
      nb.fit(X_train_df, y_train)
      y_hat_train = nb.predict(X_train_df)
      y_hat_test = nb.predict(X_test_df)
  
      # Calculate and print iteration # and f1 scores, and store f1 scores
      train_f1 = f1_score(y_train, y_hat_train)
      test_f1 = f1_score(y_test, y_hat_test)
      print(f&quot;Iteration {iterations}&quot;)
      print(f&quot;Train f1: {train_f1}&quot;)
      print(f&quot;Test f1: {test_f1}&quot;)
      train_f1s.append(train_f1)
      test_f1s.append(test_f1)
     
      # Generate predictions and probabilities for unlabeled data
      print(f&quot;Now predicting labels for unlabeled data...&quot;)
  
      pred_probs = nb.predict_proba(X_unlabeled_df)
      preds = nb.predict(X_unlabeled_df)
      prob_0 = pred_probs[:,0]
      prob_1 = pred_probs[:,1]
  
      # Store predictions and probabilities in dataframe
      df_pred_prob = pd.DataFrame([])
      df_pred_prob['preds'] = preds
      df_pred_prob['prob_0'] = prob_0
      df_pred_prob['prob_1'] = prob_1
      df_pred_prob.index = X_unlabeled.index
      
      # Separate predictions with &gt; 99% probability
      high_prob = pd.concat([df_pred_prob.loc[df_pred_prob['prob_0'] &gt; 0.99],
                             df_pred_prob.loc[df_pred_prob['prob_1'] &gt; 0.99]],
                            axis=0)
      
      print(f&quot;{len(high_prob)} high-probability predictions added to training data.&quot;)
      
      pseudo_labels.append(len(high_prob))
  
      # Add pseudo-labeled data to training data
      X_train = pd.concat([X_train, X_unlabeled.loc[high_prob.index]], axis=0)
      y_train = pd.concat([y_train, high_prob.preds])      
      
      # Drop pseudo-labeled instances from unlabeled data
      X_unlabeled = X_unlabeled.drop(index=high_prob.index)
      
      print(f&quot;{len(X_unlabeled)} unlabeled instances remaining.\n&quot;)
      
      # Update iteration counter
      iterations += 1
  </code></pre>
  <p>I think I'm doing something wrong.. Because when I see the f1 scores it is decreasing. Please help me guys :'( I'm stressed.
  <a href="https://i.stack.imgur.com/acnUh.jpg" rel="nofollow noreferrer">f1 scores image</a></p>
  <h2>=================EDIT=================</h2>
  <p>So I've search on journal, then I think that I've got misunderstanding about the concept of data splitting in pseudo-labelling.</p>
  <p>I initially thought that, the steps starts from splitting the data into labeled and unlabeled data, then from that labeled data, it was splitted into train and test.</p>
  <p>But after surfing and searching, I found in <a href="http://ejnteti.jteti.ugm.ac.id/index.php/JNTETI/article/view/418/339" rel="nofollow noreferrer">this journal</a> that my steps is incorrect. This journal says that the steps pseudo-labeling should start from splitting the data into train and test sets at first, and then from that train sets, data is splited to labeled and unlabeled datasets.</p>
  <p>According to that journal, it reach the best result when splitting data into 90% of train sets and 10% of test sets. Then, from that 90% train set, it is splitted into 20% labeled data and 80% unlabeled data sets. This journal trying evidence range from 0.7 till 0.9 as boundary to drop the pseudo labeling, and on that proportion of splitting, the best evidence threshold value is 0.74. So I fix my steps with that new proportion and 0.74 threshold, and I finally got the F1 scores is increasing. Here are my steps:</p>
  <ol>
  <li>Split data 0.9 train, 0.1 test sets (I labeled the test sets, so I can measure the f1 scores)</li>
  <li>From 0.9 train, I split: 0.2 labeled, and 0.8 unlabeled data</li>
  <li>Making vectorizer for X value of labeled train, test and unlabeled training datasets</li>
  <li>Build first model from labeled train, then labeling the unlabeled training datasets. Then measure the F-1 scores according to the test sets (that already labeled).</li>
  <li>Concatting train labeled data with prediction of unlabeled that have probability &gt; 0.74 (threshold based on journal). We call this new data as pseudo-labelled, likened to the actual label), and make the second model from new train data sets.</li>
  <li>Remove selected pseudo-labelled from unlabeled datasets</li>
  <li>Use the second model to predict the remaining of unlabeled data, then iterate step 3 until there are no probability of predicted pseudo-labelled&gt;0.74</li>
  <li>So the last model is the final.</li>
  </ol>
  <p>My syntax is still the same, I just changing the split proportion and I finally got my f1 scores increasing through 4 iterations: <a href="https://i.stack.imgur.com/xN1Ds.jpg" rel="nofollow noreferrer">my new f1 scores</a>.</p>
  <p>Am I doing something right? Thank you for all of your attention guys.. So much thank you..</p>