title: Building ML classifier with imbalanced data
body: <p>I have a dataset with 1400 obs and 19 columns. The Target variable has values 1 (value that I am most interested in) and 0. The distribution of classes shows imbalance (70:30).</p>
  <p>Using the code below I am getting weird values (all 1s). I am not figuring out if this is due to a problem of overfitting/imbalance data or to feature selection (I used Pearson correlation since all values are numeric/boolean).
  I am thinking that the steps followed are wrong.</p>
  <pre><code>import numpy as np
  import math
  import sklearn.metrics as metrics
  from sklearn.metrics import f1_score
  
  y = df['Label']
  X = df.drop('Label',axis=1)
  
  def create_cv(X,y):
      if type(X)!=np.ndarray:
          X=X.values
          y=y.values
   
      test_size=1/5
      proportion_of_true=y[y==1].shape[0]/y.shape[0]
      num_test_samples=math.ceil(y.shape[0]*test_size)
      num_test_true_labels=math.floor(num_test_samples*proportion_of_true)
      num_test_false_labels=math.floor(num_test_samples-num_test_true_labels)
      
      y_test=np.concatenate([y[y==0][:num_test_false_labels],y[y==1][:num_test_true_labels]])
      y_train=np.concatenate([y[y==0][num_test_false_labels:],y[y==1][num_test_true_labels:]])
  
      X_test=np.concatenate([X[y==0][:num_test_false_labels] ,X[y==1][:num_test_true_labels]],axis=0)
      X_train=np.concatenate([X[y==0][num_test_false_labels:],X[y==1][num_test_true_labels:]],axis=0)
      return X_train,X_test,y_train,y_test
  
  X_train,X_test,y_train,y_test=create_cv(X,y)
  X_train,X_crossv,y_train,y_crossv=create_cv(X_train,y_train)
      
  tree = DecisionTreeClassifier(max_depth = 5)
  tree.fit(X_train, y_train)       
  
  y_predict_test = tree.predict(X_test)
  
  print(classification_report(y_test, y_predict_test))
  f1_score(y_test, y_predict_test)
  </code></pre>
  <p>Output:</p>
  <pre><code>     precision    recall  f1-score   support
  
             0       1.00      1.00      1.00        24
             1       1.00      1.00      1.00        70
  
      accuracy                           1.00        94
     macro avg       1.00      1.00      1.00        94
  weighted avg       1.00      1.00      1.00        94
  </code></pre>
  <p>Has anyone experienced similar issues in building a classifier when data has imbalance, using CV and/or under sampling? Happy to share the whole dataset, in case you might want to replicate the output.
  What I would like to ask you for some clear answer to follow that can show me the steps and what I am doing wrong.</p>
  <p>I know that, to reduce overfitting and work with balance data, there are some methods such as random sampling (over/under), SMOTE, CV. My idea is</p>
  <ul>
  <li>Split the data on train/test taking into account imbalance</li>
  <li>Perform CV on trains set</li>
  <li>Apply undersampling only on a test fold</li>
  <li>After the model has been chosen with the help of CV, undersample the train set and train the classifier</li>
  <li>Estimate the performance on the untouched test set
  (f1-score)</li>
  </ul>
  <p>as also outlined in this question: <a href="https://stackoverflow.com/questions/67537605/cv-and-under-sampling-on-a-test-fold">CV and under sampling on a test fold</a> .</p>
  <p>I think the steps above should make sense, but happy to receive any feedback that you might have on this.</p>