title: How to deal with small AND unbalanced datasets for machine learning classification problems
body: <p>I am dealing with a very challenging classification problem where I have three problems: A small dataset (about 800 samples), unbalanced dataset (4 classes where 1 - 600 samples, 2/3/4 - 50 samples each) and missing data in one of the features. </p>
  
  <p>Some things that I have been considering:</p>
  
  <ul>
  <li><p>Generate synthetic data, for example using SMOTE (Synthetic Minority Over-sampling Technique). </p></li>
  <li><p>Turn the classification into a binary classification between the minority and majority. </p></li>
  <li><p>Combine different classifiers giving more weight on negative samples (in case I turn into a binary classifier). </p></li>
  <li><p>Cost sensitive learning by applying specific weights in the cost function (kinda similar to the previous, but using all 4 classes). </p></li>
  </ul>
  
  <p>I intend to use as classifiers Naive Bayes, SVM, Random Forests and Neural Networks and 2 fold cross validation. Later I might move to 5 to 10 folds. </p>
  
  <p>Some characteristics of the features:</p>
  
  <ul>
  <li><p>5 continuous, where 3 of them are just different properties based on graph location (min, max and distribution) and some of them have very low variance and repeated data</p></li>
  <li><p>2 binary features where one of them have missing data.</p></li>
  </ul>
  
  <p>Snippet of the data:</p>
  
  <pre><code>Y   X1  X2_min  X2_max  X2_distribution X3  X4  X5
  3   6   1       11      3.3058739       0   1   1
  3   662 1       11      1.7779095       1   15  1
  1   6   1       7       3.060274        0   1   1
  3   8   1       6       2.9697127       0   1   1
  3   82  1       14      3.0341356       0   1   1
  2   39  1       7       4.2189913       0   1   1
  4   1   3       14      4.6185904           1   1
  </code></pre>
  
  <p>I would appreciate very much any second thought.</p>