Data splitting for a binary classification model I'm trying to build a binary classification model that will tell who's going to buy the product and who's not. I've heard that splitting a dataset into two different subsets is a common way when you prepare an input data.


  [ ================ Training Data 80% ================= ] [ ==== Test Set 20% ==== ]


Is it just mindlessly splitting a chunk of dataset by some amount of proportion like above? Is it that simple?

Imagine I have this simple dataset below.

UserId,UserName,AppId,Purchased
1,Lianne,1,1
1,Lianne,2,1
1,Lianne,3,1
1,Lianne,4,1
1,Lianne,5,1
1,Lianne,6,0
1,Lianne,7,0
1,Lianne,8,0
1,Lianne,9,0
1,Lianne,10,0   


As the common recommended way, I splitted it into two groups.

// Training Data Set
UserId,UserName,AppId,Purchased
1,Lianne,1,1
1,Lianne,2,1
1,Lianne,3,1
1,Lianne,4,1
1,Lianne,5,1
1,Lianne,6,0
1,Lianne,7,0
1,Lianne,8,0

// Test Set
UserId,UserName,AppId,Purchased
1,Lianne,9,0
1,Lianne,10,0


Would this work? well it seemed not and it turned out it actually didn't. The model was wrong about predicting on the appId of 6,7,8,9. It thought the user number one would buy them with a slightly high chance. The metrics look like...


TP : 5
FP : 4
FN : 1
Accuracy : 0.5
Auc : NaN
F1Score : NaN
Precision : 0
Negative Precision : 1
Negative Recall : 0.5


To make a proper model, what my test dataset should look like on this sample training data?
My 2 cents:
the number of records in the data set used here is very small. If we have a look into the data set we can see that the target variable split is exactly 50:50 which means the probability is half. Its like flipping a coin to get heads or tail.

The training set contains a known output and the model learns on this data in order to be generalized to other data later on. The dependent variables and the independent variable should be in splatted and  then do a train test fit.

You can use the library from scikit learn as well
from sklearn.model_selection import train_test_split