Purpose of test data in binary classification I have a highly biased training dataset where AppId 6,7,8,9,10 are almost never purchased. I made this up just to see how good my comprehension of calculating the classification metrics is such as Accuracy, F1Score, and etc.
UserId,UserName,AppId,Label
1,Lin,1,1
1,Lin,2,1
1,Lin,3,1
1,Lin,4,1
1,Lin,5,1
1,Lin,6,0
1,Lin,7,0
1,Lin,8,0
1,Lin,9,0
1,Lin,10,0
2,Tim,1,1
2,Tim,2,1
2,Tim,3,1
2,Tim,4,1
2,Tim,5,1
2,Tim,6,0
2,Tim,7,0
2,Tim,8,0
2,Tim,9,0
2,Tim,10,0
3,Kirstin,1,1
3,Kirstin,2,1
3,Kirstin,3,1
3,Kirstin,4,1
3,Kirstin,5,1
3,Kirstin,6,0
3,Kirstin,7,0
3,Kirstin,8,0
3,Kirstin,9,0
3,Kirstin,10,0
4,Michael,1,1
4,Michael,2,1
4,Michael,3,1
4,Michael,4,1
4,Michael,5,1
4,Michael,6,0
4,Michael,7,0
4,Michael,8,0
4,Michael,9,0
4,Michael,10,1
5,Timmy,1,0
5,Timmy,2,0
5,Timmy,3,0
5,Timmy,4,1
5,Timmy,5,1
5,Timmy,6,0
5,Timmy,7,1
5,Timmy,8,1
5,Timmy,9,1
5,Timmy,10,0
6,Lick,1,1
6,Lick,2,0
6,Lick,3,0
6,Lick,4,0
6,Lick,5,1
6,Lick,6,1
6,Lick,7,0
6,Lick,8,0
6,Lick,9,1
6,Lick,10,0
7,Shawn,1,1
7,Shawn,2,0
7,Shawn,3,1
7,Shawn,4,0
7,Shawn,5,1
7,Shawn,6,0
7,Shawn,7,0
7,Shawn,8,0
7,Shawn,9,1
7,Shawn,10,1

To test the trained model, I used this test set.
UserId,UserName,AppId,Label
8,Lianne,1,1
8,Lianne,2,0
8,Lianne,3,1
8,Lianne,4,0
8,Lianne,5,0
8,Lianne,6,0
8,Lianne,7,0
8,Lianne,8,0
8,Lianne,9,1
8,Lianne,10,0

Following is the prediction results.
8,Lianne,1,1  =&gt; Predicted true
8,Lianne,2,0  =&gt; Predicted false
8,Lianne,3,1  =&gt; Predicted true
8,Lianne,4,0  =&gt; Predicted true
8,Lianne,5,0  =&gt; Predicted true
8,Lianne,6,0  =&gt; Predicted false
8,Lianne,7,0  =&gt; Predicted false
8,Lianne,8,0  =&gt; Predicted false
8,Lianne,9,1  =&gt; Predicted false
8,Lianne,10,0 =&gt; Predicted false

// Sum this up...
TP = 2
TN = 5
FP = 2
FN = 1

Accuracy = 2 + 5 / 2 + 5 + 2 + 1 =&gt; 7 / 10 =&gt; 0.7
Recall = 2 / 2 + 1 =&gt; 0.66
Precision = 2 / 2 + 2 =&gt; 0.5
F1Score = 0.5 * 0.66 / 0.5 + 0.66 * 2 =  0.33 / 1.16 * 2 = 0.5689

The calculation was proven alright and as I expected, the model is right on almost all cases except the app id 2. While I'm doing this, this idea comes into my mind.

&quot; What test data is for? &quot;
&quot; Is it for checking out the desired model metrics like above? &quot;
&quot; Why bother with some proportion of data to perform a test? when you just need a certain, small group of dataset to evalute the model? &quot;
&quot; Even if I have a thousands of million rows of training dataset, wouldn't it be enough to have a complete small, unique set of dataset that covers the whole variations? do I still need some proportion of testdata from the training set? &quot;

Well, if I already know that the app id range is from 1 to 10, isn't it fairly enough to just prepare only 10 rows of test set as long as it has app id 1 to 10? even when the training data set is gigantic?
Why do I need to care about N% of test data set from the training data set?
One of the well known problems of machine learning is overfitting. The main reason for having a test data set is to 


Obtain a realistic measure of how good your model is doing.
Prevent overfiting the validation set.


Now think about the second point, quite often it might occur that you tune up the hyper parameters so that your validation set provides a good measure, but this means you might be overfitting your validation set and when you deploy your shiny model in the real world, boom! poor results are returned.

As per the amount of data to reserve for testing, it depends on how big is your data set, normally the rule of 60% for training, 20% for validation and 20% for test has been used, but if your data set has 5 millions of points, you probably just need 1% for testing (as that will provide you with 5000 examples for testing anyway).

So, yes, your last question makes sense, you will probably be fine taking a small portion of the original data set (again, say 1%) in order to get good results.