How much more data does my classifier need? I have a dataset of ~100k news articles, and I'm trying to build a classifier based on the headlines and leads of every article. The dataset is not pre-labeled so I'm manually labelling a subset of the articles to train on. So far, I have 9 categories with ~600 labeled articles for each category. I'm using doc2vec to create document vectors and the SVC classifier from sklearn to make predictions.

My cross validation score (with 10 splits and shuffling of data) is hovering around 0.89.

When I plot a learning curve for the training set and test set, I interpret it as the classifier suffering from high variance, and that I need to collect more data. But is there any way I can approximate how much more data is needed to get a cross validation score of say, 0.95?

Here is my learning curve and the mean scores:



Training scores: 1.0, 0.99919679, 0.9991984 , 0.99791667, 0.99615385, 0.99550155, 0.99487179.

Testing scores: 0.40769912, 0.72283119, 0.78529511, 0.83461723, 0.85527486, 0.86151579, 0.86471912.

EDIT: I performed a grid search on the C and gamma parameters to the SVC model, and adjusted gamma to 0.015 which made the two lines converge a bit more. Adding plot and new scores.



New training scores: 0.97758621, 0.98703072, 0.98177172, 0.97540872, 0.96717739, 0.96479244, 0.96241702.

New testing scores: 0.45800254, 0.78917394, 0.83526468, 0.853326, 0.86849859, 0.87232875, 0.87508134.
I think you already have enough data. Your problem seems to be one of generalization, which is in general a tricky one to solve.
For SVM, depending on what kind you have, you could try tinkering with the parameters and kernels.
Have a look here. -- https://stats.stackexchange.com/questions/35276/svm-overfitting-curse-of-dimensionality?utm_medium=organic&amp;utm_source=google_rich_qa&amp;utm_campaign=google_rich_qa