Handle 50,000 classes in OneVsRestClassifier I'm new to data science and NLP. I'm trying to solve a problem that is having 1 million rows and some 50,000 distinct classes. The dataset has some text column as a predictor and the other one is the multilabel responses. I have been using tfidf to represent the text fields and MultiLabelBinarizer to transform the labels. But MultiLabelBinarizer is giving MemoryError.

And there is no way I can pass the legacy multi-label data representation using a sequence of sequences as it seems no more to be supported in the sklearn package. So, what should be my approach?

Any help is appreciated. Thanks in advance.
As you already know, the main culprit here is the large number of classes (50k). For every sample data, you have a label of size 50k. Even if a sample only belongs to a single class, the label will still be of size 50k.

For example, sample1 has a label of A and B while sample2 has a label of A. Sample3 on the other hand has C for its label and sample4 has A. We can visualize it like this:

sample1 - [1, 1, 0, 0, 0, 0, ..., 0] 
sample2 - [1, 0, 0, 0, 0, 0, ..., 0] 
sample3 - [0, 0, 1, 0, 0, 0, ..., 0] 
sample4 - [1, 0, 0, 0, 0, 0, ..., 0]


One solution here is to use Siamese neural network. This takes in two input at a time. Instead of training directly to learn classes as the output, we are training to learn the similarity between two samples. We first select random pairs. We use the label 1 if two samples belong to the same class; else 0.

sample1 &amp; sample2 - 1
sample1 &amp; sample3 - 0
sample2 &amp; sample4 - 1
sample3 &amp; sample4 - 0


This way, even if there is a lot of classes, you are only training with 2 labels, 1 or 0, thus, avoiding memory error.

The problem now is that you are doing a multi-classification task. For instance, sample1 and sample 2 are the same class A but sample1 also belongs to class B. My suggestion, but I'm not totally sure here, is to add multiple instance of that pair.

sample1 &amp; sample2 - 1
sample1 &amp; sample2 - 0
sample1 &amp; sample3 - 0
sample2 &amp; sample4 - 1
sample3 &amp; sample4 - 0


As for the large number of training samples (1 million), this can be handled by using small batches during training.