How to use text classification where the training source are txt files in categorized folders? I have 200 *.txt unique files for each folder:



Each file is a lawsuit initial text separated by legal areas (folders) of public advocacy.

I would like to create a training data to predict new lawsuit by its legal area.

Last year, I have tried use PHP-ML, but it consume too much memory, so I would like to migrate to Python.

I started the code, loading each text file in a "json-alike" structure, but I don't know the next steps:

import pandas as pd
from sklearn.feature_extraction.text import CountVectorizer,TfidfVectorizer
from sklearn.base import TransformerMixin
from sklearn.pipeline import Pipeline
import os

path = 'C:\wamp64\www\machine_learning\webroot\iniciais\\'

files = {}

for directory in os.listdir(path):
    if os.path.isdir(path+directory):
        files[directory] = [];
        full_path = path+directory
        for filename in os.listdir(full_path):
                full_filename = path+directory+"\\"+filename
                if full_filename.endswith(".txt"):
                    with open(full_filename, 'r', encoding='cp437') as f:
                        files[directory].append(f.readlines())


Thanks in advance
Assuming that your folders are your classes, you can match any document with the correspondent tag.
Then, for every document:
1.- Normalize the text, i.e. remove stop words (unless they make sense), stem and / or lemmatize it (unless it doesn't makes sense).
2.- Vectorize the documents, you can choose TFIDF, BOW, word embeddings etc
3.- Depending on your documents train with an MLP (in case of BOW) or an LSTM in case of words embeddings.

When you have a new document, you need to repeat the procedure using the vocabulary you created for the training set.

I had a similar use case and was enough to use BOW with a multi layer perceptron, the accuracy was above 95%, but, documents were different for each category and I removed most frequent words because there where to common.

Another solution is to perform a topic modeling on documents binding those topics to the category and then training a simple classifier (an MLP or SVM will work)