Feature extraction from pure text I have a dataset (~52k rows) with a column containing just pure sentences (upper and lowercase, with punctuation and stop words) in each row. What can I do to represent this data in a meaningful way for my model. I tried using a CountVectorizer but it's giving me a dataframe with way too many columns and the model is having a hard time using it. What other options do I have?
That is the real-world! in text analysis you usually end up with a extremely high-dimensional representation. First of all, I always recommend TF-IDF over original BoW (CountVectrizer).
To end up with a low-dimensional space, you may use this naive approach which is not written efficient in computational way but the approach gives you an idea where you need to go. It will be more clear when I add explanations to the notebook soon.