Duplicate values on text feature with deep learning e.g LSTM
Depending on how much duplicate you have the effect can be quite transparent for the training. At the extreme if you have too many of them it won't generalize well because your dataset will be artificially large but will lack diversity to generalize well.
Most problematic effect of duplicate is when their labels (or targeted value) are different but their features are identical. This can throw off training and prediction badly if there're too many of them.
I just wonder, what happen if i have dataset with duplicate text value for my LSTM model, it will affect training, or it can make overfit? thanks in advance