NLP: Are hashtags tokenised? I am exploring a potential NLP project and was wondering what generally is done with the hashtags words (e.g. #hello). Are those words ignored? is the # removed and the word tokenised? is it tokenised with the #?
Thank you
Word including Hash tag need to be tokanised. It has a special meaning and given a context. The word by itself has a different meaning.