Doc2vec '-' symbol occurrence Currently working on resume parser and struggled with embedding words with '-' symbols in them. Such as 'IT-manager'.

Vector representations of these words are incorrectly classified by doc2vec.


  ['it-manager']
  [('salary', 0.23328335583209991), ('responsibilites', 0.22327110171318054), ('schedule', 0.14869527518749237), ('position', 0.12755176424980164)]


But when I remove '-' symbol, it is tokenized and classified right.


  ['it', 'manager']
  [('position', 0.9306046962738037), ('schedule', 0.6630333662033081), ('responsibilites', 0.6081600189208984), ('salary', 0.5934453010559082)]


How do you work with such data properly? For this kind of task, I guess, it is better to exclude the symbol. But there may be a way to tell Doc2vec to treat these words like two different ones. Or perhaps tell the word_tokenizer to tokenize them in this fashion?
gensim's Phrases module may also be helpful:

from gensim.models import Phrases
documents = [
    "the mayor of new york was there",
    "machine learning can be useful sometimes",
    "new york mayor was present"
]

sentence_stream = [doc.split(" ") for doc in documents]

bigram = Phrases(sentence_stream, min_count=1, threshold=2)

sent = [u'the', u'mayor', u'of', u'new', u'york', u'was', u'there']
print(bigram[sent])
# Expected output:
# [u'the', u'mayor', u'of', u'new_york', u'was', u'there']


That code is from this other answer (I've copy-pasted it above for convenience).

For more on the Phrases module, check this page out.