How does Doc2Vec treat numerical data which is a part of text data?
If you are passing in the numbers as a string type Doc2Vec should just accept these as part of the vocabulary without an issue. Have you tried running the Doc2Vec training with the numbers still included? 

One way to solve the issue if you can't just pass the values in as is is to give them a unique identifier (e.g. &lt; NUM >). It shouldn't matter what the value is as Doc2Vec should in theory not be able to draw any information (and really shouldn't) from that. Instead if you teach it when a number occurs in a document (by using the identifier) that might improve accuracy.