Multi-Class Text Classification: Doc2Vec performing very bad compared to Hashing Vector
If I read your model correctly, you only performed 5 epochs with the Doc2Vec model.  This is probably not be enough for the network to learn the word embeddings.  Has your loss leveled out after 5 epochs?  Try running it for 50 epochs and see if it makes any difference.  Conceivably, you could need thousands of epochs to achieve a reasonable model.