Creating similarity metric with Doc2Vec and additional features
You could think of your similarity measure as a search problem if you consider one record a query, and the "near" records as search results.

I've had some good results following this paper: https://arxiv.org/pdf/1602.01137.pdf

As I understand it, the document vectors used in the paper were only good for improving the relevance of search results for results that were already decent (the top N results).

That to me suggests you might try developing a similarity score that works with your other attributes first, and then do something like a weighted average, where the significance of the doc2vec score decays quickly based on the first metric.