Would keeping all punctuation make any sense in word2vec?
You are correct to keep punctuation if you want to be able to predict it.

Tokenization of your input should actually work for any character, be it a letter or punctation. In fact there have already been exmaples of people modelling mathematics using Word2Vec and then generating very realistic maths, via $\LaTeX$ !  Look at the subsection called Algebraic Geometry in Karpathy's now famous blog post.

There is a good note on the matter here, whcih a specific example given in seq2seq learning (basically translation within the realm of NLP). Be sure to read the comments on the accepted answer there.

to answer your final question, I don't think it would be possible to use your generated model to place to required punctuation back into your model, as something like an LSTM would not have a representation for, say a comma, as it had never seen one.