Word embeddings and punctuation symbols I have a decent understanding of word embeddings (at its core, one can think of a word being converted into a vector of, say, 100 dimensions, and each dimension given a particular value... this allows to do math with the words, also it makes the training sets to be non-sparse...)

But today something came to my mind, what about punctuation symbols such as , . () ? ! ... ?

They do have a huge impact on the meaning of sentences and, like words, the position and context in which they are used is relevant.

So the question is, how should this be modeled? are pretrained sets like GloVe including punctuation symbols? should I simply remove punctuation symbols from text?
Punctuation such as , . () ? ! ... ? are all included in pretrained word vectors such as Glove. However, it is common to find phrases, abbreviations, and misspellings that are not contained within pretrained vectors. What to do with those phrases is highly dependent on the goals of the analysis and context of the text.