Why do language models place less importance on punctuation?
Language models almost always map every word to an embedding. There are many embedding algorithms with most of them having interpolation properties i.e. If $E(word)$ represents embedding of a word then $E(king)-E(male)+E(female) \sim E(queen)$. The smoother the interpolation properties the better the model understands the word, these properties don't exactly make much sense when it comes to delimiters.

Yet, there are instances where a delimiter embedding is learned ( always has an embedding ). While using these first all punctuation in the text is converted to one specific word, say 'dlmt', the embedding algorithm learns an embedding for this word treating it as if it would have any word. This maintains interpolation properties where the delimiter is understood to be a word that is used to split context.

I have observed that delimiters such as question mark or exclamation marks at the end of sentence are also understood to be breaks in context, in these cases the model learns if the statement is a question or so just by the context given by the words and stops in the sentence