Choosing the size of Character Embedding for Language Generation models
There is a theoretical lower bound for embedding dimension

I would urge you to read this paper, but the gist of it is dimension could be chosen based on corpus statistics

GLOVE paper discussed embedding, check page 7 for graphs. What I want to say with this reference is that you can treat it as hyperparameter and find your optimal value.

EDIT: Here is my personal/borrowed from google rule of thumb. Embedding vector dimension should be the 4th root of the number of categories is start with that, and then I play around it. Read this toward the end when they explain their embedding. Why COULD (it must not) make sence: What is BOW rather than one hot encoding of your n-grams.

Does it make sence to make it larger? it depends. On one hand you are right if we make it too big we loose the distributed representation property of the word embedding matrix, on the other hand it works in praxis.