How safe are helmet- or glasses-mounted mirrors?
I would say that there's not much more concern of the mirror gouging your eye out than your glasses in the event of an accident. Ultimately I think if you did a risk/benefit assessment of having a mirror, you'd come out in the black.
For most cycling mirrors I'd say that the mounting point is going to give way before anything breaks and becomes dangerous. If you're truly concerned, have a look at this question where I tried to put together an exhaustive list of cycling mirror options.