a lot of people seem to think it's silly to tell an LLM not to hallucinate, or to make sure it hasn't hallucinated.

it is clear to me that they're often able to tell. empirically, but also because it's the kind of thing you should expect to be an easily distinguishable pattern from factual recitation.