that's actually a thing, as we increase context length (working memory/lstm) of ai models, they increasingly become prone to hallucinations and simply making things up. every single claim in any text produced by an llm should be scrutinized as it is very likely incorrect due to it simply being a text prediction algorithm that will always add text to the user's input (not "find" or "create" an answer, unless you consider a remix to be creation, but it surely isn't novelty). unfortunately the responses must be scrutinized externally because ml, in general, is an inscrutable technology, and as such should never be used for any job that should be scrutinized (namely law, healthcare, or finance).
United States