Sadly, LLMs are trained with an autotrgressive objective, so their ability to measure epistemic/aleatoric uncertainty is absent. Maybe a mixture of experts could get them uncertainty measurements and then properly generate "I Don't know." Maybe a complete new LLM model variants will come out (e.g. approximate Bayesian inference or GFlowNets), albeit scaling laws are not seen in these approaches.