NLP Text Summarization - which metrics to use in evaluation?
So, your question is talking about whether human reference summaries are required to evaluate summarisation models. 

The short answer is yes at the moment. The most important things about an output summary that we need to assess are the following:


The fluency of the output text itself (related to the language model aspect of a summarisation model)
The coherence of the summary and how it reflects the longer input text.


The problem with have an automatic evaluation system for a text summarisation model is that, although we can assess fluency from a language model, we can't really assess whether the model has pulled "the most salient" pieces of information from the original, longer text (and this can subjective from person to person). Hence why we need multiple human reference summaries to compute ROUGE and BLEU. However, as you are aware, these metrics have their limitations. 

ROUGE is essentially a further development of BLEU, which have been commonly used a dubious proxy for output text fluency in research to compare summarisation and translation models. These metrics are dubious because they simply look at how much they overlap with reference texts from humans (https://rxnlp.com/how-rouge-works-for-evaluation-of-summarization-tasks/#.Xt1ewy-ZOi4).