VibeCheck: Discover and Quantify Qualitative Differences in Large Language Models
large language models;evaluation
Yes
No ethics review needed.
Your definition of vibes is similar to how concept axes are defined in interpretability, have you tried connecting the two?

Why discover vibes and validate on gold labels when you can actually use the gold labels directly?

Have you considered the impact of the prompts you are using on eliciting specific model vibes? 

Figure 1: It is not clear from the example provided for “What is a vibe?” that Output A differs from Output B in the friendliness direction (formal → friendly). Which answer is more formal and which answer is more friendly? Also why Vi() takes values of 1 and -1 when the prompt for the judge is asking the model to answer with A, B or equal? Figure 1 caption needs more explanation.

Table 4 - is the top table for overall results? Please indicate what the abbreviations mean - Model Matching (MM), Preference Prediction (PP)

Line 336 - typo “sensitive”
The paper is making an important observation that current LLM evaluations are mainly focusing on one textual dimension (for eg., correctness) at the expense of other aspects that matter for the end user, failing to capture the open-ended nature of LLM applications, nuanced qualitative aspects and their dependence on subjective user preferences; LLM evaluation is indeed an open and important research topic

The authors propose a definition of vibes and discuss what makes a vide valid accounting for aspects such as well-defined, differentiating, then validate the identified vibes against human gold labels and aim to quantify the differences between popular LLM models using their method
The paper’s goal is to discover LLM vibes, i.e. traits of a model which are well-defined, differentiating and user-aligned. The main contribution of the paper is a method (VibeCheck) for quantifying vibes of LLM models. The method first employs a LLM model to extract a set of vibes, then decide which model outputs are conforming to that respective vibe based on a panel of LLM judges. The authors validate to what extent their method aligns with human responses and user preferences in the context of three different tasks: text summarization, math problem solving and image captioning. This approach allows them to quantify vibes of popular LLM models, finding that Llama3 outputs tend to be more friendly in nature compared to outputs from GPT-4 and Claude, while the latter two models tend to be more formal in nature.
“Vibes” are subjective and not a well-defined concept; the paper defines vibes as identifying traits of LLM models and in contrast to concrete evaluation measures which measure objective aspects of the text, evaluating subjectivity is less clear and straightforward

The authors consider a vibe to be well-defined if multiple LLM judges agree, however it is possible that those LLM judges all have the same limitations; it is also unclear to what extent LLM judges decisions on what constitutes a valid vibe aligns with humans

The paper acknowledges that vibes are user and task dependent, however there is no accounting for these aspects in the experiments

The paper would benefit from more structure, more in-depth analysis and more detailed figure/table captions. The extra space could be used for including a discussion section to summarize all the findings and takeaways.
Yes
No ethics review needed.
- Sec 3: it could strengthen your claim of selecting those three evaluation criteria based on solid literature work. At least, it could be better to provide why these three criteria are important in measuring such qualitative behaviors of LLMs. 
- L220-225: what are the criteria for those threshold setups (0.2, 0.05)? 
- Please provide a better visualization of Table 3. It is hard to interpret the scores and their relationship with llama. 
- Appendix B: as mentioned these are direct quotes from the original HC3 paper. Please put a citation of this paper; otherwise, it could be regarded as flagging behavior for ethics review. 
- Please combine the main text and appendix into one.
- This work provides a novel ground to assess the subjective characteristics of LLM outputs that usually differ by task purposes. Not much work has focused on this angle of LLM evaluation. 
- The paper attempted to provide a comprehensive framework to develop an automatic detection of "vibes" in different LLM outputs and how aligned these outputs are with human preferences.
The paper introduces VibeCheck, an evaluation pipeline to identify and compare the qualitative characteristics ("vibes") of LLMs. They employed three stages - (1) vibe discovery using thoroughly designed prompts; (2) vibe validation; and (3) vibe iterations. They also propose a method to quantify the usefulness of VibeCheck by calculating several metrics such as agreement, model matching accuracy, and preference prediction scores. Then, the paper tested VibeCheck across various tasks such as summarization, math, and image captioning, and it revealed how different models exhibit unique behaviors across different domains.
- Overall, the paper is hard to read and understand. A large amount of effort is needed to restructure the organization of the paper.
- Too many typos and grammatical errors are found in both the paper and the appendix. Needs to be corrected. 
- Results seem too weak to claim the usefulness of VibeCheck. For example, in Table 4, VibeCheck (even 3 runs) does not perform better than the pre-defined baseline, increasing doubt about whether this framework does better than a naive GPT4o model.
Yes
No ethics review needed.
In the vibe iteration process, how many more vibes were found?
1. The problem this paper addresses is very important and the approach is creative. 
2. Table 1 shows the utility of the paper, it shows how similar the vibes are that VibeCheck finds to the ones that humans find. 
3. The authors conduct extensive experiments to show how it performs on different datasets. 
4. The paper is well-written and easy to follow.
LLM-generated responses are mostly evaluated on their correctness, dismissing other important dimensions such as its tone or other subjective dimensions. 

Distinctive traits of an LLM are called _vibes_ in this paper. The paper introduces an approach, VibeCheck, to finding and measuring such vibes. It consists of an LLM that qualitatively compares a pair of models to find distinctive vibes between the two. It identifies which dimensions the generated outputs of the models differ on to get a set of vibes. LLM judges are then used to determine where the model's output falls on the vibe.  

This is evaluated on three fronts: 
1. well-defined (agreement among multiple users)
2. differentiating (ability. to distinguish between LLMs) 
3. user-aligned (predictive of user preferences) 

The authors conduct multiple experiments to evaluate the method.
1. A batch is used to find the vibes, however, there is not much discussion about strategies to select this batch. If the batch contains similar prompts and is low in diversity, then not all vibes might be considered. E.g. for vibes on STEM prompts to work, it will need to see something similar in the vibe batch, however, it is not ensured that this will be the case. 
2. The LLMs used are still on the larger side (GPT4o-mini).
Yes
No ethics review needed.
1) Could you inspect the cases that, despite all these vibes being generally predictive, the model matching prediction or the preference predictions are wrong? Could you try to “VibeCheck” those cases?

2) In the CNN/DailyMail experiment, the preference prediction accuracy is as low as 61.42%, which is not a whole lot better than random. Why should I still believe, then, that the vibes discovered in this case are “real” and actually useful?
This paper is well-motivated, as it basically provides an automatic way of conducting the “vibe-based” evaluation which is very commonly practiced nowadays, as the evaluation of LLMs have become difficult. The writing of the paper is very clear and easy to follow. The results presented are overall convincing. Also, it is very nice to review some HCI literature in the paper.
This paper aims to automate the vibes-based evaluation that is common among LLM practitioners. They first define three characteristics of a useful vibe and then use an LLM-aided process to summarize model output into different axes of vibes and iteratively refine them with model misclassification based on these models. They then conduct validation using both a data with gold answer as well as open-ended metrics. Afterwards, they apply VibeCheck to tasks including summarization, mathematics and image captioning.
1) the author did not engage with the line of work that tries to characterize human preference into high-level principles (e.g. https://arxiv.org/abs/2406.06560, https://aclanthology.org/2024.acl-long.99/). It would be much appreciated if the authors could situate their work around these and articulate their contributions more clearly. 
***
2) In Table 4, we see that the predefined vibes get to almost the same preference as VibeCheck. Thus, I am not entirely convinced whether we would need to discover these vibe aspects at run time, as something that could be done a-priori is already quite good, also given the compute overhead as mentioned? Perhaps you could convince me if you show that individual user considers drastically different vibes or the vibes considered for different application are extremely different.
***
3) More analysis on model matching and preference predicting accuracy would be great – why are these results so drastically different for each task? Does this also point to a limitation of the applicability of your proposed method, the fact that the "explained variance" of human preference by vibes are so different across tasks?
***
4) I think it is important to note somewhere in the paper that these vibes (and the way you conduct experiment) are post-hoc explanations that are correlated with user preference. However, they are not necessarily the reason that a user actually prefers one option over the other in real time. In other words, one limitation is that your study lacks ecological validity. However, I think this is ok in this context, as long as you acknowledge it. Of course, if you could show that these vibes are also things the users actually consider when they indicate their preference real-time, I will also be convinced.
@inproceedings{
anonymous2024vibecheck,
title={VibeCheck: Discover and Quantify Qualitative Differences in Large Language Models},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=acxHV6werE},
note={under review}
}
Large language models (LLMs) often exhibit subtle yet distinctive characteristics in their outputs that users intuitively recognize, but struggle to quantify. These "vibes" - such as tone, formatting, or writing style - frequently influence user preferences, yet traditional evaluations focus primarily on comparing models based on correctness.
We introduce VibeCheck, a system for automatically comparing a pair of LLMs by discovering identifying traits of a model ("vibes") which are well-defined, differentiating, and user-aligned. VibeCheck employs an iterative approach which utilizes a panel of LLM judges to discover vibes from model outputs and quantitatively measure the utility of a vibe across these 3 criteria. 
We validate that the vibes generated by VibeCheck align with those found in human discovery and run VibeCheck on pairwise preference data from real-world user conversations with llama-3-70b VS GPT-4. VibeCheck reveals that Llama has a friendly, funny, and somewhat controversial vibe. These vibes can be used to predict human preference with 65\% accuracy and model identity with 80\% accuracy. Lastly, we run VibeCheck on a variety of models and tasks including summarization, math, and captioning to provide further insight into differences in model behavior.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
large language models evaluation
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/4957951b941ad53382132a38ccf06d5396a5bacf.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/8ea6ab2e2e902e5c627fb7dacdd71c98a8f06b27.pdf
VibeCheck: Discover and Quantify Qualitative Differences in Large Language Models
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission