How Social is It? A Benchmark for LLMs' Capabilities in Multi-user Multi-turn Social Agent Tasks
LLM;Multi-user;Social agent;Evaluation
Yes
No ethics review needed.
- What is the size of the dataset and how does a typical sample look like?
- What are the broad news topics used for creating this dataset?
- How is the human evaluation experiment performed for this dataset and what are the details of the said experiment?
- Is this dataset going to be made publicly available at any point?
- What is the motivation behind defining the HIIS evaluation metric in the manner laid out in Equation 1?

That being said, I would be convinced by nothing short of significant rewriting of this paper to improve my scores.
**Important Topic**: The authors propose to measure the ‘social intelligence’ of LLMs by putting them in a multi-party conversational scenario. This is a plausible application of LLMs in our society and hence, needs appropriate methods for research and evaluation. Overall, this paper attempts to tackle an important problem.
This paper attempts to introduce a multi-turn, multi-user conversational dataset where the users are conversing in a complex social scenario, that can be used to evaluate the capabilities of LLMs to participate in a complex social scenario. To this end, the authors create the conversational dataset from news articles using a capable LLM (GPT-4) in a step-by-step process. Additionally, they introduce an evaluation metric for measuring the performance of LLMs in this scenario. The authors conduct evaluation experiments on 5 open-source and 1 proprietary LLMs in two settings: simple prompt-based and Chain-of-thought based. Results show that contemporary LLMs lag behind human performance on this dataset.
**Poor Writing & Representation**: Unfortunately, the writing and presentation of this paper is prohibitively sub-standard such that it has been hard to understand the dataset and evaluation metrics contributions being claimed by the authors in the paper. I highly suggest a thorough revision of the content in this paper, for grammatical errors as well as better understanding of the reader.


**Dataset Details**: The paper completely lacks a representative example of the HSII dataset created in this paper. Without this example, it is difficult to understand the structure of the dataset and its value. Table 3 seems to be an example from the dataset but the format is strange and the unique characters involved in this conversational scenario are unclear. Moreover, any other dataset statistics such as the size of the dataset, the topics from news domain used for creating the dataset, distribution of number of unique characters in each sample, length of conversation, comparison to existing multi-party chat datasets etc. is missing from the paper. There are some preliminary analyses of the dataset using clustering and a figure in the Appendix (Figure 5) which is an uninformative way to represent information from clustering. Further, the paper does not mention whether this dataset is/will be publicly available at any point.


**Evaluation Details**: The evaluation metric presented by the paper is similarly hard to understand in the absence of details about the dataset itself. The pipeline in Figure 2b makes some sense. It seems that the responses are being measured along three axes i.e., target selection, response quality and long-term adherence. The mathematical symbols used in Definition 1 are misleading as $n$ is being used to represent dataset size (which is unclear) and then $n_i$ symbols are being used for other purposes throughout the evaluation process.


**Results**: Keeping in mind that the dataset and evaluation metrics are unclear, it is hard to rate the reliability of the results presented in this paper.


**Motivation based Sociology and Dialogue**: This paper iterates the motivation behind these experiments several times throughout the introduction, as being based out of sociology and dialogue literature. However, there isn’t sufficient discussion of this motivation with appropriate details from that literature.
Yes
None
No ethics review needed.
•  Discuss how CSI can improve existing psychological scales designed for humans.

•  Further explore the unintentional biases that language models may have towards everyday concepts as mentioned in section 4.1.

•  On line 424, it might be: “e.g., five positive words, four neutral words and one negative words, and so on.”
•  The CSI can effectively quantify the emotions of models, reflecting the emotional tendencies of LLMs.

•  It effectively reduces the issues of reluctance and inconsistency in traditional psychological scales.

•  The use of representative neutral words in constructing the CSI reduces the potential emotional orientation of the words themselves, better reflecting the internal emotional associations of LLMs.

•  It explores the impact of different language environments on the emotional tendencies of LLMs.
This paper introduces a novel assessment method known as the Core Sentiment Inventory (CSI) for evaluating the emotional tendencies of large language models (LLMs). Inspired by the Implicit Association Test (IAT), the CSI aims to provide a more reliable and effective way to assess the implicit emotional characteristics of LLMs. It addresses the limitations of traditional psychometric methods, such as model reluctance and inconsistency.
•  The constructed CSI is only used to assess the emotions of LLMs and has not been extended to other psychometric fields, such as personality or moral decision-making.

•  It does not introduce the calculation methods for consistency rate and reluctance rate.
Yes
No ethics review needed.
(1) I would like to understand what is the concrete output during each stage of the evaluation dataset construction pipeline. Additionally, what is the concrete output of the each step of the HSII evaluation framework. The social task example in the appendix is incomplete and confusing.

(2) I would like to see what is changed after adding CoT into the social scenario understanding stage, especially the technical details about how the reflection is made.
(1) Unlike previous social simulation work that evaluates conversations along specific dimensions, this study introduces a four-step hierarchical framework for assessing multi-user tasks and social capabilities. Additionally, the design of the social agent goes beyond simple LLM-based conversation; it draws on foundational sociological theories, which decompose social relationships into distinct components for richer interaction modeling.

(2) The HSII-Dataset is grounded in a news dataset, rather than relying solely on general daily-life scenarios. This approach ensures realistic and complex social contexts. Furthermore, clustering analysis of the dataset confirms that it reflects both the similarity and diversity present in the original source news categories, enhancing its relevance.

(3) This work introduces a novel metric, CoT-complexity, that captures the importance of reasoning abilities (often overlooked) in social scenario conversations. CoT-complexity provides a precise measure of social task complexity, focusing on the critical role of reasoning within these interactions.
This work proposes a novel benchmark named as How Social Is It (HSII) to assess LLM's social capabilities in different social agent tasks that are constructed from the news dataset. Typically, with a careful four stage design including format parsing, target selection, target switching conversation, and stable conversation, such a framework can evaluate the communication and task completion ability of LLM in interactive multi-user multi-turn complex social tasks. Moreover, to dive deeper into the CoT method on enhancing LLM's social performance, this work proposes CoT-complexity to quantify the efficiency of CoT to conduct social tasks.
(1) The dataset construction pipeline includes several elements that are not explained clearly, which makes understanding them difficult. For example, the concept of "background conversation" is mentioned twice within the pipeline but lacks a concrete definition, leaving its purpose and role ambiguous. Additionally, there are no clear examples showing the difference between a "golden response" and a generated response. This absence of examples makes it challenging to grasp what specifically distinguishes a high-quality, or “golden,” response from those generated by the models. Furthermore, details on how the agents were collected and how their profiles and conversations were generated are missing. These are critical aspects that need more clarity to fully understand the dataset’s construction process and its assumptions. Moreover, the figures included in the paper are too small and difficult to read, making it harder to visually interpret the data and results.

(2) The comparison of model performance lacks depth, as the results only indicate that Llama3-8b outperforms the other two models on metrics like R3, R4, and HSII, without providing any analysis as to why these differences exist. There’s no discussion of what specific dimensions of the models' abilities might account for Llama3-8b's superior performance, particularly on the HSII score, which measures social intelligence. This leaves open questions about what it means for a model to excel in one dimension versus another. Including case studies that compare model-generated responses with the golden responses would offer valuable insight into how these models differ and what these differences imply in practical terms.

(3) The paper does not include human-based evaluations to verify the reliability of the HSII metric. Without assessing whether HSII scores correlate strongly with human judgment, it’s challenging to assert that HSII accurately measures social intelligence. Adding a component that validates HSII through human analysis would lend greater credibility to claims about its accuracy and its suitability as an evaluation metric in this context.
A benchmark for agent capabilities in multi-user multi-turn complex social interaction based on sociology and  LLM study with novel dataset, pipeline and metrics.
@inproceedings{
anonymous2024how,
title={How Social is It? A Benchmark for {LLM}s' Capabilities in Multi-user Multi-turn Social Agent Tasks},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=RuwAMoFxzG},
note={under review}
}
Expanding the application of large language models (LLMs) to societal life, instead of primary function only as auxiliary assistants to communicate with only one person at a time, necessitates LLMs' capabilities to independently play roles in multi-user, multi-turn social agent tasks within complex social settings. However, currently the capability has not been systematically measured with available benchmarks. To address this gap, we first introduce an agent task leveling framework grounded in sociological principles. Concurrently, we propose a novel benchmark, How Social Is It (we call it HSII below), designed to assess LLM's social capabilities in comprehensive social agents tasks and benchmark representative models. HSII comprises four stages: format parsing, target selection, target switching conversation, and stable conversation, which collectively evaluate the communication and task completion capabilities of LLMs within realistic social interaction scenarios dataset, HSII-Dataset. The dataset is derived step by step from news dataset. We perform an ablation study by doing clustering to the dataset. Additionally, we investigate the impact of chain of thought (COT) method on enhancing LLMs' social performance. Since COT cost more computation, we further introduce a new statistical metric, COT-complexity, to quantify the efficiency of certain LLMs with COTs for specific social tasks and strike a better trade-off between measurement of correctness and efficiency. Various results of our experiments demonstrate that our benchmark is well-suited for evaluating social skills in LLMs.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
LLM Multi-user Social agent Evaluation
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/55035b9896fc64d06a32e950923de356642b8a37.pdf
datasets and benchmarks
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
How Social is It? A Benchmark for LLMs' Capabilities in Multi-user Multi-turn Social Agent Tasks
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission