MMRole: A Comprehensive Framework for Developing and Evaluating Multimodal Role-Playing Agents
Multimodal Role-Playing Agents;Large Multimodal Models
Yes
No ethics review needed.
* The abstract and introduction highlights the "specialized MRPA" idea. Do we know much improvement comes from the specialized reward model vs. no specialized reward model?
* Do the authors have any insight on the results generated by a finetuned MM role playing model? What works, what doesn't work, and what works better/worse than just prompting gpt?
* Simple, straightforward method that clearly works well given the model size and achieves the desired outcome.
* Create a specialized Multimodal Role-Playing Agent is a novel idea.
* Experiments demonstrate good performance given the finetuned model size.
* Comprehensive evaluation.
This paper proposes a multimodal role-playing agent data collection and training framework. The authors use a wide range of images with different prompts to prompt GPT for image-text role-playing data, and fine-tune a QWen-VL-Chat model on the dataset after some automatic filtering.
* The major technical contribution seems to come from the MM roles dataset collection process. However, there does not seem to be much data curation beyond automated filtering.
* Analysis seems to be mostly numbers and high-level results, with little technical/detailed insight.
Yes
No ethics review needed.
no questions
The paper is well-written and easy to follow.
The evaluation framework is highly relevant and potentially very impactful. The evaluation metrics are meaningful and the SOTA evaluation itself is comprehensive, providing a relevant set of baselines for future users of the dataset/framework.
The paper presents a new dataset and evaluation framework for multimodal role-playing agents. They present several complementary evaluation metrics.
The authors evaluate several recent general purpose multimodal LLMs within this framework. In addition they evaluate a specialized model fine-tuned on their dataset.
I am not sure whether I could fully follow the approach to evaluation.
Imo it would be important to run a (at least limited) evaluation with human participants scoring the output. Building models that automatically evaluate outputs seems to be a circular approach.

Furthermore, evaluting the MAE to compare between different evaluators might not adequately model differences between evaluators that are not visible in MAE.
Yes
No ethics review needed.
Why are only 320 samples used as the validation set out of 23,520 samples, with the remainder used for training?
1. The paper constructs a complete multimodal dataset (MMRole-Data) and evaluation framework (MMRole-Eval).
2. Testing across multiple LMMs lends credibility to the experimental results.
This paper introduces the concept of Multimodal Role-Playing Agents (MRPAs), develops a multimodal dataset (MMRole-Data) and evaluation framework (MMRole-Eval), and creates a specialized MRPA model, MMRole-Agent, achieving improved multimodal understanding and role consistency.
1.Overreliance on GPT-4 for Evaluation: While MMRole-Eval provides a stable evaluation mechanism, it heavily relies on GPT-4, introducing a degree of bias. The authors validated the MAE between GPT-4 (humans) and Reward Model (humans), demonstrating consistency between the reward model and GPT-4. However, comparing two MRPAs to see which performs better does not substitute for genuine human judgment on the quality of MRPA responses. While this reward model may help MMRole-Agent approach GPT-4’s performance, its potential to surpass GPT-4 or elevate MRPAs to human-level capabilities remains debatable.
2.Lack of Performance Comparison with Single-Modality RPAs: Although the concept of MRPAs is appealing, the absence of specific experimental comparisons makes it difficult to understand exactly where MRPAs improve upon performance or accomplish tasks that single-modality RPAs cannot achieve.
Yes
No ethics review needed.
1. Could you provide specific cases to analyze MMRole-Agent’s performance under In-Test and Out-Test conditions? 

2. Could you explain the "rigorous manual quality control" process in character profile generation？

3. Has the sensitivity of MMRole-Agent to different prompt templates been tested?

4. Could you discuss the primary limitations of MMRole-Agent, especially in terms of challenges encountered in practical applications and possible directions for future improvements?
1. This paper first introduces the concept of Multimodal Role-Playing Agents (MRPAs), extending traditional role-playing agents to the multimodal domain, filling a gap in existing research.

2. The MMRole framework includes both the construction of a dataset (MMRole-Data) and the design of an evaluation method (MMRole-Eval), covering eight metrics across dimensions such as fundamental conversational skills, multimodal understanding, and role-playing quality.

3. The proposed MMRole-Agent demonstrates strong performance.

4. The writing is good and easy to understand.
This paper introduces a concept of Multimodal Role-Playing Agents (MRPAs), expanding traditional role-playing agents to tackle multimodal interactions. The paper introduce a framework (MMRole) including MMRole-Data and MMRole-Eval. The MMRole-Data is a large-scale, high-quality dataset with 85 characters, 11,000+ images, and 14,000 dialogues. The MMRole-Eval is a robust evaluation method with eight metrics across three dimensions: conversational skills, multimodal understanding, and role-playing qualities.
1. The paper lacks case studies, which could help illustrate MMRole-Agent's performance across diverse roles and dialogue scenarios.

2. The paper mentions that character profiles undergo "rigorous manual quality control," it does not provide detailed quality control standards or processes.
We propose MMRole, a comprehensive framework for developing and evaluating Multimodal Role-Playing Agents, comprising a personalized multimodal dataset and a robust evaluation method.
@inproceedings{
anonymous2024mmrole,
title={{MMR}ole: A Comprehensive Framework for Developing and Evaluating Multimodal Role-Playing Agents},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=FGSgsefE0Y},
note={under review}
}
Recently, Role-Playing Agents (RPAs) have garnered increasing attention for their potential to deliver emotional value and facilitate sociological research.
However, existing studies are primarily confined to the textual modality, unable to simulate humans' multimodal perceptual capabilities.
To bridge this gap, we introduce the concept of Multimodal Role-Playing Agents (MRPAs), and propose a comprehensive framework, MMRole, for their development and evaluation, which comprises a personalized multimodal dataset and a robust evaluation method.
Specifically, we construct a large-scale, high-quality dataset, MMRole-Data, consisting of 85 characters, 11K images, and 14K single or multi-turn dialogues.
Additionally, we present a robust evaluation method, MMRole-Eval, encompassing eight metrics across three dimensions,
where a reward model is trained to score MRPAs with the constructed ground-truth data for comparison.
Moreover, we develop the first specialized MRPA, MMRole-Agent.
Extensive evaluation results demonstrate the improved performance of MMRole-Agent and highlight the primary challenges in developing MRPAs, emphasizing the need for enhanced multimodal understanding and role-playing consistency.
The data, code, and models will all be available.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Multimodal Role-Playing Agents Large Multimodal Models
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/eca59bba5e199bc5a651cbd343c66edbd8c0ced6.pdf
datasets and benchmarks
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/445fd4a36fd828fb9dd5902de9d2e22f05e076be.zip
MMRole: A Comprehensive Framework for Developing and Evaluating Multimodal Role-Playing Agents
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission