Personalized Visual Instruction Tuning
large language model;multimodal large language model
Yes
No ethics review needed.
1. Does it work well on non-human objects? 
2. For the MLLM used for comparisons, what are the settings? For example, some models are designed for one image input, how do you use it in multi-image input setting?
1. The proposed Personalized Visual Instruction Tuning (PVIT) MLLMs conduct personalized conversation based on a general MLLM for personalized context without further tuning during test time.
2. The benchmark P-Bench of around 1500 samples could be used on relevant tasks.
3. Good qualitative and quantitative results.
4. The proposal method is easy to follow and the presentation is clear.
This paper tackles personalization issue in MLLM. Current MLLM struggle to recognize and engage with specific people in images, making them less useful for personal AI assistants. The authors propose Personalized Visual Instruction Tuning (PVIT), which lets MLLMs handle personalized conversations by introducing people as a multimodal prefix without needing extra training. They created a streamlined data pipeline to automatically generate large-scale, personalized datasets and a new benchmark, P-Bench, to measure personalization abilities.
1. In other work like Yo'llava, the personalization including various types of subjects, while in this work, it is limited to human face, which is different to the claim in the paper such as in the introduction.
2. For evaluation comparisons, many actual SOTA MLLMs are not included such as GPT-4. I tried those models and found out they perform pretty good in the task. For example, for GPT4o are correct on several examples in Figure 2. 
3. No failure cases analysis. I do not think such method could resolve personalization MLLM perfectly, but it lacks the detailed analysis.
4. Some proposed mechanisms are already commonly used in previous methods. For example, Personalized Wrapper Tokens basically denote the added information with special token.
Yes
No ethics review needed.
1.  Line 277: More advanced knowledge, such as character, hobbies, and professions can be easily
extended and will be considered in future work. -> About How? Like character would be quite abstract to describe.
2. How about the temporal changes, like inputting a child's image while asking for the same person with quite different ages?
3. How about the twins with similar names and similar faces?
4. What happened to the situations where two different with similar faces yet different names or similar names yet different faces?
1. PVIT presents a novel approach to MLLM personalization by leveraging in-context learning capabilities, avoiding the need for additional training for each individual.
2. A benchmark is proposed for future research on this direction.
3. The paper is well-written and clearly explains the PVIT framework, data generation process, and experimental setup.
This paper introduces Personalized Visual Instruction Tuning (PVIT), a framework that enables MLLM to engage in personalized conversations with specific individuals. PVIT addresses the “face blindness” limitation of MLLMs by training them to recognize individuals within images and generate contextually accurate responses.
1. Some typos like Line 191-192 Two special tokens are mentioned, however, only one is described
2. The current implementation focuses primarily on names and faces.
3. The paper primarily focuses on individuals present in the training data. Some investigation on the zero-shot ability would help to understand this framework's ability.
4. This problem is an interesting setting, but the proposed methods is not that novel.
5. Current experiments are not that enough, more experiments as described in questions are welcome.
Yes
The origin or sourcing of human images used for training and testing should be claimed in this work.
Yes, Responsible research practice (e.g., human subjects, data release)
1. Which version of LLaVA is finetuned as P-LLaVA?

2. What models are used in dual-level textual information extraction and fusion?
1. PVIT’s multi-phase pipeline is designed to generate personalized training data. By integrating visual expert models, image generation tools, and large language models, the authors ensure a rich and diverse dataset that enhances the personalization capabilities of MLLMs.

2. The authors contribute P-Bench, a novel benchmark designed to evaluate personalization capabilities in MLLMs.

3. Experiments demonstrate that PVIT improves MLLMs’ accuracy in handling personalized queries across various tasks. The ablation studies further validate the impact of component designs in the pipeline, highlighting the framework’s effectiveness.
This paper introduces Personalized Visual Instruction Tuning (PVIT), a novel framework that enables multimodal large language models (MLLMs) to engage in personalized, coherent dialogues tailored to specific individuals within images. The PVIT framework addresses the limitations of existing MLLMs, which often fail to recognize individuals and deliver contextually accurate, personalized responses. The paper proposes a comprehensive pipeline for personalized multimodal data construction. The experiments demonstrate that models fine-tuned with PVIT show substantial improvements in personalization tasks.
1. Reliance on Pretrained MLLMs and Potential Hallucinations: The pipeline relies heavily on pretrained multimodal large language models (MLLMs) for tasks such as image captioning and textual information generation. However, these models are known to sometimes produce hallucinated details. Without mechanisms to detect or filter out such inaccuracies, the generated data could introduce noise into the training process, leading to degraded model performance or unpredictable responses. To strengthen the pipeline's reliability, the authors should provide examples of failed cases and outline a strategy for identifying and filtering out unreasonable or erroneous data.

2. Ethical and Privacy Concerns in Human Image Sourcing: The paper does not mention the origin or sourcing of human images used for training and testing, which is critical from both ethical and privacy standpoints. Without a clear explanation of how these images were obtained and the measures taken to ensure consent and privacy, the approach may risk ethical violations. The authors should explain more about the original metadata collection.

3. The limitation section, which is required, is missing.
Yes
This needs a section to discuss the potential ethics issues, both the dataset and models. Person (or even celebrity), even GPT, sometimes, they response with "I do not know" or "policy issue" etc.

Note: my current score is a temporary score, only evaluate this paper from a technical perspective. Final score may depend on how people feel the ethics issues for this work, especially, its motivation is "face blindness".
Yes, Discrimination / bias / fairness concerns Yes, Privacy, security and safety Yes, Potentially harmful insights, methodologies and applications
1. As suggested, one potential major issue is about the ethics issues about the people topic, which needs more discuss. And also need safety evaluation for the data, benchmark and models.

2. One question is - after you adapt the LLaVA model to the personalized setting, how its (P-LLaVA) performance on the original general benchmarks.

3. And one more question is - just curious, how is the performance of GPT-4o or GPT-4v, Gemini or these frontier models on the proposed P-Bench? How they handle this people/face issue.
Technical Strengths:

1. This paper proposes a way to curate the personalized instruction conversations, and also introduces a P-Bench for evaluation purpose.

2. It finetuned LLaVA in the personalized setting, and demonstrates proposing results.
This paper tries to tackle the "face blindness" issue of the multimodal large language models. While, it is quite questionable whether you need the model to handle "face blindness" directly, or because it is safety and ethics issues, the model needs some mechanism to handle this issue safely. This is an important question for the motivation of this paper.

Technically, this paper presents a method to collect high-quality data for personalized conversations, then proposes Personalized
Visual Instruction Tuning (PVIT) to enable the MLLMs (LLaVA in the paper) for personalized dialogues. This paper also constructs a P-Bench for evaluation purpose. Empirical experiments on P-Bench show that the P-LLaVA can do well in the personalized setting.
1. One major issue is there might lack a section to discuss the potential Ethics issues since it involves people in the dataset and models. 

2. Methodologically, the data curation is okay, the model training part is incremental.
@inproceedings{
anonymous2024personalized,
title={Personalized Visual Instruction Tuning},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=sAxdIJ4l6z},
note={under review}
}
Recent advancements in multimodal large language models (MLLMs) have demonstrated significant progress; however, these models exhibit a notable limitation, which we refer to as "face blindness." Specifically, they can engage in general conversations but fail to conduct personalized dialogues targeting at specific individuals. This deficiency hinders the application of MLLMs in personalized settings, such as tailored visual assistants on mobile devices, or domestic robots that need to recognize members of the family. In this paper, we introduce Personalized Visual Instruction Tuning (PVIT), a novel data curation and training framework designed to enable MLLMs to identify target individuals within an image and engage in personalized and coherent dialogues. Our approach involves the development of a sophisticated pipeline that autonomously generates training data containing personalized conversations. This pipeline leverages the capabilities of various visual experts, image generation models, and (multi-modal) large language models. To evaluate the personalized potential of MLLMs, we present a benchmark called P-Bench, which encompasses various question types with different levels of difficulty. The experiments demonstrate a substantial personalized performance enhancement after fine-tuning with our curated dataset.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
large language model multimodal large language model
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/c758ed06fb8f82c12fa3db6aed436c7d24f734b5.pdf
datasets and benchmarks
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Personalized Visual Instruction Tuning
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission