'No' Matters: Out-of-Distribution Detection in Multimodality Long Dialogue
Natural Language Processing;Out-of-distribution Detection;Machine Learning;Multimodality Dialogue
**Weaknesses**:

- The motivation behind the task is unclear, particularly considering the current capabilities of powerful vision-language models, which may already handle OOD scenarios effectively. The relevance of this task remains questionable.

*Responses*: For the multimodal dialogue domain, it still remains a huge challenge: How to effectively know an image has not or has been mentioned from a long dialogue during people's interaction or from human-agent interactions ([3], [4], [5], [6], [7], [8]). Imagine you are talking with another friend, and during your interactions, you two want to search the related images during your talk contents, while the searched pictures are wrong. I hope this real-life case will help you to understand both from the theory aspect and also from the real-life experience perspective. 

- The proposed method lacks substantial novelty and does not significantly differentiate itself from prior approaches.

*Responses*: It would be great to use some evidence to support your statement. For example, what prior approaches in long dialogue OOD detection, and what are these exactly?  

- The comparisons with existing methods are outdated, with the most recent being from 2018, which weakens the evaluation of the paper's contributions in the context of current research. Additionally, there is no comparison with current large vision-language models (LVLMs) for detection. How do models like GPT-4o, Claude-3.5-Sonnet, Gemini, and Qwen2-VL perform on this classification task?

*Responses*: Indeed, most of the recent Large VLMs focused on QA tasks. In fact, our framework works for single-round QA (see Results Section in the paper), but our focus on multi-round dialogues from the fact that longer dialogues provide more contextual information in real-life scenarios (e.g., imagine you are talking to someone, it is unlikely that we only have one round of the dialogue, but people will continue talking and interacting [7]), which is crucial in real-life scenarios compared to single-round QA, which typically offers limited contextual information (Singh et al., 2019). Multi-turn dialogues are essential for detecting unseen labels when users engage in conversations related to a picture requested in real-scenario settings. Based on this suggestion, we further tested the proposed score function on single-round QA by sampling the first round from the whole dialogue. The results shown in the Results section indicate that single-round QA carries less information in dialogues. The Gap is: How to improve the performance in more real-life cases without much computing resources cost (We only use one Nvidia 3080 card to run everything). 

**Questions**:
- Why is this task still essential in 2024?

*Responses*: See response to Weakness Point 1.

- How do large vision-language models (LVLMs) perform in direct OOD detection?

*Responses*: Refer to Responses for Weakness Point 3. 

**References**: 
- [1] Gan, M., Dou, X., & Jiang, R. (2013). From ontology to semantic similarity: calculation of ontology‐based semantic similarity. The Scientific World Journal, 2013(1), 793091.
- [2] Saha, A., Khapra, M., & Sankaranarayanan, K. (2018). Towards Building Large Scale Multimodal Domain-Aware Conversation Systems. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1). https://doi.org/10.1609/aaai.v32i1.11331
- [3] Yang, D., Rao, J., Chen, K., Guo, X., Zhang, Y., Yang, J., & Zhang, Y. (2024, July). IM-RAG: Multi-Round Retrieval-Augmented Generation Through Learning Inner Monologues. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 730-740).
- [4] Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., ... & Rohrbach, M. (2019). Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 8317-8326).
- [5] Lang, H., Zheng, Y., Hui, B., Huang, F., and Li, Y. (2023). Out-of-domain intent detection considering multi-turn dialogue contexts. arXiv preprint arXiv:2305.03237.
- [6] Azizi, S., Culp, L., Freyberg, J., Mustafa, B., Baur, S., Kornblith, S., Chen, T., Tomasev, N., Mitrovi´c, J., Strachan, P., et al. (2023). Robust and data-efficient generalization of self-supervised machine learning for diagnostic imaging. Nature Biomedical Engineering, 7(6):756–779.
- [7] Gao, R., Roever, C., and Lau, J. H. (2024a). Interaction matters: An evaluation framework for interactive dialogue assessment on English second language conversations. arXiv preprint arXiv:2407.06479.
- [8] Sima, C., Renz, K., Chitta, K., Chen, L., Zhang, H., Xie, C., Beißwenger, J., Luo, P., Geiger, A. and Li, H., 2023. Drivelm: Driving with graph visual question answering. arXiv preprint arXiv:2312.14150.
Response to Reviewer AqfS
Thanks for your review: 

**Weaknesses Responses**: We would appreciate these two papers recommended by the reviewer. However, after carefully reading the two works, the definitions of these two works are quite different compared with our work, our focus is how to improve the OOD label detection for multi-modality long open domain conversation with images, which is more real-life focused. And in the past two years, no single studies have focused on multimodality multi-round long conversations in OOD detection works, even in the NLP dialogue domain ([5], [6], [7]). Most works are related in single-round VQA ([8]), so it is impossible for us to find baselines in past two years. Actually for some datasets, we checked the quality ourselves by manually checking (e.g., ImageChat, and DialogCC, and PhotoChat), this process took more than two-month by the authors (we can provide all the dataset checking evidence if required).

**Questions**:
- It would be better to identify and supplement experiments with use cases where multimodal OOD is important and can be well utilized, rather than focusing on the multi-turn dialogue setting.

*Responses*: Indeed, our framework definitely works for single-round QA (see Results in the paper), but our focus on multi-round dialogues from the fact that longer dialogues provide more contextual information in real-life scenarios (e.g., imagine you are talking to someone, it is unlikely that we only have one round of the dialogue, but people will continue talking and interacting [7]), which is crucial in real-life scenarios compared to single-round QA, which typically offers limited contextual information (Singh et al., 2019). Multi-turn dialogues are essential for detecting unseen labels when users engage in conversations related to a picture requested in real-scenario settings. Based on this suggestion, we further tested the proposed score function on single-round QA by sampling the first round from the whole dialogue. The results shown in the Resluts section indicate that single-round QA carries less information in dialogues.

- It would be good to add comparative experimental results with recent papers that have proposed solutions to multimodal OOD problems.

*Responses*: In the past two years, no single studies have focused on multimodality multi-round long conversations in OOD detection works, even in the NLP dialogue domain ([5], [6], [7]). Most works are related in single-round VQA ([8]). 

**References**:
- [1] Gan, M., Dou, X., & Jiang, R. (2013). From ontology to semantic similarity: calculation of ontology‐based semantic similarity. The Scientific World Journal, 2013(1), 793091.
- [2] Saha, A., Khapra, M., & Sankaranarayanan, K. (2018). Towards Building Large Scale Multimodal Domain-Aware Conversation Systems. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1). https://doi.org/10.1609/aaai.v32i1.11331
- [3] Yang, D., Rao, J., Chen, K., Guo, X., Zhang, Y., Yang, J., & Zhang, Y. (2024, July). IM-RAG: Multi-Round Retrieval-Augmented Generation Through Learning Inner Monologues. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 730-740).
- [4] Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., ... & Rohrbach, M. (2019). Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 8317-8326).
- [5] Lang, H., Zheng, Y., Hui, B., Huang, F., and Li, Y. (2023). Out-of-domain intent detection considering multi-turn dialogue contexts. arXiv preprint arXiv:2305.03237.
- [6] Azizi, S., Culp, L., Freyberg, J., Mustafa, B., Baur, S., Kornblith, S., Chen, T., Tomasev, N., Mitrovi´c, J., Strachan, P., et al. (2023). Robust and data-efficient generalization of self-supervised machine learning for diagnostic imaging. Nature Biomedical Engineering, 7(6):756–779.
- [7] Gao, R., Roever, C., and Lau, J. H. (2024a). Interaction matters: An evaluation framework for interactive dialogue assessment on English second language conversations. arXiv preprint arXiv:2407.06479.
- [8] Sima, C., Renz, K., Chitta, K., Chen, L., Zhang, H., Xie, C., Beißwenger, J., Luo, P., Geiger, A. and Li, H., 2023. Drivelm: Driving with graph visual question answering. arXiv preprint arXiv:2312.14150.
Responses to Reviewer a5DV
Thank you for bringing your perspectives:

- Definition of OOD: Refer to the introduction on Page 1. We presented the rationale and definitions based on previous studies in the dialogue domain. 

- The purpose of this study are not about the image caption, but try to improve the robustness of OOD detection between long-dialogues and images, **despite the embeddings of specific models**, otherwise, it's just a model-specific related solution without robustness to more general VLM models. 

- Based on linguistics theory and NLP dialogue studies supported by Gan and Jiang (2013), the design of our label extractor is rooted in theories emphasizing the importance of both semantic meanings (contextual meaning of a label's token in dialogue) and ontological distances (how humans understand and process certain tokens in languages). These two aspects are considered equally important in our label selection process. For instance, the Label 'Cat' can refer to an animal, but in a different context, such as a dialogue about singers, it could refer to 'Doja Cat'. The real meaning of a label in real-life dialogues heavily depends on context. WordNet evaluates the similarity between different meanings of 'Cat' based on various semantic relations. This approach helps the OOD label by considering the relationships between words and their meanings. Indeed, we explored other tuning strategies and learning weights for the selection presented in Table 2. 

- It would be great to click on the link at the bottom of the Second Page: https://anonymous.4open.science/r/multimodal_ood-E443/README.md , with the detailed information of the datasets and all the information in our dialogue datasets, including a multi-round QA, and long real-life open-domain conversations. Our selection of the datasets is based on previous works ([2],[3],[4]) and the most suitable cases for dialogue-image detection ([5], [6]). Actually for some reviewer's proposed datasets, we also checked the quality ourselves by manually checking the details (e.g., ImageChat, and DialogCC, and PhotoChat), this process took *more than two-month* by the authors in this study (we can provide all the dataset checking evidence if required).  

-  Indeed, our framework definitely works for single-round QA (see Results in the paper), but our focus on multi-round dialogues from the fact that longer dialogues provide more contextual information in real-life scenarios (e.g., imagine you are talking to someone, it is unlikely that we only have one round of the dialogue, but people will continue talking and interacting [7]), which is crucial in real-life scenarios compared to single-round QA, which typically offers limited contextual information (Singh et al., 2019). Multi-turn dialogues are essential for detecting unseen labels when users engage in conversations related to a picture requested in real-scenario settings. Based on this suggestion, we further tested the proposed score function on single-round QA by sampling the first round from the whole dialogue. The results shown in Resluts section indicate that **single-round QA carries less information in dialogues**. 

**References:** 
- [1] Gan, M., Dou, X., \& Jiang, R. (2013). From ontology to semantic similarity: calculation of ontology‐based semantic similarity. The Scientific World Journal, 2013(1), 793091.
- [2] Saha, A., Khapra, M., & Sankaranarayanan, K. (2018). Towards Building Large Scale Multimodal Domain-Aware Conversation Systems. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1). https://doi.org/10.1609/aaai.v32i1.11331
- [3] Yang, D., Rao, J., Chen, K., Guo, X., Zhang, Y., Yang, J., & Zhang, Y. (2024, July). IM-RAG: Multi-Round Retrieval-Augmented Generation Through Learning Inner Monologues. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 730-740).
- [4] Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., ... & Rohrbach, M. (2019). Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 8317-8326).
- [5] Lang, H., Zheng, Y., Hui, B., Huang, F., and Li, Y. (2023). Out-of-domain intent detection considering
multi-turn dialogue contexts. arXiv preprint arXiv:2305.03237.
- [6] Azizi, S., Culp, L., Freyberg, J., Mustafa, B., Baur, S., Kornblith, S., Chen, T., Tomasev, N., Mitrovi´c,
J., Strachan, P., et al. (2023). Robust and data-efficient generalization of self-supervised machine
learning for diagnostic imaging. Nature Biomedical Engineering, 7(6):756–779.
- [7] Gao, R., Roever, C., and Lau, J. H. (2024a). Interaction matters: An evaluation framework
for interactive dialogue assessment on English second language conversations. arXiv preprint
arXiv:2407.06479.
Response to Reviewer ziNZ
We felt sorry to see the qualification of this year's review based on the following reasons:  

- Reviewers (e.g., In this review, Reviewer RMqc gave a 3-score with only providing **one 10-word sentence** reviewing feedback without any specific feedbacks, Reviewer AqfS gave a 3-score with challenging the OOD task in 2024), all demonstrated a lack of understanding or even the reading of this paper, and the answers/solutions to most of these feedback questions can be easily found in the abstract and introduction sections (the first two pages). 

- The reviews demonstrated a lack of understanding of the dialogue domain and led to the huge issue of domain knowledge understanding in the significance of this study (Reviewer AqfS gave a 3-score on this study, while with a limited understanding for prior approaches in multimodal dialogue domain for OOD detection task， and didn't provide enough reasons why this proposed method is the same compared with prior approaches). For the dialogue domain, tackling this question is a huge challenge, and that's why we employed such extensive experiments to ensure the performance. 

- Based on previous reasons, the reviewing marks do not align with feedback for Reviewer AqfS and Reviewer RMqc. For example, these two reviewers give 3 without pointing out a strong rationale. We sincerely hope to see a rational justification with actual reasons on the score and we would love to improve with enough justifications based on scientific review.
Responses to all review
**Weaknesses**:
- The innovation of the methods used in the paper needs to be strengthened

Responses: It is unreasonable and even outrageous to make such a statement of 'lack of novelty' and not give any specific feedback or rationals. This is a far lower than a qualified standard review.
Response to Reviewer RMqc
**Weaknesses**:
- Models like CLIP and BLIP are primarily trained for image captioning, and some previous researches suggest that they may not generate optimal text embeddings for dialogue. How does this paper address the potential limitations of using these models in a dialogue context to ensure accurate and meaningful embeddings?

Responses: The purpose and focus of this study are not about caption, but to try to improve the OOD detection ability between dialogues and imagines. This miss-understand the whole focus of this study. 

- Does the proposed method consider only yes/no question-answer dialogues as in-domain scenarios? If so, when OOD situations become more complex, it’s unclear how well the method would perform or if it would remain effective in identifying out-of-domain cases accurately.

Responses: This is a classical dialogue-related issue. If you take a closer look at the dataset we included in the paper, the dialogue is long and complex in most cases instead of just YES/NO short-format questions. 

**Question**: 

- If you click on the link at the bottom of the second page, which is: https://anonymous.4open.science/r/multimodal_ood-E443/README.md . You will see all the detailed information of the dataset, and with all the information in our dialogue datasets.
Responses to Reviewer UswD
Yes
No ethics review needed.
Can you include the example of the test set generated?
- The paper focuses on two key types of out-of-distribution (OOD) scenarios: (1) mismatches between dialogue and image inputs, and (2) inputs with previously unseen labels. It demonstrates the effectiveness of the proposed method in accurately identifying these OOD cases.
- This work marks the first attempt to address OOD detection in dialogue contexts, specifically for multi-round conversations. To support this, the authors constructed a new dataset for multi-round question-answering, enabling comprehensive evaluation of the framework’s performance in real-life dialogue settings.
This paper addresses the challenge of out-of-distribution (OOD) detection in multimodal contexts, particularly focusing on the combined input of dialogues and images in real-life applications such as open-domain conversational agents. It introduces the Dialogue Image Aligning and Enhancing Framework (DIAEF), an approach for detecting mismatches in dialogue and image pairs and identifying previously unseen input labels in conversations. DIAEF integrates visual language models with scoring metrics tailored for two primary OOD scenarios: (1) detecting mismatches between dialogue and image inputs, and (2) flagging dialogues with previously unseen labels. Experiments conducted on several benchmarks indicate that DIAEF’s integrated approach to image and multi-round dialogue OOD detection outperforms single-modality methods, especially in dialogues involving mismatched pairs and extended conversations.
- Models like CLIP and BLIP are primarily trained for image captioning, and some previous researches suggest that they may not generate optimal text embeddings for dialogue. How does this paper address the potential limitations of using these models in a dialogue context to ensure accurate and meaningful embeddings?
- Does the proposed method consider only yes/no question-answer dialogues as in-domain scenarios? If so, when OOD situations become more complex, it’s unclear how well the method would perform or if it would remain effective in identifying out-of-domain cases accurately.
Yes
No ethics review needed.
none
The starting point chosen for the paper is quite innovative.
This paper take the first attempt for OOD detection in multimodality long dialogue, propose a framework that enhances the OOD detection in cross-modal contexts，achieved the combination of OOD detection and multimodal methods. 

And it demonstrate that integrating image and multi-round dialogue OOD detection is more effective with previously unseen labels than using either modality independently.
The innovation of the methods used in the paper needs to be strengthened.
Yes
No ethics review needed.
Please refer to Weaknesses.
- This paper is well-written, especially considering that the topic of OOD detection is not easy to understand. For instance, the authors explain the problem formulation of "cross-modal OOD detection" clearly.
- The paper introduces a new paradigm and framework for OOD detection in "multi-turn interactive dialogue," along with a new scoring method, DIAEF, which utilizes vision-language models.
- They demonstrate the effectiveness of DIAEF both experimentally and theoretically and suggest its potential as an alternative scoring method, as shown in Table 1.
- Through extensive experiments, the authors show that DIAEF outperforms other OOD detection scoring methods and empirically validate their design choices (e.g., the selection of alpha).
This paper presents a new framework, Dialogue Image Aligning and Enhancing Framework (DIAEF), to improve the user experience in multi-round dialogues by efficiently detecting out-of-distribution (OOD) instances in multimodal contexts, specifically dialogue-image pairs. DIAEF integrates visual language models with novel scoring mechanisms to identify OOD cases in two main scenarios: mismatches between dialogue and image inputs and previously unseen labels in input pairs. Experimental results show that the combined use of dialogue and image data enhances OOD detection more effectively than using each modality independently, demonstrating robustness in prolonged dialogues. This approach supports adaptive conversational agents and sets a benchmark for future research in domain-aware dialogue systems.
- Although the authors clearly present the problem formulation of "cross-modal OOD detection," I still find the use of OOD terminology in the multi-modal dialogue domain unclear. Dialogue inherently has a subjective nature and a one-to-many structure (i.e., diversity [1]), meaning that even with the same query, there are multiple possible responses depending on the situation and the user in real-world interactions. Therefore, I question whether using the term "OOD" is appropriate in this context. The authors should further clarify why handling OOD detection in the multi-modal domain is necessary.
- Additionally, I am concerned that using CLIP or BLIP models may not ensure adequate understanding of dialogue, as CLIP has a limited context length of 77 tokens, and neither CLIP nor BLIP is pretrained on open-domain dialogue datasets—issues highlighted in prior works [2-3]. When determining OOD, it seems that the embedding model reflects its training distribution, yet CLIP embeddings may be ineffective for dialogue. I believe that using LongCLIP [4] could be a better alternative. Therefore, the authors should clarify their choice of CLIP or BLIP for the VLM models.
- In the DIAEF framework, training the "label extractor" is crucial; however, I don’t fully understand what constitutes a "label" in an "open-domain dialogue." Could you explain this?
- While the authors demonstrate the effectiveness of their framework, more experiments are needed to establish its robustness and reliability across additional dialogue datasets. The framework formulation includes multiple hyperparameters (e.g., $\alpha$ and $\gamma$), and the MMD dataset is not a high-quality multi-modal dialogue dataset since it is synthesized using CLIP matching, despite the application of human crowdsourcing to verify contextual relevance. Nevertheless, this dataset lacks both high quality and diversity, which is mentioned in the prior work [5]. I recommend that the authors conduct experiments on additional dialogue datasets, such as PhotoChat [6], MP-Chat [7], ImageChat [8], and DialogCC [5]. Given time constraints, it is unnecessary to experiment on the full datasets; subsampled versions would suffice.
- I am also curious as to why the authors focus on "long dialogue," as, to my knowledge, the datasets used in the experiments emphasize single-session dialogues rather than multi-session dialogues like MSC [9] or Conversational Chronicles [10].

---

**References**

[1] Li, Jiwei, et al. "A diversity-promoting objective function for neural conversation models." arXiv preprint arXiv:1510.03055 (2015).

[2] Yin, Zhichao, et al. "DialCLIP: Empowering Clip As Multi-Modal Dialog Retriever." ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024.

[3] Lee, Young-Jun, et al. "Large Language Models can Share Images, Too!." arXiv preprint arXiv:2310.14804 (2023).

[4] Zhang, Beichen, et al. "Long-clip: Unlocking the long-text capability of clip." arXiv preprint arXiv:2403.15378 (2024).

[5] Lee, Young-Jun, et al. "Dialogcc: Large-scale multi-modal dialogue dataset." arXiv preprint arXiv:2212.04119 (2022).

[6] Zang, Xiaoxue, et al. "Photochat: A human-human dialogue dataset with photo sharing behavior for joint image-text modeling." arXiv preprint arXiv:2108.01453 (2021).

[7] Ahn, Jaewoo, et al. "Mpchat: Towards multimodal persona-grounded conversation." arXiv preprint arXiv:2305.17388 (2023).

[8] Shuster, Kurt, et al. "Image chat: Engaging grounded conversations." arXiv preprint arXiv:1811.00945 (2018).

[9] Xu, J. "Beyond goldfish memory: Long-term open-domain conversation." arXiv preprint arXiv:2107.07567 (2021).

[10] Jang, Jihyoung, Minseong Boo, and Hyounghun Kim. "Conversation chronicles: Towards diverse temporal and relational dynamics in multi-session conversations." arXiv preprint arXiv:2310.13420 (2023).
Yes
No ethics review needed.
1. It would be better to identify and supplement experiments with use cases where multimodal OOD is important and can be well utilized, rather than focusing on the multi-turn dialogue setting.
2. It would be good to add comparative experimental results with recent papers that have proposed solutions to multimodal OOD problems.
1. This paperintroduces a unique approach for OOD detection by combining image and dialogue data, addressing the limitations of using single modalities.

2. The framework's use of an alignment and enhancement scoring mechanism allows for precise multimodal OOD detection.

3. By focusing on mismatched pairs and unseen labels, the framework is suited for real-world applications where dialogue and visual information often co-occur.
This paper addresses the challenge of Out-of-Distribution detection in multimodal long-dialogue systems, where text and image modalities are combined, especially in real-world or open-domain dialogue applications. The authors propose the Dialogue Image Aligning and Enhancing Framework, designed to detect two main types(Mismatch between dialogue and image, Unseen labels) of OOD cases.
This paper has chosen multimodal, multi-turn dialogue environments as the task to perform and verify OOD detection. The authors claim this is necessary for user satisfaction and trust, but this part of the argument is not convincing. They need to provide logical supplementation on why multimodal OOD detection tasks are important in dialogue, and why they are particularly important in multi-turn rather than single-turn interactions.
Similar problem definitions were found in other papers [1,2], but references to these papers are missing, and explanations and quantitative metrics are needed to show how this paper differentiates itself from these works. The baseline methods used in this paper are all methodologies from before 2019, and experiments should be designed to include methodologies from recent papers.


[1]. GENERAL-PURPOSE MULTI-MODAL OOD DETECTION FRAMEWORK, V Duong et al. 2023
[2]. MultiOOD: Scaling Out-of-Distribution Detection for Multiple Modalities, H DOng et al. 2024