ImplicitAVE: An Open-Source Dataset and Multimodal LLMs Benchmark for Implicit Attribute Value Extraction
Henry Peng Zou Vinay Samuel Yue Zhou Weizhi Zhang Liancheng Fang Zihe Song Philip S. Yu Cornelia Caragea

Federated Learning with Only Positive Labels by Exploring Label Correlations
    Xuming An,
        Dui Wang,
        Li Shen,
        Yong Luo,
        Han Hu,
        Bo Du,
        Yonggang Wen Fellow, IEEE,
        Dacheng Tao Fellow, IEEE

X. An and H. Hu are with the School of Information and Electronics, Beijing Institute of Technology, Beijing 100081, China (email: anxuming@bit.edu.cn; hhu@bit.edu.cn).
D. Wang, Y. Luo and B. Du are with the National Engineering Research Center for Multimedia Software, School of Computer Science, Institute of Artificial Intelligence and Hubei Key Laboratory of Multimedia and Network Communication Engineering, Wuhan University, Wuhan 430072, China (e-mail: wangdui@whu.edu.cn; yluo180@gmail.com; dubo@whu.edu.cn).

L. Shen and D. Tao are with the JD Explore Academy, Beijing, 100000, China (email: mathshenli@gmail.com, dacheng.tao@gmail.com).
Y. Wen is with the School of Computer Science and Engineering, Nanyang Technological University, Singapore 639798 (e-mail: ygwen@ntu.edu.sg).

    April 29, 2024
=============================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================





Existing datasets for attribute value extraction (AVE) predominantly focus on explicit attribute values while neglecting the implicit ones, lack product images, are often not publicly available, and lack an in-depth human inspection across diverse domains. To address these limitations, we present ImplicitAVE, the first, publicly available multimodal dataset for implicit attribute value extraction. ImplicitAVE, sourced from the MAVE dataset, is carefully curated and expanded to include implicit AVE and multimodality, resulting in a refined dataset of 68k training and 1.6k testing data across five domains. We also explore the application of multimodal large language models (MLLMs) to implicit AVE, establishing a comprehensive benchmark for MLLMs on the ImplicitAVE dataset. Six recent MLLMs with eleven variants are evaluated across diverse settings, revealing that implicit value extraction remains a challenging task for MLLMs. The contributions of this work include the development and release of ImplicitAVE, and the exploration and benchmarking of various MLLMs for implicit AVE, providing valuable insights and potential future research directions. Dataset and code are available at 
<https://github.com/HenryPengZou/ImplicitAVE>










§ INTRODUCTION


Attribute Value Extraction (AVE) identifies the value of product attributes from the product information, which is critical in e-commerce for product representation, recommendation, and categorization <cit.>. The attribute values can be categorized into two types: (1) Explicit values can be directly found as a segment in the product text <cit.>, while (2) Implicit values are never mentioned in the text and can only be inferred from the product image, contextual clues, or prior knowledge <cit.>. Consider the example in Figure <ref>. The  “rain boot” of the  “boot style” is implicit since it is not explicitly stated in the product text but can be inferred from its image or context from keywords such as “transparent” and “waterproof.”













Nonetheless, existing datasets for attribute value extraction exhibit several key limitations: (1) They predominantly focus on explicit attribute values, neglecting implicit attribute values <cit.>, which are more challenging and commonly encountered in real-world scenarios; (2) Many datasets lack product images <cit.>, limiting their applicability in multimodal contexts; (3) The limited number of publicly available datasets lack human inspection and cover only a few domains, resulting in inaccurate and restricted benchmarks <cit.>. Table <ref> compares these aspects for various AVE datasets.

To address these issues, we present ImplicitAVE, the first publicly available multimodal dataset for implicit attribute value extraction. We initially sourced product text data from the MAVE dataset <cit.> and then curated the data by eliminating unhelpful attributes and redundant or irrelevant values. Subsequently, we transformed and expanded the dataset to include implicit attribute value extraction and multimodality and finally validated the test set annotations through two rounds of human inspection. This yields a more refined and quality-improved dataset of 68k training and 1.6k testing data spanning five diverse domains with 25 attributes and corresponding attribute values suitable for implicit attribute value extraction. Detailed statistics of our dataset are shown in Tables <ref>, <ref>.


























Given the cutting-edge performance of Multimodal Large Language Models (MLLMs) <cit.> and the absence of previous exploration of their application to implicit attribute value extraction, we establish a comprehensive benchmark for MLLMs on our ImplicitAVE dataset. We cover six recent MLLMs with 11 variants and compare them with the fine-tuned previous SOTA method. We evaluate their performance across diverse settings, including full/few-shot and zero-shot scenarios, domain-level and attribute-level performance, and single/multi-modality performance. We find that implicit value extraction remains a challenging task for open-source MLLMs despite their effective capabilities. 

Our contributions are summarized as follows: (1) The development and release of ImplicitAVE, the first open-source multimodal dataset for implicit AVE; (2) The exploration and benchmarking of various MLLMs for implicit attribute value extraction across diverse settings, revealing intriguing insights and potential future research directions.

























§ DATASET CONSTRUCTION



























We outline our approach to constructing the first open-source multimodal implicit attribute value extraction dataset, ImplicitAVE. The dataset construction pipeline is illustrated in Figure <ref>. It contains four steps: data collection, curation, expansion, and validation. Next, we explain them in detail. 




 §.§ Initial Data Collection


Initially, we sourced product text information, including titles, categories, and corresponding attribute-value annotations, from the publicly available MAVE dataset <cit.>, comprising 2.2 million products spanning diverse e-commerce domains. Despite its extensive coverage, the MAVE dataset exhibits several significant limitations, making it unsuitable for implicit AVE: (1) It contains inappropriate attributes and values that are not facilitative to implicit AVE tasks (see Step 2); (2) It is designed solely for explicit attribute-value extraction; (3) It solely comprises textual information and lacks multimodal data sources; (4) Annotations within the MAVE dataset are machine-generated and lack human inspection, resulting in notable inaccuracies. 



 §.§ Data Curation for Implicit AVE


We further refine the sourced data by removing unhelpful attributes and redundant or irrelevant values for Implicit AVE. Concretely: 182 Removing Inference-Infeasible Attributes. We manually inspect and remove attributes where the specific values are almost impossible to infer if the values are not mentioned explicitly in the text, such as display resolution, storage capacity, and battery life; 183 Removing Subjective Attributes. The attributes that are rather subjective and ambiguous, such as the degree of comfort and product quality, are also removed; 184 Value Merging and Cleaning. Attribute values with similar semantic meanings are consolidated. This includes unifying variations in grammar forms (e.g., Short-Sleeve, Short sleeves, short sleeved for the attribute Sleeve Style), eliminating extraneous words (e.g., running and running shoes), and merging synonyms (e.g., floral and flower, leopard and cheetah, crew neck and round neck, plaid and tartan, etc.) In addition, we notice some values are irrelevant to their parent attributes (e.g., the value “Clear Stamps” of the attribute “Material of Artwork”), so these values are removed as well. The value merging and cleaning is achieved collaboratively by lexicon-based scripts, prompting with GPT-4, and human inspection.

This curation results in a more refined and quality-improved dataset with 25 attributes and corresponding attribute values spanning five domains suitable for implicit attribute value extraction. We randomly sample up to 1000 instances per attribute value to limit the dataset size. The selected domains and attributes in ImplicitAVE are shown in Table <ref>.


































































 §.§ Data Expansion


To extend the data for implicit attribute value extraction and multimodality, we perform the following processing steps: 182 Implicit Value Creation. We remove all explicit attribute value mentions from the input text for its corresponding attribute for each data point. As a result, attribute values in these data can only be inferred from the product images, indirect text context, or prior knowledge. That is, these values become implicit attribute values given the modified inputs. We then drop instances with the same product ID or image to prevent potential information leakage across instances based on the same product. 183 Multimodality Creation. We systematically collect product images from the Amazon website using the product identification number and thus expand our dataset with multimodal information. 





 §.§ Human Inspection & Re-Annotation


Through manual inspection, we observed that the original attribute-value annotations from MAVE contain noticeable errors. This is because they were annotated by ensembling predictions from five variations of AVEQA models <cit.>[AVEQA <cit.> is a question-answering model that regards each query attribute as a question and determines the answer span that matches the attribute value within the product text information.] without human inspection. To rectify incorrect annotations and ensure a high-quality test set for implicit attribute value extraction and MLLMs evaluation, we engage five Ph.D. students to manually inspect and re-annotate our evaluation set. 




This process first involves sampling ten instances per attribute value from the constructed dataset, resulting in 1,676 instances. The human inspection and re-annotation process then unfold in two rounds:
In the first round, annotators assess each instance's product image, text contexts, and relevant attributes to determine the correctness of the original attribute value annotation. If annotators think the original annotation is incorrect, they select the best attribute value from the corresponding value list (of that attribute) or mark `N/A' if the annotator believes no suitable value is provided or multiple values are suitable. Additionally, annotators can suggest improvements such as merging, removing, adding, or replacing attribute values. Of the total instances, 1,448 original annotations are correct, 172 are incorrect, and 56 are marked as `N/A,' yielding an agreement rate of 86.4%. Ten, one, one, and one attribute values are suggested for merging, removing, adding, and replacing, respectively. Instances with disagreed annotations are subject to a second round of inspection and re-annotation, wherein three well-trained annotators participate, and a majority vote determines the final annotation for each instance. 








 §.§ Dataset Statistics


The overall domain-level dataset statistics is provided in Table <ref>.  We have 68,604 training instances and 1,610 high-quality evaluation instances. Our dataset covers 5 diverse domains and 25 carefully curated attributes specially for the task of implicit attribute value extraction. We also provide detailed attribute-level statistics in Table <ref>. Different attributes contain different numbers of value options that are meticulously selected and processed and we have a total of 158 diverse attribute values. In addition, we visualize the data distribution of domains, attributes and their values for our training set and evaluation set in Figure <ref>(a) and <ref>(b), respectively. It can be observed that compared to the training set, each attribute in the evaluation set has a considerably balanced value distribution, making it more suitable for zero-shot MLLMs evaluation.











§ EXPERIMENT & BENCHMARK



In this section, we describe our experiment results evaluating the effectiveness of various MLLMs and the previous SOTA method on our ImplicitAVE dataset in diverse settings. 



 §.§ Experimental Setting



  
Evaluation Setup
We benchmark different models on our datasets from both attribute and domain levels: 

∙ Attribute-Level Results refer to the micro-F1 score calculated between the ground truth answer and the model-generated answer for each query/interested attribute. 

∙ Domain-Level Results refer to the micro-F1 score calculated between the ground truth answer and the model-generated answer for all query/interested attributes in each domain.



We determine whether the generated answer is correct by checking whether the generated answer contains the true answer.




  
Models for Zero-Shot
We utilize the following multimodal LLM frameworks in zero-shot settings:

∙ BLIP-2 <cit.> proposes a Query Transformer and employs an efficient two-stage vision-and-language pre-training strategy leveraging a frozen image encoder and an LLM. We provide benchmarks of BLIP-2 with two backbone LLM models, FLAN-T5-XL and FLAN-T5-XXL.

∙ InstructBLIP <cit.> enhances vision-language models through instruction tuning with an instruction-aware Query Transformer introduced. We also report the performance with two backbone LLMs, Vicuna-7B and FLAN-T5-XXL.

∙ LLaVA <cit.> connects the visual encoder of CLIP <cit.> with the language decoder, and performs fine-tuning on GPT-4 generated language-image instructions. We provide benchmarks of LLaVA with Vicuna-13B.

∙ LLaVA-1.5 <cit.> advances its predecessor by focusing on efficient visual instruction tuning, integrating a fully-connected vision-language cross-modal connector for enhanced interaction between visual and textual modality. We provide benchmarks of LLaVA-1.5 using Vicuna-7B and Vicuna-13B as the language models.

∙ Qwen-VL <cit.> proposes a novel visual receptor and a position-aware adapter, optimizing through a three-stage training pipeline on a multilingual and multimodal dataset. We report the performance of both Qwen-VL and the chat version, Qwen-VL-Chat.

∙ GPT-4V[https://chat.openai.com/] integrates vision into the GPT-4 architecture, one of the cutting-edge close-sourced LLMs fine-tuned by reinforcement learning from human feedback.










  
Models for Finetuning Due to the resource constraints, we fine-tuned and evaluated the following two open-source models in both few-shot and full-data tuning settings: 

∙ LaVIN <cit.> introduces a novel mix-of-modularity adaptation module, optimizing the integration of visual inputs into large language models through lightweight adapters and enabling efficient end-to-end training. 

∙ DEFLATE <cit.> is a multi-modal generative-discriminative framework designed for both explicit and implicit attribute value extraction and is the previous SOTA model for implicit AVE.







 §.§ Experimental Results





  §.§.§ Domain-Level Results



We present the domain-level results of all evaluated models in Table <ref>. GPT-4V outperformed every other model in both the zero-shot and finetune setting in every single domain. Among the two models that were finetuned, LAVIN outperformed DEFLATE in every single domain by a minimum of 2.62 points (in the Home Product domain) and a maximum of 11.51 (in the Clothing domain). Among the open-source MLLMs, no single model outperformed all other models across all the domains, but Qwen-VL had the best scores in the Jewelry&GA and Food domains. From Table <ref> we also note that other than for LLaVA 1.5 in the Footwear domain, all other models that had multiple variants with different LLM sizes had significantly better performance on average from the variant with the larger size LLM in each domain in comparison to the variant with the smaller sized LLM. For example, in the Clothing domain, there was a minimum improvement of 15.04 micro-F1 points from the model variant with the smaller LLM (InstructBLIP w/ Vicuna 7B) to the model variant with the larger LLM (InstructBLIP w/ FLAN-T5-XXL) and an overall average of 18.29 micro-F1 point increase when using a model variant with a larger LLM in the Clothing domain. Similar trends can be seen among all domains.  



Additionally, among zero-shot methods, Clothing had the lowest micro-F1 across all domains for all models and model variants except for BLIP2 w/ FLAN-T5-XL and Qwen-VL. This leads us to believe that the Clothing domain is the most challenging domain in the dataset. We performed a comprehensive manual investigation and we believe there are two primary reasons why the Clothing domain presents more challenges, while other domains such as the Home domain are comparatively easier (We show examples from our manual investigation in Figures <ref>, <ref> for clarity): 



  
(1) Attributes within the Clothing domain demand a more nuanced understanding of local details in product images. For example, the attribute ‘Sleeve Style’ in cases 1-4 and ‘Neckline’ in cases 7-12 (Figure <ref>). In contrast, attributes in the home domain only require a global understanding of product pictures and text, such as attribute ‘Special Occasion’ in cases 13-16, ‘Shape’ and ‘Material’ in cases 17 and 21 (Figure <ref>).



  
(2) The values of attributes in the Home domain are significantly more straightforward to identify compared to those in the Clothing domain. For instance, the attribute `Special Occasion' includes values like [`Birthday', `Christmas', `Easter', `Graduation', `Halloween', `Patriotic', `Thanksgiving'], which are clearly more distinguishable than the values for 'Sleeve Style' [`Sleeveless', `Long Sleeve', `3/4 Sleeve', `Strappy', `Short Sleeve'] in the Clothing Domain.













  §.§.§ Attribute-Level Results






Table <ref> presents the attribute-level performance of all evaluated models.
As was observed in Table <ref>, GPT-4V vastly outperforms all other models. We can see in Table <ref> that only in the `Shoulder Style' (InstructBLIP), `Container' (LLaVA 1.5) and `Sleeve Style' (LAVIN) attributes do a model outperform GPT-4V. InstructBLIP struggled significantly with the `Attachment Method' attribute as did LLaVA 1.5 with `Shoulder Style' compared to other models. 

On the other hand, Table <ref> shows that both finetuned models perform better than all of the open-source MLLMs in the zero-shot setting for the `Heel Height' attribute. This may indicate that there are attributes within the dataset for which prior pretrained knowledge of MLLMs is not sufficient for implicit value extraction of these attribute values and finetuning is needed to learn the mapping between instances of these attributes and the correct attribute values belonging to them. 

In addition, all models struggled on the `Sleeve Style' and `Neckline' attributes compared to each model's performance on other attributes. Representative error cases for different attributes are presented in Figures <ref> and <ref> in Appendix <ref> along with a comprehensive error analysis. Here we provide our observations from the attribute-level error analysis:






(1) Models often confuse attribute values that are similar yet distinct, such as `3/4 Sleeve' versus `Long Sleeve' in cases 1-2, `Short Sleeve' versus `Sleeveless' in cases 3-4, and `Crew Neck' versus `Scoop Neck' in case 8 (Figure <ref>).



(2) Attributes that demand a detailed understanding of small image parts typically challenge models, leading to errors. For instance, mistakes in identifying `Shoulder Style’ in cases 5-6 and `Neckline’ in cases 7-9 (Figure <ref>).



(3) Errors can also arise from conflicting modality inferences, as seen in case 13 (Figure <ref>), where the word `Snow Village' in the product text suggested Christmas, but the image aligned more with Halloween.





















  §.§.§ Challenges and Opportunities


Challenging Domains & Attributes: It can be observed in Table <ref>, <ref> that GPT-4V works well on some domains and attributes, but not on all of them, e.g., it only achieves 77.4% micro-F1 on the Clothing domain and 66.0% for the Sleeve Style attribute. Some examples of challenging domains, attributes, and the performance of various MLLMs are highlighted in Table <ref>. Besides, we can observe that the open-source models are lagging behind GPT-4V in many domains and attributes, and our dataset provides a good benchmark that points out the gap between them and provides opportunities for researchers to close it.

Furthermore, inspired by the error cases in Section <ref> and Appendix <ref>, we point out some remaining challenges and opportunities:



  
Model-Aspect:
(1) Enhance the ability to understand image details, including small areas and text in images; (2) Devise mechanisms to distinguish similar attribute values; (3) Properly handle conflicting modality inferences; (4) Reduce the performance gap in implicit AVE between open-source models and advanced closed models like GPT-4V.


  
Dataset-Aspect:
Our ImplicitAVE dataset does not consider multi-valued attributes and negative instances, i.e., "none" as attribute values. We leave this extension for future work.


















  §.§.§ Few-Shot Results



Figure <ref> shows the performance comparison of DEFLATE and LAVIN models in various few-shot settings. We note that in most K-shot settings, LAVIN outperforms DEFLATE by a noticeable amount. 

Also, we notice that different domains performed differently for the two models. In the 5-shot setting, `Food' was the lowest scoring domain for DEFLATE whereas `Food' was the highest scoring domain for LAVIN, but for 10-shot the domain trends for both models became similar (i.e., `Food' and `Home Product' domains were the two best-performing domains and `Clothing' and `Footwear' were the worst performing domains). For DEFLATE from 25-shot to 50-shot, the largest increase in micro-F1 was for the `Clothing' and `Footwear' domains whereas the increase was less significant for the other domains. This indicates that the model's ability to learn the attributes and attribute values in the `Clothing' and `Footwear' domains may continue to increase as the number of training examples increases. On the contrary, LAVIN saw the biggest increase in micro-F1 for the `Jewelry&GA' and `Footwear' domains thereby hinting that increasing the training examples for these domains in LAVIN would enable the model to substantially increase its ability to categorize instances of these two domains. 



  §.§.§ Modality-Level Results



Figure <ref> visualizes performance comparisons of DEFLATE, LAVIN, and GPT-4V with different modalities.
Firstly, it is evident that for LAVIN and DEFLATE, the image-only modality performed extremely poorly compared to the text-only and combined modalities. This leads us to believe that these models’ image understanding capabilities may be too poor to extract implicit value from product images. However, it is worth noting that in all domains except `Footwear' for DEFLATE, both LAVIN and DEFLATE perform better in the multimodal modality over the text-only modality thereby indicating that the image information does in fact help the model predict attribute values of instances. With GPT-4V we notice a very high performance in the image-only modality and only minimal improvement with the multimodal modality in comparison to the image-only modality. This speaks to the strength of GPT-4V’s zero-shot image classification capabilities, especially in comparison to LAVIN and DEFLATE. Even though GPT-4V boasts impressive performance in most regards, it is worth noting that GPT-4V’s text-only modality performance in the `Clothing' domain was especially poor. It scored even lower than the text-only scores of LAVIN and DEFLATE and, in the `Clothing' domain, the multimodal performance for GPT-4V was lower than the image-only performance thereby indicating that the text component confused the model causing it to perform worse than it did without the text component. 










 §.§ Ablation Study on Prompt Templates

In order to obtain baseline results that accurately reflect the quality of our dataset we conducted ablations on the prompt for the open-source MLLMs. Observing drastic micro-F1 score differences on the evaluation set by using different prompts in the early stages of experimenting led us to conduct a standardized ablation study on 8 different prompts listed in Table <ref>. Each prompt had three components: context containing the title of the product with explicit mention of the attribute value removed, question, and options to answer from. To conduct a fair evaluation of the prompts, across all models we fixed the random seed at 42 as well as the hyperparameters: temperature = 1, top_p = 0.8, max_new_tokens = 17, min_length = 1, and num_beams = 5. Our results are shown in Fig <ref> and the best prompt for each model type was used for all variants of that model. 







§ RELATED WORK



 §.§ Attribute Value Extraction Dataset


Attribute Value Extraction (AVE) has emerged as a crucial task for online shopping, aiming to identify the values of product attributes from various data sources. 
At the heart of many e-commerce applications, such as product comparison, retrieval, recommendation, and the construction of product graphs and online shop assistants, lies the extraction of attribute values <cit.>. 

Although several AVE datasets have been introduced, each exhibits certain limitations, as shown in Table <ref>.

The OpenTag dataset <cit.>, one of the early datasets collected from Amazon, highlights the importance of open-world value sets. In contrast, the AE-110K dataset <cit.> expands the scope of AVE datasets to include more products, a broader range of attributes, and denser attribute coverage per product, though it lacks human expert annotation. The AdaTag dataset <cit.> focuses on the rich information contained in product bullets, excluding product descriptions, which facilitates more efficient training and inference for such tasks but lacks diversity in product domains and is not publicly available. The MAVE dataset <cit.>, a large public dataset for AVE research, encompasses a wide range of categories and diverse attributes, constructing structured product files as text inputs. However, in real-world scenarios, text information alone may not imply certain attributes of interest, making product images a complementary source of information for indicating or validating the answers to specific attributes. To address this, the MEPAVE <cit.> and DESIRE <cit.> datasets were introduced to include multimodal product information such as product titles, descriptions and images. While several explicit AVE datasets exist, implicit AVE is much more challenging and under-explored.  To advance multi-modal AVE research further, we introduce the first publicly available multimodal implicit AVE dataset, ImplicitAVE, featuring careful human annotation and a versatile range of items from multiple domains. Our dataset is considerably different from DESIRE, as detailed in Appendix <ref>.











 §.§ Multimodal Large Language Models


Multimodal Large Language Models (MLLMs) have demonstrated impressive performance on a variety of tasks <cit.>.
BLIP-2 <cit.> uses frozen pre-trained image models and language models, and proposes a lightweight querying transformer Q-Former to bridge the two modalities. InstructBLIP <cit.> outperforms BLIP-2 <cit.> by using vision-language instruction tuning, where the instruction tuning data is collected from publicly available datasets, by manually transforming them into instruction tuning format. To improve the diversity and in-depth reasoning in the instruction, LLaVa <cit.> proposes to use language-only GPT-4 to construct multimodal language-image instruction tuning data. mPLUG-Owl <cit.> and Qwen-VL <cit.> propose novel training paradigms for LLMs.
However, since most popular open-source MLLMs are parameter-heavy, LAVIN <cit.> proposes a novel and efficient solution for vision-language instruction tuning by adopting lightweight modules, i.e., adapters, to bridge the gap between LLMs and vision modules, which does not require expensive vision-language pretraining to align text and image embedding beforehand. Despite achieving significant progress, the performance of MLLMs on implicit AVE has not been well-studied. Recent work EIVEN <cit.> finetuned an efficient MLLM framework for implicit AVE but did not compare with exiting MLLMs in zero-/few-shot settings. Our work establishes the first comprehensive benchmark of multimodal LLMs for implicit AVE under diverse settings and reveals intriguing
insights and potential future research directions in Section <ref>.








§ CONCLUSION





In this paper, we introduced ImplicitAVE, the first publicly accessible multimodal dataset specifically designed for implicit attribute value extraction, aimed at overcoming the limitations of existing datasets focused on explicit attribute values. By carefully curating attribute values and incorporating both implicit attribute values and product images, ImplicitAVE comprises 6.8K training instances and 1.6K human re-annotated high-quality evaluation instances across five diverse domains. Moreover, we benchmarked the performance of six recent multimodal large language models on it under diverse settings, highlighting the challenges of implicit value extraction. In the future, we plan to further expand our ImplicitAVE dataset to include multi-valued attributes and negative instances.




§ ACKNOWLEDGEMENTS


This research is partially supported by NSF grant #210751 and UIC DPI Seed Program. Any opinions, findings, and conclusions expressed here are those of the authors and do not necessarily reflect the views of NSF. We thank our reviewers for their insightful feedback and comments which helped improve the quality of our paper. 






§ LIMITATION


Our ImplicitAVE dataset does not consider multi-valued attributes and negative instances, i.e. "none" as attribute values. We leave this extension as future work. Due to computational resource constraints and limited budgets, we did not evaluate open MLLMs with parameters larger than 13B. 




§ ETHICS STATEMENT


The datasets that we sourced from are publicly available. In this work, we propose a multimodal Implicit AVE dataset and provide a comprehensive benchmark of MLLMs. We do not expect any direct ethical concern from our work.





















































































































































































































§ NOVELTY AND CONTRIBUTION OF DATASET


In Table <ref>, we compare our dataset with existing AVE datasets from different aspects. While several explicit AVE datasets exist, implicit AVE is much more challenging and underexplored. To the best of our knowledge, our work introduces the first open-source dataset that is expressly designed to address the task of implicit AVE. Here, we would like to clarify that our dataset is considerably different from DESIRE <cit.> regarding:



  
  * Accessibility: In DESIRE, all data are encrypted, and image data are encoded by DALL-E <cit.>, with raw images unavailable. Moreover, DESIRE does not provide statistics on how many implicit AVE examples are included. In contrast, our dataset contains 70k+ curated implicit AVE examples, which are immediately available with raw images provided.
  
  * Domain Scope: DESIRE only contains the `Food' domain, while our dataset contains five domains, including more challenging domains such as `Clothing' and `Footwear'.
  
  * Language: DESIRE is based on Chinese while our dataset is in English.






§ DOMAIN, ATTRIBUTE, AND VALUE


The ontology of our data adheres to the domain-attribute-value structure, where (1) Each domain contains relevant attributes that characterize different aspects of the domain product. For instance, the `Clothing' domain contains attributes such as `Sleeve Style' and `Neckline'; (2) Each attribute comprises a set of possible values (also called "attribute values") and we aim to extract its ground truth value from product images and text contexts. For example, the attribute `Sleeve Style' may include values such as `Long-sleeve', `3/4 sleeve', and `Strappy'. The full details are depicted in Table <ref>. Figure <ref> also presents a few examples of products in different domains, with different attributes and values.

Therefore, in Tables <ref> and <ref>, Attribute-level results refer to the micro-F1 score calculated between the ground truth answer and the model-generated answer for each query/interested attribute. Domain-level results refer to the micro-F1 score calculated between the ground truth answer and the model-generated answer for all query/interested attributes in each domain. We determine whether the generated answer is correct by checking whether the generated answer contains the true answer.










§ DETAILED ERROR ANALYSIS AND REMAINING CHALLENGES


We have conducted an exhaustive analysis of cases incorrectly predicted by various models, with a particular focus on GPT-4V. Representative error cases for different domains and attributes are presented in Figure <ref> and Figure <ref>. Here we provide a more detailed error analysis from different perspectives:




  
Attribute-Level:
(1) Models often confuse attribute values that are similar yet distinct, such as `3/4 Sleeve' versus `Long Sleeve' in cases 1-2, `Short Sleeve' versus `Sleeveless' in cases 3-4, and `Crew Neck' versus `Scoop Neck' in case 8.
(2) Attributes that demand a detailed understanding of small image parts typically challenge models, leading to errors. For instance, mistakes in identifying `Shoulder Style’ in cases 5-6 and `Neckline’ in cases 7-9.
(3) Errors can also arise from conflicting modality inferences, as seen in case 13, where the word `Snow Village' in the product text suggested Christmas, but the image aligned more with Halloween.



  
Domain-Level:
(1) `Clothing' is the hardest domain for most models because it contains many attributes that require fine-grained understanding of product images. For example, ‘Sleeve Style’ in cases 1-4 and ‘Neckline’ in cases 7-12.
(2) While easier domains such as `Home' usually consist of attributes that only need a more global understanding of product image and text context, such as ‘Special Occasion’ in cases 13-16, ‘Shape’ and ‘Material’ in cases 17 and 21.



  
Model-Level:
(1) Open-source models are not good at recognizing and leveraging text in images. Taking case 19 as an example, Qwen-VL, DEFLATE, and LAVIN fail to utilize the text words ‘BATHROOM, Teeth, Toilet’ in the image.
(2) Interestingly and uniquely, when GPT-4V considers none of the provided options suitable, it will answer `None' and then give an answer it feels is a better match, as shown in cases 6, 18.
(3) LLaVA 1.5 tends to provide multiple answers in ambiguous situations, as can be seen in cases 16, 18.


Inspired by the above error cases, we point out some remaining challenges and opportunities:



  
Model-Aspect:
(1) Enhance the ability to understand image details, including small areas and text in images; (2) Devise mechanisms to distinguish similar attribute values; (3) Properly handle conflicting modality inferences; (4) Reduce the performance gap in implicit AVE between open-source models and advanced closed models like GPT-4V.


  
Dataset-Aspect:
Our ImplicitAVE dataset does not consider multi-valued attributes and negative instances, i.e. “none” as attribute values. We leave this extension for future work.







§ IMPLEMENTATION DETAILS

All open-source MLLMs are evaluated on a single A100 GPU. Due to RAM and Disk space constraints, all model pre-trained weights were loaded in at 4 or 8 bits using the bitsandbytes library for quantization. All model variants using the Vicuna-13B or Flan-T5-XXL are loaded in at 4 bits, and all other models are loaded in at 8 bits. Additionally, eight different prompts are tested, with the best performance reported for each open-source model (BLIP2, InstructBLIP, LLaVA, LLaVA 1.5, Qwen-VL). A list of valid attribute value options is provided when prompting the MLLMs. For all settings, we use micro-F1/accuracy as evaluation metrics. The prompt templates are available in the Appendix <ref>.






§ PROMPT TEMPLATES



Table <ref> provides our prompt templates for all zero-shot methods except GPT-4V. The best results are displayed. The prompt template we use for GPT-4V is: “What is the {attribute_names} of this product? Context: [Category] {category} {texts}. Choose the most appropriate one from the options: {Options}.”


























































