Seeing Beyond Classes: Zero-Shot Grounded Situation Recognition via Language Explainer
Jiaming Lei Lin Li Chunping Wang Jun Xiao Long Chen













printacmref=false



  ^1Zhejiang University     ^2Finvolution Group     ^3The Hong Kong University of Science and Technology
  
  
  


† Corresponding author 

































Benefiting from strong generalization ability, pre-trained vision-language models (VLMs), , CLIP, have been widely utilized in zero-shot scene understanding. Unlike simple recognition tasks, grounded situation recognition (GSR) requires the model not only to classify salient activity (verb) in the
image, but also to detect all semantic roles that participate in the action. This complex task usually involves three steps: verb recognition, semantic role grounding, and noun recognition. Directly employing class-based prompts with VLMs and grounding models for this task suffers from several limitations, , it struggles to distinguish ambiguous verb concepts, accurately localize roles with fixed verb-centric template[The class-contained/verb-centric template contains both verb class and its associated semantic roles, which is provided in the dataset.] input, and achieve context-aware noun predictions. In this paper, we argue that these limitations stem from the model's poor understanding of verb/noun classes. To this end, we introduce a new approach for zero-shot GSR via Language EXplainer (LEX), which significantly boosts the model's comprehensive capabilities through three explainers: 1) verb explainer, which generates general verb-centric descriptions to enhance the discriminability of different verb classes;
2) grounding explainer, which rephrases verb-centric templates for clearer understanding, thereby enhancing precise semantic role localization; and 3) noun explainer, which creates scene-specific noun descriptions to ensure context-aware noun recognition. By equipping each step of the GSR process with an auxiliary explainer, LEX facilitates complex scene understanding in real-world scenarios. Our extensive validations on the SWiG dataset demonstrate LEX's effectiveness and interoperability in zero-shot GSR.






<ccs2012>
<concept>
<concept_id>10010147.10010178.10010224.10010225.10010227</concept_id>
<concept_desc>Computing methodologies Scene understanding</concept_desc>
<concept_significance>500</concept_significance>
</concept>
</ccs2012>


[500]Computing methodologies Scene understanding

























Seeing Beyond Classes: Zero-Shot Grounded Situation Recognition via Language Explainer
    Jiaming Lei^1, Lin Li^1,3†, Chunping Wang^2, Jun Xiao^1, Long Chen^3
    
======================================================================================





§ INTRODUCTION


Traditional recognition methods heavily rely on the quality of vast dataset annotations, often limited in generalizing across diverse or unseen scenarios <cit.>. Zero-shot learning emerges to enable models to identify classes they have never been directly trained on, thus greatly enhancing their universality in real-world settings <cit.>. Recently, advancements in pre-trained vision-language models (VLMs) <cit.>, , CLIP <cit.>, have shown excellent generalization capabilities across various recognition tasks, setting new benchmarks in zero-shot learning. Particularly, CLIP incorporates dual encoders: an image encoder and a text encoder. The former processes visual input into visual features, and the latter translates texts into semantic features. This architecture facilitates the alignment of visual and text data within a unified semantic space. Leveraging class-based prompts like “A photo of [NOUN CLASS]” or “A photo of [VERB CLASS]”, CLIP effectively compares images against prompts in the learned semantic space, enabling the zero-shot recognition of both unseen objects <cit.> and actions <cit.>.





Nevertheless, training-free zero-shot grounded situation recognition (ZS-GSR) presents a more intricate challenge than basic object or action recognition <cit.>. It requires not only identifying actions (verbs) depicted in images but also discerning and localizing the semantic roles involved <cit.>. Such a comprehensive task contributes to effectively understanding what is happening (, buying in Figure <ref>), who is involved (, AGENT and SELLER), where it is taking place (, PLACE), Consequently, ZS-GSR extends beyond simple class recognition to a structured understanding of scenes, requiring accurate modeling of the relationships among objects within an event. As illustrated in Figure <ref>, a typical GSR approach typically consists of three steps:

     
  * Verb Recognition. This step aims to identify the verb category of the entire image. The most common zero-shot method for this sub-task uses a set of class-based prompts “a photo of [VERB CLASS]” for each verb category <cit.>. These prompts are then fed into the text encoder to obtain semantic embedding,
while the whole image is passed through an image encoder to get visual embedding. By comparing these two embeddings in the same semantic space, CLIP can achieve zero-shot verb recognition.

  * Semantic Role Grounding. 
Given the predicted verb class and its predefined templatefootnote:template, this step aims to precisely localize each semantic role within the image. Existing methods for this sub-task often adopt an open-world grounding model, , Grounding DINO <cit.>, with an object-contained grounding language as input <cit.>. As for GSR, the input can be the provided templates in the dataset, , the template of buys: “the AGENT buys GOODS with PAYMENT from the SELLER in a PLACE”.

  * Noun Recognition. This step aims to identify the noun category for each semantic role. Similar to the verb, a most direct zero-shot method can use noun class-based prompts like “A photo of {NOUN CLASS}” to compare with visual features of grounded semantic role for classification <cit.>. For example in Figure <ref>, the grounded semantic role AGENT is classified as woman.









However, these simple methods that rely entirely on class-based prompts at every step are limited by: 1) Ambiguous Action Concepts: Foundational VLMs primarily concentrate on understanding images or bag of entities, often overlooking the semantics and structures inherent in actions <cit.>. Directly classifying verbs with class-based prompts may not fully capture the nuanced meanings of actions, leading to CLIP misunderstanding the verb’s embedding and generating inaccurate predictions. For instance in Figure <ref> (a), the model projects the two embeddings into adjacent locations in semantic space and recognizes an activity as “coloring” due to the colorful background. It overlooks the actual activity of “studying” that entails a complex interaction with stationery and books. 2) Constrained Role Grounding: The effectiveness of semantic role grounding within visual grounding models critically depends on the quality of the text prompt <cit.>. The employment of fixed and simplex templatesfootnote:template for semantic role grounding is inherently constrained by the simplified sentence structure and ambiguous verb. Once the model encounters unfamiliar or complex verb classes, it easily leads to misalignment and localization errors for the involved roles. As depicted in Figure <ref> (b), with templates centered on the unfamiliar “climb” class, grounding models have difficulty in locating the “TOOL”.
3) Context-agnostic Noun Prediction: Noun prediction is often conditioned on the specific verb and semantic role present within a scene <cit.>. However, when employing class-based prompts for noun classification, CLIP tends to concentrate on the category with the highest prediction confidence. This method overlooks the crucial consideration of whether the noun accurately fits the specific role defined in the scene's template, leading to contextually inappropriate predictions. For example, in Figure <ref> (c), VLMs directly recognize the “hand” that occupies the main area of the image, ignoring the characteristics of the AGENT.

In this paper, we argue that these limitations stem from the model's insufficient understanding of individual classes. When humans encounter an unfamiliar word, such as “soliloquy”, we often turn to a dictionary to serve as an explainer, providing a clear and accessible definition that illuminates the term's meaning from a common understanding perspective. For instance, the term “soliloquy” in a theatrical script refers to “a character speaking their thoughts aloud when alone or regardless of any hearers”. Fortunately, recent advancements in large language models (LLMs), , GPT <cit.>, have benefited from extensive training across diverse datasets, endowing them with broad world knowledge.

Inspired by human reliance on external sources for deeper understanding, we propose the method for zero-shot grounded situation recognition via Language EXplainer (LEX). It employs LLMs to serve as explainers at crucial steps of the GSR process. Specially, for verb recognition, we devise a verb explainer 
    < g r a p h i c s >
 to prompt LLMs to generate general verb-centric descriptions. This approach provides CLIP with multiple enriched “explanations” of each verb, , “books or notes displayed” in Figure <ref> (a), leading to more accurate verb classification. Additionally, we implement an offline description weighting strategy that takes into account the discriminability of the verb-centric descriptions without any training data. As for semantic role localization, we design a grounding explainer 
    < g r a p h i c s >
 to prompt the LLMs to rephrase the original verb-centric template from multiple perspectives. These reformulated sentences, as seen in Figure <ref> (b), serving as text inputs for the grounding model, are capable of enhancing the model's understanding of the intricate relationships between semantic roles. Lastly, for noun recognition, we propose a noun explainer 
    < g r a p h i c s >
 to prompt LLMs to generate noun descriptions conditioned on both verb and semantic roles. As is illustrated in Figure <ref> (c), noun explainer prevents the misclassification of a noun such as “hand” when the contextual role and action pertain to “woman” applying lotion, thus mitigating the risk of contextually inappropriate classifications.

To verify the effectiveness of our LEX, we conduct
extensive experiments and ablation studies on SWiG dataset. Each of our explainers can be utilized as a plug-and-play module to improve the overall performance of zero-shot GSR.

In summary, we make the following contributions in this paper:

 
  * We propose a novel training-free zero-shot GSR framework LEX that incorporates LLMs as explainers within each process, enhancing understanding of complex visual scenes.
 
  * We introduce three explainers: verb explainer, grounding explainer, and noun explainer. Each serves as a plug-and-play module to enhance the performance of zero-shot GSR.
 
  * We devise a strategy to weight descriptions by discriminability, independent of training data.
 
  * Extensive experiments on the SWiG dataset demonstrate the effectiveness and interpretability of LEX.





§ RELATED WORK


Grounded Situation Recognition (GSR). 
Despite deep learning's notable achievements in image classification <cit.>, object detection <cit.>, and image segmentation <cit.>, its comprehension of complex scenes remains limited. Techniques, , scene graph generation <cit.> and human-object interaction detection <cit.>, aim to parse scene contents via relation graphs. Admittedly, these tasks can provide structured visual scene representations to assist downstream tasks <cit.>. However, they only focus on modeling with dyadic relationships between a subject and an object, ignoring the diversity of roles in visual events <cit.>.
GSR emerges as a solution for holistic scene understanding by predicting actions (verbs) and detecting noun entities of each semantic role <cit.>. Pratt  <cit.> first proposed an RNN-based two-stage framework, which detects verbs in the first stage and predicts the nouns with bounding boxes in the second stage. Subsequent methods<cit.> improved this two-stage pipeline by employing a transformer-based model and considering semantic relations among verbs and semantic roles. 
However, previous efforts that depend on annotated training samples may face challenges like sample noises <cit.> and long-tailed distribution <cit.>, potentially limiting real-world applicability. Our approach predicts verbs, nouns, and their associated bounding boxes without training data, showcasing robust generalization and interpretability in real-world scenarios.

Foundation Models. 
Foundation models are typically pre-trained on extensive training data. Owing to their strong generalization capabilities, foundation models are often applied to a variety of downstream tasks <cit.>. Particularly in the natural language processing (NLP) area, large language models (LLMs) <cit.>, such as GPT-3 <cit.>, PaLM <cit.>, OPT <cit.>, and LLaMA <cit.>, trained on extensive web-scale text datasets, exhibit formidable capabilities ranging from text prediction to contextually relevant text generation. 

In the field of cross-modal research, vision-language models (VLMs), , CLIP <cit.> and ALBEF <cit.>, have been trained on extensive image-text pairs via contrastive learning, facilitating the cohesive integration of information between these modalities. Subsequent models, such as BLIP <cit.> and BLIP-2 <cit.>, leverage diverse datasets and Q-Former's trainable query vectors to further refine this alignment. 


Despite the remarkable transferability of VLMs, prevailing approaches often rely on class-based hard prompts as input to the text encoder, leading to a narrow focus on specific visual features and disregarding contextual information in the surroundings. In this paper, we employ various prompts to guide LLMs as different explainers to generating “class explanations”, significantly enhancing VLM's understanding of intricate scenes. 





§ ZS-GSR VIA LANGUAGE EXPLAINER

Formulation. Given an image I, zero-shot GSR aims to identify a structured frame ℱ_v = {y_v, ℱℛ_v}, where y_v ∈𝒱 denotes the salient verb category, ℱℛ_v = {(r,y_n,𝐛)| r∈ℛ_v} represents the set of filled semantic roles. The ℛ_v represents the set of predefined semantic roles associated with verb-centric template TP_v.
Each role r∈ℛ_v is filled with a noun y_n ∈𝒩 grounded by a bounding box 𝐛∈ℬ.
For instance in Figure <ref>, the frame can be detected as ℱ_v= {buying, {(AGENT, woman, ), (GOODS, book, ), (PAYMENT, cash, ), (SELLER, man, ), (PLACE, store, )}}.




Baseline for Zero-Shot GSR. As mentioned above, the baseline involves three steps:  
1) Verb Recognition, the class-based prompts is fed into text encoder T(·)
of CLIP to obtain verb text features {𝐭_v}. Similarly, the entire image is fed into image encoder V(·) to extract visual feature 𝐯_v. Subsequently, the cosine similarity between 𝐭_v and 𝐯_v is computed across different verb categories for final prediction. 2) Semantic Role Grounding, Grounding DINO integrates a verb-centric template TP_v as text input and image I as visual input to generate a series of candidate bounding boxes with semantic role labels and scores. The bounding boxes with the highest score for each semantic role are selected. 3) Noun Recognition, CLIP compares cropped region feature {𝐯_n^r} of each role r with the text features {𝐭_n} of class-based prompts for noun classification. 

To address the constraints of the baseline, we propose a new framework LEX for ZS-GSR. As illustrated in Figure <ref>, it comprises three components: verb recognition via verb explainer, role localization via grounding explainer, and noun recognition via noun explainer. LEX enhances the model's understanding of different classes by adding an auxiliary explainer at each step. 







 §.§ Verb Recognition via Verb Explainer


This component aims to recognize the salient verb category within an image. It comprises three steps: verb-centric description generation, description weighting, and verb classification.



  §.§.§ Verb-Centric Description Generation
 To enhance the discriminative capacity of the CLIP model in distinguishing verb categories, we introduce a verb explainer to “explain” basic class-based prompts for each category. Inspired by zero-shot image classification <cit.>, we introduce a verb-centric description generation prompt to make LLMs as verb explainer to generate general verb-centric descriptions 𝒟_v={d_v} for verb class y_v∈𝒱 from multiple perspectives:

    𝒟_v = LLM(in-context examples, y_v, instruction)_prompt input,

where LLM(·) is the decoder of the LLM. The verb-centric description generation prompt input consists of three parts[The detailed description generation prompts are in the Appendix.]: 1) In-context examples, the description instruction for the verb, along with some examples of the generated results. This part adopts in-context learning <cit.> to make LLM generate analogous results. 2) Verb class y_v, the target category that needs to generate the description. 3) Instruction, the sentence used to command the LLM to generate the description, , “what are the useful visual features for the event of `rehabilitating': AGENT rehabilitates ITEM at a PLACE”.

These generated descriptions highlight various unique visual features associated with specific action categories, for example, “looking around for searching goods”, “pushing a shopping cart or carrying a basket”, contribute to enhancing the distinguishability between “shopping” and other similar actions like “buying”.



  §.§.§ Description Weighting
 
 
Considering the variances in description quality and its different impact on verb classification, we devise a description weighting strategy.
Recent methods try to select descriptions from the perspective of the discrimination of image features <cit.> and the coverage among concepts <cit.>. However, these methods require a large number of annotated training images and are prone to overfitting to seen scenarios, which are not suitable for training-free zero-shot GSR. Thanks to the property of VLMs to align visual and text modes in a shared space, we can replace an annotated image with a complex scene description that contains the verb category <cit.>. 



Specifically, we first utilize the LLMs to generate a set of complex scene descriptions 𝒟_s|v= {d_s} for each verb y_v ∈𝒱. These scene descriptions are fed into the text encoder T(·) to generate scene text embeddings {𝐭_s}, used as “image” for subsequent description weightingfootnote:description. For instance in Figure <ref>, the verb biting's scene description is used instead of an annotated image.

Intuitively, the greater the distinctiveness of a verb description in differentiating between its own class of scenes and other classes, the more important it is. Following <cit.>, we apply the discriminability score Dis(·) to measure the distinctiveness of a verb description. The correlation ρ(·,·) between a verb description embedding and a set of scene text embeddings with class y_v is denoted as:

    ρ(y_v, d_v) = 1/|𝒟_s|v|∑_d_s∈𝒟_s|vϕ (𝐭_s,𝐭_d_v),

where ϕ(·,·) denotes the cosine similarity, 𝐭_d_v is the text embedding of a verb description d_v. The conditional likelihood of aligning scenes of a class given d_v can be written as: 

    ρ (y_v,d_v) = ρ(y_v,d_v)/∑_y_v^'∈𝒱ρ(y_v^',d_v).

The discriminability score Dis(d_v) can be measured by the entropy over all verb classes, written as: 

    Dis(d_v) =  - ∑_y_v^'∈𝒱ρ (y_v^'|d_v)log (ρ (y_v^'|d_v)).

The smaller the discriminability score, the greater the distinction d_v offers for its category. Hence, the weight of a general verb description can be denoted as:

    w_v(d_v) = exp(1/Dis(d_v))/∑_d_v^'∈𝒟_vexp(1/Dis(d_v^')),

where w_v(d_v) represents the weight of verb description d_v.



  §.§.§ Verb Classification
 
 
In this step, we compute the similarity score between visual and textual features to obtain the probability distribution of verbs. We calculate the final distribution as follows:

    Score(y_v) = (1-λ)ϕ(𝐯_v,𝐭_v)_class-based + λ∑_d_v∈𝒟_vw_v(d_v) ϕ(𝐯_v,𝐭_d_v)_description-based,

where λ is utilized as a balance factor between classed-based prompts and description-based prompts. 






 §.§ Role Localization via Grounding Explainer
 This module aims to locate each semantic role more accurately via an auxiliary grounding explainer. Specifically, we first employ a grounding explainer to rephrase the verb-centric templates in a more comprehensible manner. Subsequently, we use the Grounding DINO to generate candidate bounding boxes for the semantic roles.





  §.§.§ Rephrased Template Generation


Due to the ambiguity of the original fixed verb-centric template, using it as grounding text input for the Grounding DINO model may result in limited or inaccurate semantic role localizations. 
To enhance the comprehensibility of grounding language, we introduce a grounding explainer to “explain” these fixed templates. To be specific, we employ a rephrased template generation prompt, utilizing LLM as grounding explainers to produce rephrased grounding templates 𝒟_g={d_g} for each verb class y_v∈𝒱, expressed as:

    𝒟_g = LLM(in-context examples,  TP_v, instruction)_prompt input.

Similarly, the rephrased template generation prompt input also comprises three components: in-context examples, template TP_v, and instructionfootnote:description. The first two components are analogous to verb-centric description generation. As for the last component, the instruction sentences are designed to enable LLM to generate role-contained templates that are easy to understand, , “Please generate new sentences that detailedly rephrase this sentence to make it easier to understand {TEMPLATE}. Note that you must keep the original words: {SEMANTIC ROLES}”. 



  §.§.§ Candidate Box Generation
Given an unlabeled input image I and the grounding text input d_g, the Grounding DINO model is adept at associating visual features with corresponding object labels (, semantic roles). More concretely, the Grounding DINO model outputs a sequence of bounding boxes ℬ_g ={𝐛}, associated object labels ℛ_g = {y_r}, and confidence scores 𝒞_g={c} as follows: 





    ℬ_g, ℛ_g, 𝒞_g = DINO(I,d_g),






where DINO(·) is the decoder of Grounding DINO. It is possible to generate multiple boxes for each object. We select the box with the highest confidence score c for each object y_r that appeared in the predefined role set R_v as candidates. 




The chosen ℬ_g of each grounding description d_g constitutes the candidate boxes ℬ = {ℬ_g|d_g ∈𝒟_g}. These candidate bounding boxes of semantic roles are utilized for the next noun recognition.





 §.§ Noun Recognition via Noun Explainer
 
In this module, we utilize the noun explainer to achieve context-aware noun classification. To be specific, it consists of four steps: noun filtering, scene-specific noun description generation, noun pre-prediction, and noun refinement, as displayed in Figure <ref>. 





  §.§.§ Noun Filtering
 

This step filters out noun categories that are unlikely to appear in the specific semantic roles from class-level.
Leveraging the common sense contained in LLM, we can inquire about reasonable noun categories through a simple prompt that includes a specified semantic rolefootnote:description. For instance, we can query “Given entity list {Noun CLASS}, which entities are most likely to be the result of a predicted semantic role PLACE” to filter unreasonable noun categories, , man. 
This filtering step not only enhances the computational efficiency of subsequent noun classification by reducing the number of traversed categories but also improves the accuracy of classification results.





  §.§.§ Scene-Specific Noun Description Generation
 In particular, the cropped candidate box inevitably contains more than one object, and the object normally has more than one possible category. Reliance on class-based prompts often results in CLIP focusing on the most salient noun category directly, thereby ignoring the specific scene context. To address this, we incorporate a noun explainer to “explain” class-based prompts associated with each noun category within diverse scenesfootnote:description. This noun explainer adopts a scene-specific noun description generation prompt, guiding LLM to produce descriptions 𝒟_n^r={d_n} that reflect the intricate context of each scene for each noun class y_n∈𝒩:

    𝒟_n^r=LLM(in-context examples, scene, y_n, instruction)_prompt input.


Different from verb-centric description generation, the input prompt for scene-specific noun description generation is conditioned on extra scene information (verb-centric template and semantic role) to provide contextual information. The instruction is like this: “Please describe the visual features that can distinguish the noun entity {NOUN CLASS} corresponding to {SEMANTIC ROLE} in the scene: AGENT writes on TARGET using a TOOL at a PLACE”. The generated descriptions (, “holding a pen and other tools in hand”) reflect the characteristic of corresponding semantic roles (, AGENT), thereby ensuring more accurate and contextually relevant noun categorization.





  §.§.§ Noun Pre-prediction


In this step, we first compute the similarity between visual embeddings of all cropped boxes and text embedding to obtain the probability distribution of nouns. Similar to the verb classification process, this entails the incorporation of both class-based and description-based approaches to yield the pre-predict results for nouns:

    Score(y_n) =(1-λ)ϕ(𝐯_n^r,𝐭_n)_class-based  +λ∑_d_n∈𝒟_n^rw_n(d_n) ϕ(𝐯_n^r,𝐭_d_n)_description-based,

where 𝐭_d_n is the text features of the noun description d_n. 
Due to the requirement of invoking a large number[Since noun description is conditioned on both verb and semantic role, the call number of LLMs at least is  |𝒱|× |ℛ_v|× |𝒩| ] of LLMs for generating scene descriptions akin to description weighting in verb recognition, the weights w_n(d_n) are uniformly assigned as 1/|𝒟_n^r|.



  §.§.§ Noun Refinement


After localization by Grounding DINO, a semantic role may correspond to multiple candidate boxes due to the integration of different d_g ∈𝒟_g. Directly selecting the box with the highest confidence often relies solely on the localization model's understanding of the semantic role, overlooking the contextual relevance within the scene. To mitigate this, we propose to refine noun classification and bounding box selection via a global comparison. This process contrasts the visual features of the entire image 𝐯_v against the text features {𝐭_tp} of templates {TP_v} whose semantic roles are filled iteratively with predicted nouns {y_n} in the previous step, , “woman is studying in library” in Figure <ref>. The noun filled in the template with the highest similarity along with its corresponding bounding box is the final prediction, written as:

    TP_v^*   = maxϕ(𝐯_v,{𝐭_tp})
    ℱℛ_v    = { (r,y_n,b)|y_n∈TP_v^*}.

As seen in Figure <ref>, the noun “woman” for the semantic role “AGENT” and its bounding box is selected as the final prediction result.



§ EXPERIMENT





 §.§ Experimental Setup



  §.§.§ Dataset
 We evaluated our method on the SWiG dataset <cit.>, which extends the imSitu dataset <cit.> by incorporating additional bounding box annotations for all visible semantic roles (69.3% of semantic roles have bounding boxes). In the SWiG dataset, each image is annotated with a verb, followed by a set of semantic roles ranging from 1 to 6 (3.55 on average), each verb is annotated with three verb frames by three separate annotators. SWiG contains 25200 testing images with 504 verb categories, 190 semantic role categories, and 9929 noun entity categories.



  §.§.§ Evaluation Metrics
 We employed the same five evaluation metrics as <cit.>, which include: 1) verb: the accuracy of verb prediction. 2) value: the noun prediction accuracy for each semantic role. 3) value-all (val-all): the noun prediction accuracy across the entire set of semantic roles. 4) grounded-value (grnd): the accuracy of the bounding box for each semantic role, where the predicted bounding box must achieve an IoU value of at least 0.5 with the ground-truth bounding box. 5) grounded-value-all (grnd-all): the accuracy of bounding box across the entire set of semantic roles. Besides, these metrics were presented under three evaluation settings: 1) Top-1-verb, 2) Top-5-Verb and 3) Ground-Truth-Verb, verbs are selected based on the top-1 prediction, top-5 predictions, and corresponding ground-truth, respectively. If the predicted verbs are incorrect in the Top-1/5-verb settings, the other four metrics (value, val-all, grnd, and grnd-all) are considered incorrect.





  §.§.§ Implementation Details
 We used the vision transformer with a base configuration (ViT-B/32) as the default backbone for CLIP. We adopted GPT-3.5-turbo for LLM, recognized as a widely utilized LLM in existing works. More details are left in the Appendix.










  §.§.§ Baselines
 We compared our proposed method LEX with four strong baselines: 1) CLS, which uses class-based prompts to calculate the text embedding for both verb and noun classification. Leverage verb-centric template as the text input of Grounding DINO for role grounding. 2) TEM, which uses the verb-centric template to enhance the original class-based prompts for verb classification, compared with CLS. 3) CLSDE, which generates general descriptions of each noun category without considering scene and role by LLMs to enhance the original class-based prompts for noun classification, different from CLS. 4) RECODE, which generates semantic role descriptions as visual cues to enhance the original class-based prompts for verb classification, compared to CLS. 




 §.§ Quantitative Comparison Result

We evaluated the performance of our proposed LEX and four strong baselines on the test set of SWiG dataset. From Table <ref>, we have the following observations: 1) The CLS baseline relying solely on class-based prompts, demonstrates inferior performance, particularly in the prediction of bounding boxes and nouns. 2) In the Top-1 and Top-5 verb settings, TEM exhibits a slight performance advantage over CLS, attributed to the enhanced scene information for verb recognition. 3) CLSDE achieves a slight performance gain in the accuracy of noun classification because general noun descriptions often fail to apply in specific contexts. 4) RECODE fails to improve the accuracy of verb recognition, likely due to the insufficient local information provided by semantic roles (, not clear and detailed enough to understand actions). 5) The proposed LEX exhibits significant performance gains across all metrics compared to all baseline models, , 32.41% and 58.34% in terms of verb accuracy under Top-1-Verb and Top-5-Verb, and 29.92% noun accuracy under Ground-Truth, respectively. This indicates the effectiveness of using explainers in zero-shot GSR.





 §.§ Ablation Studies










As aforementioned, each component of our proposed LEX framework can serve as a plug-and-play module for zero-shot GSR. This section[More ablation studies are left in the Appendix.] ablated all the proposed components on the test split of SWiG dataset <cit.>. 





  §.§.§ Key Components in Verb Recognition
 We first investigated the two major elements utilized in verb recognition: 1) Verb Explainer, which denotes using verb-centric descriptions for verb classification (Sec. <ref>); and 2) Weighting, which denotes the employment of description weighting for the text embeddings of all generated verb-centric descriptions (Sec. <ref>).  Results are shown in Table <ref>. The first row corresponds to the performance of the CLS baseline. As seen, CLS achieves 30.18% top-1 and 55.49% top-5 verb accuracy. Upon applying the proposed verb explainer (the second row), we observe consistent and substantial improvements for both top-1 accuracy (30.18% → 32.13%) and top-5 accuracy (55.49% → 57.84%). This highlights the importance of our “recognition with explainer” strategy and validates the viability of our motivation. Moreover, LEX achieves better performance across all metrics with the weighting strategy. This indicates that the proposed verb explainer and weighting strategy can work synergistically.



  §.§.§ Key Component in Semantic Role Grounding
 We next studied the impact of our Grounding Explainer, which adopts rephrased verb-centric templates as grounding input (Sec. <ref>). As illustrated in Table <ref>, our grounding explainer proves to be effective, , 0.81% accuracy improvement in terms of the grounding (grnd) metrics.













  §.§.§ Key Components in Noun Recognition

We further analyzed the influence of three major components proposed for noun recognition: 1) Filter, which denotes filtering those unreasonable nouns by using LLMs (Sec. <ref>); 2) Noun Explainer, which denotes using scene-specific noun descriptions for noun prediction (Sec. <ref>); and 3) Refine, which denotes using contextual information of the scene to refine final results (Sec. <ref>). From the results in Table <ref>, we have the following findings: 1) Filtering those unreasonable noun candidates leads to significant performance gains across all noun metrics(, 15% in value metrics and 3.64% in val-all metrics). 2) Substantial improvements can be made by incorporating the noun explainer which aims to provide more contextual information (, 28.51% → 29.16% in value metrics). 3) Furthermore, after incorporating the refine module, we achieve considerable gains of 0.88% in value metrics and 1.51% in grnd metrics. 4) Finally, by integrating three core components, LEX delivers the best performance across all metrics. This validates the effectiveness of our comprehensive modular designs.









  §.§.§ Different Architectures of CLIP
 Last, we examined the impact of the utilized CLIP's architectures. As outlined in Table <ref>, regardless of the visual encoder employed, our LEX demonstrates consistent and substantial improvements across all metrics. Such impressive results further verify the robustness of our approach.









 §.§ Qualitative Comparison Result


We visualized CLIP's attention maps for various images and query prompts in Figure <ref>. As is displayed in Figure <ref> (a), we can observe that class-based approaches might focus on areas irrelevant to the query verb. For example, given the class-based prompt for the verb “sneezing”, CLIP wrongly attends to the child's hair and arms. With the detailed guidance of verb-centric descriptions, CLIP can focus on the correct areas, which indicates the importance of providing more contextual information instead of only class names. Similar conclusions can be drawn in terms of noun recognition, as depicted in Figure <ref> (b). More qualitative results are left in the Appendix.



§ CONCLUSION

In this paper, we proposed a novel approach LEX for zero-shot GSR utilizing large language models to provide rich, contextual explanations at each critical stage of the GSR process. By innovating beyond the constraints of conventional class-based prompt inputs, our approach leverages LLMs to clarify ambiguous actions, accurately ground semantic roles, and ensure context-aware noun identification. Furthermore, our framework enhances model interpretability and adaptability across varied visual scenes without requiring direct training on annotated datasets. Additionally, we introduced a description weighting mechanism to measure the contribution of generated descriptions and assign corresponding weights, thereby enhancing the discriminative power of the method. Extensive validation on the SWiG dataset affirmed the effectiveness and interoperability of our proposed LEX. 




















ACM-Reference-Format



