Efficiency in Focus: LayerNorm as a Catalyst for Fine-tuning Medical Visual Language Pre-trained Models
Jiawei Chen Dingkang Yang Yue Jiang Mingcheng Li Jinjie Wei Xiaolu Hou Lihua Zhang







  Academy for Engineering and Technology, Fudan University
   




indicates the corresponding authors.

  Academy for Engineering and Technology, Fudan University
   





  Academy for Engineering and Technology, Fudan University
   



  Academy for Engineering and Technology, Fudan University
   



  Academy for Engineering and Technology, Fudan University
   


[1]


  Academy for Engineering and Technology, Fudan University
   






In the realm of Medical Visual Language Models (Med-VLMs), the quest for universal efficient fine-tuning mechanisms remains paramount, especially given researchers in interdisciplinary fields are often extremely short of training resources, yet largely unexplored.

Given the unique challenges in the medical domain, such as limited data scope and significant domain-specific requirements, evaluating and adapting Parameter-Efficient Fine-Tuning (PEFT) methods specifically for Med-VLMs is essential. 
Most of the current PEFT methods on Med-VLMs have yet to be comprehensively investigated but mainly focus on adding some components to the model's structure or input. However, fine-tuning intrinsic model components often yields better generality and consistency, and its impact on the ultimate performance of Med-VLMs has been widely overlooked and remains understudied.

In this paper, we endeavour to explore an alternative to traditional PEFT methods, especially the impact of fine-tuning Layer Normalization (LayerNorm) layers, Feedforward Networks and Attention layers on the Med-VLMs. Our comprehensive studies span both small-scale and large-scale Med-VLMs, evaluating their performance under various fine-tuning paradigms across tasks such as Medical Visual Question Answering and Medical Imaging Report Generation. The findings reveal unique insights into the effects of intrinsic parameter fine-tuning methods on fine-tuning Med-VLMs to downstream tasks and expose fine-tuning solely the LayerNorm layers not only surpasses the efficiency of traditional PEFT methods but also retains the model's accuracy and generalization capabilities across a spectrum of medical downstream tasks. The experiments show LayerNorm fine-tuning's superior adaptability and scalability, particularly in the context of large-scale Med-VLMs. We hope this work will contribute to the ongoing discourse on optimizing efficient fine-tuning strategies for Med-VLMs. The code will be released upon acceptance.







<ccs2012>
   <concept>
       <concept_id>10010147.10010257.10010293.10010294</concept_id>
       <concept_desc>Computing methodologies Neural networks</concept_desc>
       <concept_significance>500</concept_significance>
       </concept>
   <concept>
       <concept_id>10002951.10003227.10003251.10003255</concept_id>
       <concept_desc>Information systems Multimedia streaming</concept_desc>
       <concept_significance>300</concept_significance>
       </concept>
   <concept>
       <concept_id>10010147.10010178.10010179.10003352</concept_id>
       <concept_desc>Computing methodologies Information extraction</concept_desc>
       <concept_significance>300</concept_significance>
       </concept>
 </ccs2012>


[500]Computing methodologies Information extraction
[300]Computing methodologies Neural networks
[300]Information systems Multimedia streaming
























Efficiency in Focus: LayerNorm as a Catalyst for Fine-tuning Medical Visual Language Pre-trained Models
    Lihua Zhang
    
=======================================================================================================



§ INTRODUCTION



Visual language models (VLMs) have become pivotal in facilitating multimodal tasks within the medical domain, such as medical visual question answering (Med-VQA) and medical imaging report generation (Med-IRG). The pretraining-finetuning paradigm, heralded for its success in domain adaptation and transfer, now stands as the predominant training approach for Medical VLMs (Med-VLMs). Nonetheless, the substantial data and computational resource demands for VLMs' pretraining pose significant challenges. Despite the success of visual language pre-training paradigms like CLIP <cit.> and BLIP <cit.> fostering a series of open-source medical visual language pre-trained (VLP) models contributed by the community, adapting these models for specific downstream tasks remains a formidable task for those constrained by resource availability. Especially considering the inherent minor variability in medical imaging across different medical centers or imaging devices, which necessitates researchers to frequently and rapidly fine-tune medical VLP models (Med-VLPs).



The recent surge in Large Visual Language Models (LVLMs) has exacerbated these challenges. Although a series of Parameter-Efficient Fine-Tuning (PEFT) methods <cit.> have been developed in the Large Language Model (LLM) domain, their applicability and effectiveness in the context of LVLMs are yet to be ascertained<cit.>. Some empirical studies<cit.> have shown that the performance of certain PEFT methods contradicts their demonstrated competitiveness in the original LLM domain when fine-tuning domain-specific LVLMs because fine-tuning parameters for different parts of the LVLM can have drastically different effects. Distinct from universal domains, the medical field presents unique challenges, such as limited dataset size and specialized data characteristics, which necessitate a tailored approach to fine-tuning. These domain-specific requirements underscore the need for a dedicated evaluation of PEFT methods on Med-VLMs to ensure their efficacy and appropriateness for medical tasks. Even so, research on the impact of imposing different PEFT methods on different scale Med-VLMs for different tasks remains completely blank. Meanwhile, current PEFT methods typically focus on domain adaptation by adding extra components to the model structure (i.e., Reparameterized Fine-tuning) <cit.> or prefixes to the input (i.e., Additive Fine-tuning) <cit.>, while the role of fine-tuning intrinsic structural parameters of models has been widely neglected, especially in vertical domain fine-tuning. As shown in Figure <ref>, extrinsic tuning methods focus on concatenating additional components to the dense layer or adding prefixes to the inputs while intrinsic tuning methods select the intrinsic units of the transformer to be tuned.


In this paper, we focus on efficiently fine-tuning Med-VLPs for specific downstream tasks, aiming to explore an innovative method that achieves task adaptation by fine-tuning a small subset of the model's intrinsic parameters. To find a universally efficient fine-tuning method applicable to various types of Med-VLMs, regardless of their scale, we turn our attention to common foundational layers in transformer-based components, including Attention layers, Feed-Forward Neural Networks (FFN), and Layer Normalization (LayerNorm) layers. Through systematic experiments, we demonstrate that LayerNorm can serve as the most efficient method for fine-tuning Med-VLPs. To further explore the efficiency, adaptability, and performance of fine-tuning the intrinsic parameters for transferring Med-VLPs to downstream tasks, we conduct extensive evaluations across both large-scale and small-scale Med-VLPs, encompassing core medical visual language tasks like Med-VQA and Med-IRG. Our findings expose the variability of intrinsic parameter fine-tuning methods in fine-tuning Med-VLMs to a downstream task that is different from traditional extrinsic parameter fine-tuning methods. We underscore not only the superior efficiency of LayerNorm fine-tuning over existing PEFT methods but also its remarkable adaptability and transferability for fine-tuning Med-VLPs across diverse downstream tasks. The main contributions of this article are as follows:



  * To our best knowledge, we are the first to centre on fine-tuning a small subset of the Med-VLP's inherent parameters to adapt to downstream tasks. 


  * We conduct a comprehensive series of experiments fine-tuning foundational components of Med-VLMs, including systematic comparisons with existing PEFT methods centred on tuning extrinsic components.


  * Our research identifies LayerNorm fine-tuning as a highly efficient strategy for adapting Med-VLPs to specific downstream tasks within the medical domain. 






§ RELATED WORK




 §.§ Medical Visual Language Models


In the medical domain, Med-VLMs play a pivotal role in automating visual-language tasks, such as VQA and IRG. Initially, these models <cit.> leverage Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) to extract visual and linguistic features separately. Yet, such approaches frequently fell short in terms of generalizability and transferability across different tasks due to the limitations of their structure. Driven by learning-based technologies <cit.>, modern Med-VLMs <cit.> primarily adopt the transformer architecture, following a pretraining-finetuning paradigm. They undergo initial pretraining on extensive, generalized medical image-text pair datasets, followed by comprehensive fine-tuning on more focused, task-specific datasets. For example, MISS <cit.>, utilizing the ALBEF <cit.> methodology, begins its training on 38,800 selectively curated image-text pairs from the MedICaT dataset<cit.> before undergoing fine-tuning for VQA tasks. Similarly, LLaVA-Med <cit.> employs a dual-phase pretraining strategy, starting with image-text feature alignment on two million pairs from PubMed, then enhancing conversational capabilities using instruction-format data, culminating in full-scale fine-tuning for VQA tasks. These approaches consistently rely on full-model fine-tuning for task adaptation, a method that, despite its efficacy, demands substantial resources, particularly for large-scale models such as LLaVA-Med. The restricted dataset sizes available for downstream task training further jeopardize the model's generalizability, leading to potential catastrophic forgetting and diminishing its broader applicability in medical contexts.


 §.§ Efficient Fine-tuning Techniques


The fine-tuning of large-scale Pre-trained Language Models (PLMs) is a demanding process <cit.>, requiring extensive computational resources and data. To alleviate these burdens, PEFT techniques <cit.> have been introduced. These methods <cit.> typically incorporate trainable components into the PLMs while maintaining the rest of the model's parameters in a frozen state. Some strategies <cit.> also involve the nuanced manipulation of input embeddings across different layers to minimize or negate modifications to the original model's architecture.

PEFT methods have demonstrated efficacy in transitioning large-scale PLMs to new tasks or downstream applications and have been instrumental in converting LLMs into multimodal LLMs <cit.>. For instance, LLaVA <cit.> uses an MLP adapter to connect a vision feature extractor with a large language model, selectively training the MLP adapter while keeping both components static, thus adapting the LLM into a VLM.

<cit.> introduces an efficient strategy where tuning LayerNorm layers suffices to yield strong performance to transform an LLM into an LVLM. Nonetheless, the capability of existing PEFT methods to efficiently adapt pre-trained VLMs to specialized, especially medical, tasks remains largely uninvestigated. With the diverse architectures of LVLMs, the most effective application of PEFT methods is uncertain, and their generalizability to non-textual encoders/decoders is limited <cit.> (e.g., prefix-tuning and p-tuning are not viable for Vision Transformers (ViT) <cit.>). Consequently, investigating the adjustment of a model's intrinsic parameters for efficient fine-tuning emerges as a critical necessity.

In this paper, we propose a novel method that eschews adding components to the original model structure or input, focusing instead on fine-tuning the model's inherent parameters. This strategy is designed to ensure the method's broad applicability for efficient fine-tuning across various Med-VLM types.



§ PRELIMINARIES




 §.§ Mainstream Architectures of Med-VLMs


Contemporary generative Med-VLMs, irrespective of their scale—be it large-scale or small-scale, tend to follow a similar architectural framework. This typical structure comprises a vision feature extractor, a text feature extractor, a connector that integrates the former two, and a Language Model (LM) head. Most Med-VLMs opt for ViT <cit.> as the vision feature extractor, while the text encoder is based on mainstream frameworks such as BERT <cit.> or GPT <cit.>. Despite possible minor variations in their structural implementations, the transformer-based layer serves as their common denominator, with FFN, Attention mechanisms, and LayerNorm being indispensable core components. 


















 §.§ Previous PEFT Methods


Transitioning from the core mechanisms of attention and layer normalization, which provide stability and specificity within the model's architecture, we delve into the domain of extrinsic PEFT methods. These methods are categorized primarily into two types: Reparameterized Fine-tuning (i.e., LoRA (Low-Rank Adaptation)) and Additive Fine-tuning (i.e., Prefix-tuning). 


LoRA-Tuning: LLM maps data into a high-dimensional space for processing. 
 
LoRA indirectly trains the dense layers in the network by optimizing the rank-decomposition matrix that changes in the adaptation process of the dense layer, thereby achieving the best fine-tuning effect by optimizing only the rank-decomposition matrix of the dense layer.
For the pretrained parameters θ_0^D, the dense layer weight parameter matrix on a specific downstream task is defined as W_0∈ℝ^d × k and the intrinsic rank of it is θ^d; the specific downstream task's parameters θ^D is calculated as θ^D = θ_0^D+θ^dM, where M is the rank-decomposition matrix. For W_0∈ℝ^d × k,  LoRA updates it with the following equation:

    W_0+Δ W = W_0+BA,B ∈ℝ^d × r, A ∈ℝ^r × k,

where d is the output dimension of the previous layer, and k is the input dimension of the next layer. For input x, the forward propagation process is calculated as follows:

    h = W_0x+Δ Wx = W_0x +BAx.




v v0Prefix-Tuning: Inspired by the In-Context Prompting method adopted by GPT3 <cit.>, Li et al <cit.> propose the Prefix-tuning method for generation tasks. Instead of the discrete text used in prompt tuning, continuous vectors are prefixed to the input text. Specifically, the generation task is deemed as a table-to-text task, the input x is treated as a linear table and the output y represents a short text. For an encoder-decoder model, different prefixes are attached to the beginning of the encoder and decoder with the input defined as:

    z = [PREFIX, x, PREFIX'],

and the prefixes are generated by a trainable matrix P_θ∈ℝ^|P_idx| × dim(h_i), the global training objective is defined as:

    max_ϕlog P_ϕ(y|x) = max_ϕ∑_i ∈ Y_idxlog P_ϕ(z_i|h_<i).




 §.§ Medical Visual Language Tasks

Medical Visual Language Answering: The primary objective of Med-VQA is to provide answers based on professional questions posed by the inquirer regarding medical images, enhancing the understanding of medical images, and facilitating patient care through the automated interpretation of visual data. The tasks are categorized into open-ended questions, which require detailed descriptive answers, and close-ended questions, which demand concise, often binary responses like “yes” or “no”. This interdisciplinary domain requires the Med-VLM to interpret and provide insights into complex medical imagery, such as X-rays, MRI scans, and CT images. 

Medical Imaging Report Generation: Medical IRG involves the automatic creation of textual descriptions for medical images, using Med-VLMs. This task aims to analyze visual medical data and produce accurate, coherent, and clinically relevant reports. The goal is to assist radiologists and healthcare professionals by reducing their workload and improving diagnostic efficiency while maintaining high standards of patient care. 



 §.§ Difference between Universal VLMs and Med-VLMs' Downstream Tuning


The differences between Med-VLMs and Universal VLMs make it necessary to study the tuning of Med-VLMs separately in addition to using the PEFT approach for Universal VLMs.





From the data perspective, the dataset used for downstream task fine-tuning in the medical domain is extremely narrow compared to the universal domain, for example, the current largest radiological image dataset used for the Med-VQA task includes only 14,028 image-question pairs, which makes the fine-tuning of LVLMs fall into the problems of overfitting and catastrophic forgetting. At the same time, the answers of the textual Instruction pairs embedded in the dataset usually include only one or two simple words. Furthermore, the current training loss used by generative models in the fine-tuning process makes it very easy to fall into the learning of the data distribution from the long text to its short text rather than the learning of the correct image-text association. From the model perspective, most Med-VLMs are obtained by transfer learning from VLMs in the universal domain, a process where the visual coder is usually frozen, however, the domain gap between natural images and medical images affects the performance of Med-VLMs on medical tasks, where most of the current PEFT methods are not available for ViTs or or have not demonstrated their effective impact on visual encoders for transfer learning from general domain to medical domain. Therefore, a separate study of efficient fine-tuning methods for Med-VLM on downstream tasks is necessary.





§ TUNING SETTINGS




 §.§ Baseline Model



To explore a method that achieves task adaptation by fine-tuning a small subset of the model's own parameters, we choose two different-scale pre-trained Med-VLMs for different tasks: 1) small-scale VLM MISS <cit.> and 2) large-scale VLM LLaVA-Med <cit.> for Med-VQA and Med-IRG.

These baseline models cover generative Med-VLM at different scales and for different tasks so that we can provide comprehensive insights into the impact of different PEFT methods on fine-tuning Med-VLM to downstream tasks.



 §.§ Tuning within Transformer-based Layer






The transformer-based layer, serving as a fundamental structure across vision encoders, language models, and certain VLM connectors, undergoes fine-tuning through both intrinsic parameter adjustments and the application of extrinsic component fine-tuning methods. This process aims to assess the impact of various tuning approaches on the overall model performance. 


Strategic Intrinsic Adjustments: We emphasize the transformer-based layer's role as the computational core of the model and selectively fine-tune its intrinsic parameters. Attention layers, LayerNorm layers, and FFNs constitute the critical units of this layer. As depicted in Figure <ref>, when one of these three components is set to be trainable, the remaining parameters within the transformer-based layer are kept frozen. 

Incorporation of PEFT Techniques: LoRA-tuning and Prefix-tuning, representing the forefront of PEFT methods, are chosen for comparison against intrinsic parameter adjustments. Figure <ref> illustrates the application details of these PEFT methods within our study. For LoRA-tuning, low-rank matrices are selectively applied to the parameters of the query and value matrices within the attention layer, mapping data into a low-dimensional subspace for processing. In the case of Prefix-tuning, we follow prevalent practices by appending prefix vectors to the origin input x of the key and value within the attention layer, and the final input embeddings can be defined as z = [PREFIX, x]. 




 §.§ Tuning within the VLM Architecture

In our pursuit to uncover the nuanced impact of various modules within the full VLM's architecture on overall model performance, we embark on a strategic fine-tuning expedition which entails selectively training specific modules within the VLM framework while employing efficient fine-tuning methods for certain components or maintaining others in a frozen trainable state. Such a strategy allows us to dissect the individual contributions of each component to the model's efficacy in medical visual language tasks, offering insights into optimizing Med-VLMs for enhanced performance and efficiency.
This selective fine-tuning approach aims to validate the hypothesis that certain components within the Med-VLM architecture wield more significant influence over the model's performance on medical tasks. By applying focused fine-tuning strategies to individual modules, we seek to delineate the performance impact of targeted adjustments versus broad model updates.


Details for Small-scale Med-VLMs: For small-scale Med-VLMs, such as MISS, we experiment with applying efficient fine-tuning techniques to some modules at a time, with the rest of the model's parameters set to remain fully trainable This is because for small-scale Med-VLM, making either module completely frozen may make the model unable to transfer to downstream tasks <cit.>.
For instance, when the language model undergoes LayerNorm-tuning (LN-tuning), the vision feature extractor, connector, and LM head are kept in a state that allows full parameter adjustments. This strategy allows us to compare the impact of different efficient fine-tuning methods on model performance under the premise that we can evaluate the impact of fine-tuning different module parameters of small-scale Med-VLMs on the overall task performance.

Strategy for Large-scale Med-VLMs: In the case of large-scale Med-VLMs, like LLaVA-Med, our fine-tuning strategy is more nuanced, reflecting the diverse requirements of comparative analysis. As shown in Figure <ref>, “Snowflakes or flames” indicate that any module has the option to be adjusted or frozen. Depending on the specific experimental setup, modules within these larger models may be categorized into three states: fully trainable (T), efficiently fine-tuned some of the parameters (PEFT), and completely frozen (F). This flexible approach comprehensively evaluates how different tuning states across various modules influence large-scale VLMs' performance on complex medical visual language tasks.



 §.§ Downstream Fine-tuning

Benchmarks:

In this paper, we use a total of four datasets, Slake Dataset <cit.> and VQA-RAD dataset <cit.> for Med-VQA, and OpenI dataset <cit.> and MIMIC dataset<cit.> for Med-IRG. The Slake dataset consists of 14,028 QA pairs, of which 70% are used for training, 15% for validation, and 15% for testing. The VQA-RAD dataset is used for the zero-shot performance of the model on the VQA task including 3515 QA pairs, of which 451 pairs are used for testing. The OpenI dataset is used for the training of the Med-IRG task including 6,459 images and 3,955 reports, the instructions are the same as those adopted in <cit.>. The MIMIC test set was chosen for the evaluation of the model's Med-IRG performance, which includes 5,159 images and 3,269 reports. For the inference of the IRG task, we uniformly use the phrase “Describe the given chest x-ray image in detail.” as the instruction.



Application Details:

For the fine-tuning of our chosen models (MISS and LLaVA-Med), a consistent set of hyperparameters is employed to ensure uniformity across our experiments. Each model is fine-tuned with an initial learning rate of 2e-5, utilizing the Adam optimizer for its well-regarded efficiency in handling the optimization landscapes of deep learning models. Specifically, MISS underwent training for 120 epochs with a batch size of 16, adopting a weight decay of 0.05 to encourage regularization. In contrast, LLaVA-Med's fine-tuning is characterized by a warmup ratio of 0.03 and a cosine learning rate scheduler, alongside specific adjustments such as enabling tensor float 32 for enhanced computational performance, and employing FSDP strategies for memory efficiency, with settings like “full_shard auto_wra” and targeting the “LlamaDecoderLayer” for wrapping. During all the inferences, Med-LLaVA generates outputs using a set of predefined generation parameters, including sampling methods and beam search configurations (num-beams=1), and the temperature is kept at 0.2.
Detailed information regarding the hyperparameter settings for each model, along with additional configurations and the rationale for their selection, is provided in the Appendix for further reference.





§ EXPERIMENT RESULTS AND DISCUSSION



 §.§ Small-scale MISS Result







As shown in Table <ref>, we employ both supervised fine-tuning (SFT) and performance assessment on the Slake dataset's training and testing sets.
The reported performance metrics include accuracy rates for `opened' and `closed' types, which means open-ended and closed-ended questions, as well as a global accuracy rate that averages the performance across both types.

Given the potentially catastrophic impact of freezing any module on the overall performance of small-scale models, when certain modules underwent efficient fine-tuning, the remaining modules were maintained fully trainable. In the context of the MISS model, ViT, JTM, and DEC represent the visual encoder, joint text-multimodal encoder, and text decoder, respectively. The term “trainable params” refers to the total volume of trainable model parameters, with “#Params” indicating the ratio of trainable to total parameters. “PEFT params” denotes the proportion of parameters fine-tuned using PEFT methods, with “#PEFT Params” reflecting the proportion of PEFT-tuned parameters relative to the total parameters within the corresponding module.

When the baseline model is fully fine-tuned, it achieves the highest open-ended question accuracy and global accuracy rates of 82.91% and 82%, respectively. Under the premise of maintaining the visual encoder fully trainable and only efficiently tuning one module at a time, keeping the JTM encoder fully trainable enabled the model to achieve optimal performance. Compared to scenarios where DEC underwent full parameter training while JTM was efficiently tuned, the model's global accuracy rates under LayerNorm, attention, and FFN intrinsic parameter tuning methods were higher by 4%, 9%, and 12%, respectively. 


Maintaining the visual encoder fully trainable while efficiently tuning all the remaining modules resulted in significantly poor model performance, failing to correctly judge the closed-source questions in all the fine-tuning methods except Attention-tuning. Under LayerNorm, FFN, LoRA, and Prefix-tuning methods, the model never answered `yes' to any close-ended question, with accuracy rates lower than random guessing at 38.03%, 54.65%, 50.70%, and 32.95%, respectively.

Comparing different fine-tuning methods, the effect of LN-tuning is remarkable, achieving the best accuracy on close-ended questions with the lowest PEFT Params, even surpassing full parameter tuning and reaching 84.51%. In contrast, although Attention-tuning and FFN-tuning slightly outperform LN-tuning in terms of global accuracy, this came at the cost of tuning over 40% of the parameters in their respective modules. The LoRA method fine-tuning model using the [T, PEFT, T] paradigm tuned approximately five times more PEFT parameters than LN-tuning (only 56,823), with Prefix-tuning at twenty times more. This underscores the viability of LN-tuning as a comparable method to the most classical PEFT methods in small-scale fine-tuning scenarios that require saving certain parameter volumes. From a global parameter tuning perspective, Attention-tuning achieves performance closest to full fine-tuning by saving 23% of trainable parameters, marking it as another viable fine-tuning approach for small-scale Med-VLMs.




 §.§ Large-scale LLaVA-Med Result










Furthermore,
we conduct comprehensive evaluations on LLaVA-Med, a large-scale model designated for Med-VQA tasks. Our approach encompassed four distinct training paradigms: [PEFT, F, PEFT, F], [PEFT, T, PEFT, T], [F, F, PEFT, F], and [F, T, PEFT, T]. Considering the substantial parameter size of LVLMs, we aimed to restrict the volume of fine-tuning parameters to within about 40%, thereby excluding full parameter training of the ViT and FFN-tuning methods that involve adjusting ViT. Table <ref> showcases the experimental results of LLaVA-Med, trained and tested on the Slake dataset, employing the aforementioned fine-tuning paradigms.

When opting to keep both the connector and LM head trainable, the model's performance did not exhibit significant improvement, despite a substantial increase in the volume of adjusted parameters. Specifically, when fine-tuning adopted the [F, T, PEFT, T] paradigm, changes in global accuracy rates for LN, attention, and FFN tuning compared to [F, F, PEFT, F] are -0.5%, -2.1%, and -3.3% respectively. This contradicts the common notion that more parameter adjustments correlate with better SFT performance, indicating that full parameter adjustments of the connector and LM head during efficient fine-tuning of LLMs do not guarantee the expected outcomes.

The performance changes are inconsistent under the [PEFT, F, PEFT, F] and [PEFT, T, PEFT, T]. For LN-tuning, fine-tuning the image encoder led to respective increases in global accuracy of 0.57% and 0.29%, while Attention-tuning resulted in changes of -0.28% and +0.85%. Such subtle differences do not conclusively indicate whether adjusting parameters of the image encoder benefits or hinder model performance, especially when considering Recall metrics. The increase in ViT-adjusted parameter volume did not regularly alter recall, suggesting that larger adjustments to ViT parameters do not consistently improve model recall.

Comparing different intrinsic parameter adjustments revealed that increasing the volume of fine-tuned parameters indeed enhances the model's recall of generated content: as fine-tuning parameters shifted from 0.003% to 44%, recall correspondingly increased from 46.24% to 52.07%. This indicates that enlarging the volume of fine-tuned parameters allows the model to learn the distribution of ground-truth tokens in the vocabulary space more effectively, both quantitatively and spatially. However, considering accuracy—a gold standard in medical tasks—significant increases in parameter volume do not necessarily elevate all accuracy metrics concurrently. LN-tuning under the [PEFT, F, PEFT, F] paradigm once again achieved state-of-the-art (SOTA) accuracy for close-ended questions, which was consistent with observations in small-scale VLMs. Across two models of different scales, pre-trained on distinct datasets and tasks, LN-based fine-tuning consistently enhanced their accuracy on close-ended questions.

While Attention-tuning and FFN-tuning marginally surpassed LN-tuning in global accuracy, achieving peak open-ended question accuracies of 64.34% and global accuracy of 65.22%, this came at the cost of escalating the volume of tuned parameters from 262,144 to 2,885,943,296—a millionfold increase. Furthermore, following peak performance under current fine-tuning paradigms, the model ceased learning intrinsic relations of the features, instead focusing on the quantitative distribution of ground-truth tokens. This shift manifested in minimal accuracy improvements and a dramatic reduction in average output length, with mean tokens dropping from 28.27 to 17.38. Figure <ref> compares the generated outcomes across different intrinsic and extrinsic fine-tuning methods under the [F, F, PEFT, F] paradigms, illustrating this phenomenon driven by inherent large model training patterns and suboptimal training data. Most datasets employed for fine-tuning Med-VQA tasks comprise answers in short text formats, with close-ended answers typically being `yes' or `no', and open-ended answers containing only a few words. For LLMs seeking interpretability of the answer, adjusting more parameters paradoxically impairs generative performance.



When applying LoRA-tuning and Prefix-tuning to LLaVA-Med, the model's performance did not exhibit notable improvement. LoRA-tuning's recall reached 47.55% and 47.26%, indicating a closer alignment of the model's output with the ground truth distribution in the vocabulary space. However, this did not translate to enhanced evaluation accuracy, with accuracies under [F, F, PEFT, F] and [F, T, PEFT, T] fine-tuning paradigms reaching only 60.6% and 61.36%, respectively. This suggests that LoRA-tuning failed to deepen the multimodal model's understanding of joint image-text features, merely aligning output closer to the ground truth distribution.

 Figure <ref> displays the loss curves for various methods under the [F, T, PEFT, T] fine-tuning paradigm. Compared to LN-tuning, LoRA-tuning's minimum fine-tuned parameter volume is approximately fifty times larger, yet its accuracy is roughly 3% lower, with average output lengths of 25.43 and 25.72, trailing behind LN-tuning. These factors collectively indicate that existing PEFT methods may not directly enhance text-based accuracy in discerning medical images in multimodal model downstream fine-tuning, underscoring the advantages of LN-tuning over traditional PEFT approaches.
 


 §.§ Large-scale VLM IRG Result


To further explore the impact of various fine-tuning methods on the performance of large-scale Med-VLMs in the Med-IRG context, we employed the [F, F, PEFT, F] fine-tuning paradigm using the OpenI dataset to fine-tune Med-LLaVA. The model's OOD performance was then tested on the MIMIC dataset's test set to assess how it handles variations in input text domains. As shown in Table <ref>, the performance of the models fine-tuned with LN, Attention, LoRA, and Prefix methods showed minimal differences, with output text lengths averaging around 122.7. In contrast, FFN tuning significantly outperformed other fine-tuning approaches, demonstrating its superior capability in learning the underlying representations of ground-truth in long text generation tasks like Med-IRG.









 §.§ Out of Distribution Performance Testing





To assess whether the performance of LLaVA-Med on a familiar dataset like Slake correlates with its performance on a novel dataset, we conducted an OOD testing on the VQA-RAD dataset. This test serves to evaluate the model's robustness and flexibility by applying it to a different domain within the same field but with unseen data. 
More specifically, the images in the VQA-RAD dataset belong to the proximity domain with the Slake dataset but are quite different from the Slake dataset in terms of question formulation. Such experiments allow us to consider the ability of different fine-tuning methods to reason on non-proximity-domain text over similar medical images, in order to speculate on the ability of the models fine-tuned with the VQA dataset to be applied to the real Med-VQA scenarios.

In this experiment, we observe various fine-tuning paradigms, focusing particularly on the role of the transformer-based LayerNorm, Attention, and FFN adjustments. The results show a notable variance in the model's ability to generalize the learned features to the VQA-RAD dataset.
As Table <ref> shows, fine-tuning methods that show comparable results on the Slake dataset exhibit significant performance variances on the OOD VQA-RAD dataset. Notably, under the [F, T, PEFT, T] training paradigm, LoRA-tuning underwent a remarkable reversal, surpassing the performances of Attention-tuning and FFN-tuning, which were previously effective on the Slake dataset. It achieves a global accuracy of 65.41% and matches the best-closed question accuracy of 73.71%, initially noted with LN-tuning.
Conversely, FFN-tuning, despite being superior at learning adjacent training text representations, disappointed in its OOD performance. While it excels in Rouge-L metrics with scores of 26.35%, 9.80%, and 13.51%, significantly surpassing other fine-tuning methods, it only managed accuracy scores of 56.50%, 64.54%, and 60.98%. Moreover, its mean output length plummeted to 18.51, the lowest among all methods. This combination of metrics further validates that although FFN-tuning can closely fit the training data during SFT, it predominantly learns the distribution of ground-truth tokens rather than enhancing the model's ability to generalize image-text reasoning. A similar pattern is observed with Attention-tuning; the global accuracy decreases by 7.54% compared to LN-tuning when the connector and LM head are trained more. However, the text length is optimal at this time. Comprehensively, the mean token length of attention tuning under the same paradigm in Table <ref> can show that attention tuning slightly overlearns the text in the adjacent domains during SFT training, but does not cause large damage to the model's output ability of textual diversity in textual reasoning in the non-adjacent domains.

When enlarging the perspective to compare the accuracy performances across different fine-tuning methods, LN-tuning consistently displayed formidable strength. Under the [F, F, PEFT, F] tuning paradigm, LN-tuning, utilizing the smallest parameter adjustment, reached the highest scores in opened, closed, and global accuracy at 54.5%, 75.3%, and 66.08%, respectively. It also maintains a longer mean text output than any other method under the same tuning conditions. These results, coupled with the Rouge-L metrics from Table <ref>, indicate that LN-tuning manages to enhance the model's understanding of multimodal feature interrelations, significantly minimizing the model's overemphasis on learning ground-truth text token distributions due to low training data quality. This is evidenced by the lowest recall rate of 1.5% and the highest global accuracy of 66.08%. In contrast, LoRA-tuning, despite adjusting 50 times more parameters, did not significantly outperform LN-tuning. 





 §.§ Zero-shot Capability Investigation





 
To further investigate the zero-shot capabilities of different fine-tuning paradigms on Med-VLMs, we conduct extensive evaluations of the LLaVA-Med model on the MIMIC test dataset after SFT on the Slake dataset. This analysis aims to understand the impact of various intrinsic tuning methods on the model's ability to generalize and adapt to new tasks within the medical domain, particularly for IRG tasks. The evaluation employs metrics such as METEOR score, Rouge-L, and mean token length to measure factual accuracy, linguistic precision, and diversity of output in medical report generation.
From Table <ref>, experimental results indicate that different tuning methods exhibit varying impacts on the model’s zero-shot performance. LN-tuning consistently showed robust performance across different configurations, achieving the highest precision scores (18.54% and 18.26% under different paradigms), which underscores its effectiveness in preserving the factualness of model outputs. In contrast, the Attention and FFN methods, although effective in some scenarios, demonstrate greater variability in their influence on model generalization.

Notably, FFN-tuning, which previously excelled in VQA tasks, scored the lowest in precision (17.84%) under the [F, F, FFN, F] paradigm on the MIMIC dataset. This suggests that the model may have overlearned task-specific features from the Slake dataset, thus hindering its generalization and transfer capabilities. Furthermore, METEOR scores positively correlated with the number of adjusted parameters, increasing from 11.86% to a high of 12.81%, indicating that a larger volume of tuned parameters enhances the model’s linguistic alignment capabilities in medical text generation tasks.
Examining the effects of freezing versus tuning the connector and LM head reveals no clear pattern in performance metrics between [F, T, PEFT, T] and [F, F, PEFT, F] configurations. Changes in Rouge-L score and precision are minimal, suggesting that extensive fine-tuning of the connector and LM head does not necessarily contribute to improved zero-shot performance across these metrics.

Comparison between intrinsic tuning and traditional methods, such as LoRA-tuning, did not exhibit standout performance in the zero-shot setting. Under the [F, T, PEFT, T] paradigm, LoRA-tuning shows lower METEOR scores, Rouge-L, and mean token length compared to intrinsic methods, indicating that LoRA-tuning might not effectively maintain the overall transferability and generalization of the model in medical applications. Thus, intrinsic tuning methods, particularly LN-tuning with minimal parameter adjustments, might be a better choice, especially under the [F, F, PEFT, F] paradigm, where it outperforms more parameter-intensive methods like Attention-tuning in maintaining the generalization capabilities. These observations underscore the efficacy of LN-tuning in preserving the generalization of LVLMs for diverse medical tasks.




§ CONCLUSION

This study presents a thorough examination of intrinsic parameter fine-tuning and exposing LN-tuning, as a potent alternative to traditional PEFT methods for Med-VLMs. Our extensive experimental analysis across both small-scale and large-scale Med-VLMs demonstrated that fine-tuning the LayerNorm layers significantly enhances the models' adaptability, efficiency, and scalability in performing specialized medical tasks, such as Med-VQA and Med-IRG. 



We hope this work will enhance the clinical applicability of Med-VLMs in real-world medical settings.


This work is supported in part by the National Key R&D Program of China (2021ZD0113503) and in part by the Shanghai Municipal Science and Technology Committee of Shanghai Outstanding Academic Leaders Plan (No. 21XD1430300).








53


















#1     #1#1#1     #1     #1     #1     #1#1          #1#1           






[Brown et al(2020)]
        GPT3
authorpersonTom Brown, personBenjamin Mann, personRyder, et al year2020.
Language models are few-shot learners.
journalAdvances in neural information processing systems  volume33 (year2020), pages1877–1901.



[Chen et al(2024a)]
        chen2024can
authorpersonJiawei Chen, personYue Jiang, personDingkang Yang, personMingcheng Li, personJinjie Wei, personZiyun Qian, and personLihua Zhang. year2024a.
Can LLMs' Tuning Methods Work in Medical Multimodal Domain?
journalarXiv preprint arXiv:2403.06407 (year2024).



[Chen et al(2024b)]
        miss
authorpersonJiawei Chen, personDingkang Yang, personYue Jiang, et al year2024b.
MISS: A Generative Pretraining and Finetuning Approach for Med-VQA.
journalarXiv preprint arXiv:2401.05163 (year2024).



[Chen et al(2022)]
        m3ae
authorpersonZhihong Chen, personYuhao Du, personHu, et al year2022.
Multi-modal masked autoencoders for medical vision-and-language pre-training. In booktitleMICCAI. Springer, pages679–689.



[Dai et al([n. d.])]
        instructblip
authorpersonW Dai, personJ Li, personD Li, personAMH Tiong, personJ Zhao, personW Wang, personB Li, personP Fung, and personS Hoi. year[n. d.].
InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning. arXiv 2023.
journalarXiv preprint arXiv:2305.06500 (year[n. d.]).



[Devlin et al(2018)]
        devlin2018bert
authorpersonJacob Devlin, personMing-Wei Chang, personLee, et al year2018.
Bert: Pre-training of deep bidirectional transformers for language understanding.
journalarXiv preprint arXiv:1810.04805 (year2018).



[Do et al(2021)]
        10.1007/978-3-030-87240-3_7
authorpersonTuong Do, personBinh X. Nguyen, et al year2021.
Multiple Meta-model Quantifying for Medical Visual Question Answering. In booktitleMICCAI. addressCham, pages64–74.
978-3-030-87240-3


[Dosovitskiy et al(2020)]
        ViT
authorpersonAlexey Dosovitskiy, personLucas Beyer, personAlexander Kolesnikov, personDirk Weissenborn, personXiaohua Zhai, personThomas Unterthiner, personMostafa Dehghani, personMatthias Minderer, personGeorg Heigold, personSylvain Gelly, et al year2020.
An image is worth 16x16 words: Transformers for image recognition at scale.
journalarXiv preprint arXiv:2010.11929 (year2020).



[Du et al(2021)]
        du2021learning
authorpersonYangtao Du, personDingkang Yang, personPeng Zhai, personMingchen Li, and personLihua Zhang. year2021.
Learning Associative Representation for Facial Expression Recognition. In booktitleIEEE International Conference on Image Processing (ICIP). pages889–893.



[Gong et al(2022)]
        9802503
authorpersonHaifan Gong, personGuanqi Chen, personMao, et al year2022.
VQAMix: Conditional Triplet Mixup for Medical Visual Question Answering.
journalIEEE Transactions on Medical Imaging volume41, number11 (year2022), pages3332–3343.

<https://doi.org/10.1109/TMI.2022.3185008>



[Han et al(2024)]
        han2024parameter
authorpersonZeyu Han, personChao Gao, personJinyang Liu, personSai Qian Zhang, et al year2024.
Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey.
journalarXiv preprint arXiv:2403.14608 (year2024).



[He et al(2023)]
        he2023parameter
authorpersonXuehai He, personChunyuan Li, personPengchuan Zhang, personJianwei Yang, and personXin Eric Wang. year2023.
Parameter-efficient model adaptation for vision transformers. In booktitleProceedings of the AAAI Conference on Artificial Intelligence, Vol. volume37. pages817–825.



[Houlsby et al(2019)]
        adapter_tuning
authorpersonNeil Houlsby, personAndrei Giurgiu, personJastrzebski, et al year2019.
Parameter-efficient transfer learning for NLP. In booktitleInternational Conference on Machine Learning. PMLR, pages2790–2799.



[Hu et al(2021)]
        lora
authorpersonEdward J Hu, personShen, et al year2021.
Lora: Low-rank adaptation of large language models.
journalarXiv preprint arXiv:2106.09685 (year2021).



[Johnson et al(2019)]
        mimic
authorpersonAlistair EW Johnson, personTom J Pollard, personSeth J Berkowitz, personNathaniel R Greenbaum, personMatthew P Lungren, personChih-ying Deng, personRoger G Mark, and personSteven Horng. year2019.
MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports.
journalScientific data volume6, number1 (year2019), pages317.



[Lau et al(2018)]
        lau2018dataset
authorpersonJason J Lau, personGayen, et al year2018.
A dataset of clinically generated visual questions and answers about radiology images.
journalScientific data volume5, number1 (year2018), pages1–10.



[Lei et al(2023)]
        lei2023text
authorpersonYuxuan Lei, personDingkang Yang, personMingcheng Li, personShunli Wang, personJiawei Chen, and personLihua Zhang. year2023.
Text-oriented Modality Reinforcement Network for Multimodal Sentiment Analysis from Unaligned Multimodal Sequences.
journalarXiv preprint arXiv:2307.13205 (year2023).



[Lester et al(2021)]
        prompt
authorpersonBrian Lester, personRami Al-Rfou, and personNoah Constant. year2021.
The power of scale for parameter-efficient prompt tuning.
journalarXiv preprint arXiv:2104.08691 (year2021).



[Li et al(2023b)]
        llavamed
authorpersonChunyuan Li, personCliff Wong, personZhang, et al year2023b.
Llava-med: Training a large language-and-vision assistant for biomedicine in one day.
journalarXiv preprint arXiv:2306.00890 (year2023).



[Li et al(2023a)]
        blip-2
authorpersonJunnan Li, personDongxu Li, personSilvio Savarese, and personSteven Hoi. year2023a.
Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.
journalarXiv preprint arXiv:2301.12597 (year2023).



[Li et al(2022)]
        li2022blip
authorpersonJunnan Li, personDongxu Li, personXiong, et al year2022.
Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In booktitleICCV. pages12888–12900.



[Li et al(2021)]
        li2021align
authorpersonJunnan Li, personRamprasaath Selvaraju, personGotmare, et al year2021.
Align before fuse: Vision and language representation learning with momentum distillation.
journalNIPS  volume34 (year2021), pages9694–9705.



[Li et al(2024)]
        li2024unified
authorpersonMingcheng Li, personDingkang Yang, personYuxuan Lei, personShunli Wang, personShuaibing Wang, personLiuzhen Su, personKun Yang, personYuzheng Wang, personMingyang Sun, and personLihua Zhang. year2024.
A Unified Self-Distillation Framework for Multimodal Sentiment Analysis with Uncertain Missing Modalities. In booktitleProceedings of the AAAI Conference on Artificial Intelligence (AAAI), Vol. volume38. pages10074–10082.



[Li et al(2023c)]
        li2023towards
authorpersonMingcheng Li, personDingkang Yang, and personLihua Zhang. year2023c.
Towards Robust Multimodal Sentiment Analysis under Uncertain Signal Missing.
journalIEEE Signal Processing Letters  volume30 (year2023), pages1497–1501.



[Li and Liang(2021)]
        li2021prefix
authorpersonXiang Lisa Li and personPercy Liang. year2021.
Prefix-tuning: Optimizing continuous prompts for generation.
journalarXiv preprint arXiv:2101.00190 (year2021).



[Liu et al(2021a)]
        10.1007/978-3-030-87196-3_20
authorpersonBo Liu, personLi-Ming Zhan, and personXiao-Ming Wu. year2021a.
Contrastive Pre-training and Representation Distillation for Medical Visual Question Answering Based on Radiology Images. In booktitleMICCAI 2021. publisherSpringer International Publishing, addressCham, pages210–220.



[Liu et al(2021b)]
        liu2021slake
authorpersonBo Liu, personLi-Ming Zhan, personXu, et al year2021b.
Slake: A semantically-labeled knowledge-enhanced dataset for medical visual question answering. In booktitle2021 ISBI. pages1650–1654.



[Liu et al(2023a)]
        llava
authorpersonHaotian Liu, personChunyuan Li, personWu, et al year2023a.
Visual instruction tuning.
journalarXiv preprint arXiv:2304.08485 (year2023).



[Liu et al(2024)]
        dora
authorpersonShih-Yang Liu, personChien-Yi Wang, personHongxu Yin, personPavlo Molchanov, personYu-Chiang Frank Wang, personKwang-Ting Cheng, and personMin-Hung Chen. year2024.
DoRA: Weight-Decomposed Low-Rank Adaptation.
journalarXiv preprint arXiv:2402.09353 (year2024).



[Liu et al(2023b)]
        p-tuning
authorpersonXiao Liu, personYanan Zheng, personZhengxiao Du, personDing, et al year2023b.
GPT understands, too.
journalAI Open (year2023).



[Nguyen et al(2019)]
        10.1007/978-3-030-32251-9_57
authorpersonBinh D. Nguyen, personThanh-Toan Do, personBinh X Nguyen, et al year2019.
Overcoming Data Limitation in Medical Visual Question Answering. In booktitleMICCAI. addressCham, pages522–530.



[Radford et al(2021)]
        CLIP
authorpersonAlec Radford, personJong Wook Kim, personHallacy, et al year2021.
Learning transferable visual models from natural language supervision. In booktitleInternational conference on machine learning. PMLR, pages8748–8763.



[Sanjay Subramanian et al(2020)]
        subramanian-2020-medicat
authorpersonSachin Mehta Sanjay Subramanian, Lucy Lu Wang et al year2020.
MedICaT: A Dataset of Medical Images, Captions, and Textual References. In booktitleFindings of EMNLP.



[Sung et al(2022)]
        sung2022vl
authorpersonYi-Lin Sung, personJaemin Cho, and personMohit Bansal. year2022.
Vl-adapter: Parameter-efficient transfer learning for vision-and-language tasks. In booktitleProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages5227–5237.



[Thawkar et al(2023)]
        xraygpt
authorpersonOmkar Thawkar, personAbdelrahman Shaker, personSahal Shaji Mullappilly, personHisham Cholakkal, personRao Muhammad Anwer, personSalman Khan, personJorma Laaksonen, and personFahad Shahbaz Khan. year2023.
Xraygpt: Chest radiographs summarization using medical vision-language models.
journalarXiv preprint arXiv:2306.07971 (year2023).



[Wang et al(2024)]
        wang2024parameter
authorpersonHaixin Wang, personXinlong Yang, personJianlong Chang, personDian Jin, personJinan Sun, personShikun Zhang, personXiao Luo, and personQi Tian. year2024.
Parameter-efficient tuning of large-scale multimodal foundation model.
journalAdvances in Neural Information Processing Systems  volume36 (year2024).



[Wang et al(2017)]
        openi
authorpersonXiaosong Wang, personYifan Peng, personLe Lu, personZhiyong Lu, personMohammadhadi Bagheri, and personRonald M Summers. year2017.
Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In booktitleProceedings of the IEEE conference on computer vision and pattern recognition. pages2097–2106.



[Xu et al(2023)]
        xu2023parameter
authorpersonLingling Xu, personHaoran Xie, personSi-Zhao Joe Qin, personXiaohui Tao, and personFu Lee Wang. year2023.
Parameter-efficient fine-tuning methods for pretrained language models: A critical review and assessment.
journalarXiv preprint arXiv:2312.12148 (year2023).



[Yang et al(2023a)]
        yang2023context
authorpersonDingkang Yang, personZhaoyu Chen, personYuzheng Wang, personShunli Wang, personMingcheng Li, personSiao Liu, personXiao Zhao, personShuai Huang, personZhiyan Dong, personPeng Zhai, and personLihua Zhang. year2023a.
Context De-Confounded Emotion Recognition. In booktitleProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pages19005–19015.



[Yang et al(2022a)]
        yang2022disentangled
authorpersonDingkang Yang, personShuai Huang, personHaopeng Kuang, personYangtao Du, and personLihua Zhang. year2022a.
Disentangled Representation Learning for Multimodal Emotion Recognition. In booktitleProceedings of the 30th ACM International Conference on Multimedia (ACM MM). pages1642–1651.



[Yang et al(2022b)]
        yang2022contextual
authorpersonDingkang Yang, personShuai Huang, personYang Liu, and personLihua Zhang. year2022b.
Contextual and Cross-Modal Interaction for Multi-Modal Speech Emotion Recognition.
journalIEEE Signal Processing Letters  volume29 (year2022), pages2093–2097.



[Yang et al(2022c)]
        yang2022emotion
authorpersonDingkang Yang, personShuai Huang, personShunli Wang, personYang Liu, personPeng Zhai, personLiuzhen Su, personMingcheng Li, and personLihua Zhang. year2022c.
Emotion Recognition for Multiple Context Awareness. In booktitleProceedings of the European Conference on Computer Vision (ECCV), Vol. volume13697. pages144–162.



[Yang et al(2023b)]
        yang2023aide
authorpersonDingkang Yang, personShuai Huang, personZhi Xu, personZhenpeng Li, personShunli Wang, personMingcheng Li, personYuzheng Wang, personYang Liu, personKun Yang, personZhaoyu Chen, personYan Wang, personJing Liu, personPeixuan Zhang, personPeng Zhai, and personLihua Zhang. year2023b.
AIDE: A Vision-Driven Multi-View, Multi-Modal, Multi-Tasking Dataset for Assistive Driving Perception. In booktitleProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pages20459–20470.



[Yang et al(2022d)]
        yang2022learning
authorpersonDingkang Yang, personHaopeng Kuang, personShuai Huang, and personLihua Zhang. year2022d.
Learning Modality-Specific and -Agnostic Representations for Asynchronous Multimodal Language Sequences. In booktitleProceedings of the 30th ACM International Conference on Multimedia (ACM MM). pages1708–1717.



[Yang et al(2024a)]
        yang2024MCIS
authorpersonDingkang Yang, personMingcheng Li, personDongling Xiao, personYang Liu, personKun Yang, personZhaoyu Chen, personYuzheng Wang, personPeng Zhai, personKe Li, and personLihua Zhang. year2024a.
Towards Multimodal Sentiment Analysis Debiasing via Bias Purification.
journalarXiv preprint arXiv:2403.05023 (year2024).



[Yang et al(2023c)]
        yang2023target
authorpersonDingkang Yang, personYang Liu, personCan Huang, personMingcheng Li, personXiao Zhao, personYuzheng Wang, personKun Yang, personYan Wang, personPeng Zhai, and personLihua Zhang. year2023c.
Target and source modality co-reinforcement for emotion understanding from asynchronous multimodal sequences.
journalKnowledge-Based Systems  volume265 (year2023), pages110370.



[Yang et al(2024b)]
        yang2024SuCi
authorpersonDingkang Yang, personDongling Xiao, personKe Li, personYuzheng Wang, personZhaoyu Chen, personJinjie Wei, and personLihua Zhang. year2024b.
Towards Multimodal Human Intention Understanding Debiasing via Subject-Deconfounding.
journalarXiv preprint arXiv:2403.05025 (year2024).



[Yang et al(2024c)]
        yang2024robust
authorpersonDingkang Yang, personKun Yang, personMingcheng Li, personShunli Wang, personShuaibing Wang, and personLihua Zhang. year2024c.
Robust Emotion Recognition in Context Debiasing.
journalarXiv preprint arXiv:2403.05963 (year2024).



[Yang et al(2023d)]
        yang2023how2comm
authorpersonDingkang Yang, personKun Yang, personYuzheng Wang, personJing Liu, personZhi Xu, personRongbin Yin, personPeng Zhai, and personLihua Zhang. year2023d.
How2comm: Communication-efficient and collaboration-pragmatic multi-agent perception. In booktitleThirty-seventh Conference on Neural Information Processing Systems (NeurIPS).



[Zeng and Lee(2023)]
        zeng2023expressive
authorpersonYuchen Zeng and personKangwook Lee. year2023.
The expressive power of low-rank adaptation.
journalarXiv preprint arXiv:2310.17513 (year2023).



[Zhang et al(2023)]
        llama-adapter
authorpersonRenrui Zhang, personJiaming Han, personAojun Zhou, personXiangfei Hu, personShilin Yan, personPan Lu, personHongsheng Li, personPeng Gao, and personYu Qiao. year2023.
Llama-adapter: Efficient fine-tuning of language models with zero-init attention.
journalarXiv preprint arXiv:2303.16199 (year2023).



[Zhao et al(2023)]
        lntuning
authorpersonBingchen Zhao, personHaoqin Tu, personChen Wei, personJieru Mei, and personCihang Xie. year2023.
Tuning LayerNorm in Attention: Towards efficient multi-modal llm finetuning.
journalarXiv preprint arXiv:2312.11420 (year2023).



[Zhu et al(2023)]
        minigpt
authorpersonDeyao Zhu, personJun Chen, personXiaoqian Shen, personXiang Li, and personMohamed Elhoseiny. year2023.
Minigpt-4: Enhancing vision-language understanding with advanced large language models.
journalarXiv preprint arXiv:2304.10592 (year2023).







