Instruction Matters, a Simple yet Effective Task Selection Approach in Instruction Tuning for Specific Tasks
Changho Lee Janghoon Han Seonghyeon Ye Stanley Jungkyu Choi Honglak Lee Kyunghoon Bae

[
    Cédric Pilatte
    Accepted XXX, Received YYY; in original form ZZZ
====================================================







Instruction tuning has shown its ability to not only enhance zero-shot generalization across various tasks but also its effectiveness in improving the performance of specific tasks. A crucial aspect in instruction tuning for a particular task is a strategic selection of related tasks that offer meaningful supervision, thereby enhancing efficiency and preventing performance degradation from irrelevant tasks. Our research reveals that leveraging instruction information alone enables the identification of pertinent tasks for instruction tuning. This approach is notably simpler compared to traditional methods that necessitate complex measurements of pairwise transferability between tasks or the creation of data samples for the target task. Furthermore, by additionally learning the unique instructional template style of the meta-dataset, we observe an improvement in task selection accuracy, which contributes to enhanced overall performance. Experimental results demonstrate that training on a small set of tasks, chosen solely based on the instructions, leads to substantial performance improvements on benchmarks like P3, Big-Bench, NIV2, and Big-Bench Hard. Significantly, these improvements exceed those achieved by prior task selection methods, highlighting the efficacy of our approach.[Our code and dataset will be released after the review process.]





§ INTRODUCTION

Recently, instruction tuning has gained attention as an innovative approach for improving zero-shot performance <cit.>. This method offers the advantage of improving a model's generalization capabilities to unseen tasks by training on diverse tasks accompanied by instructions. The robustness of instruction tuning improves as the diversity of training tasks increases <cit.>. 
In this light, recent studies have concentrated on broadening the diversity and increasing the number of tasks within the meta-dataset <cit.> used for instruction tuning. For example, <cit.> introduce a collection of over 1,600 tasks, and <cit.> expand to 1,836 tasks through the integration of additional datasets.
Moreover, <cit.> automatically generate a substantial volume of instruction tuning data using Large Language Models (LLMs) instead of human creation.


Besides applying instruction tuning for general unseen tasks, there is also growing interest in instruction tuning as a methodology to improve the performance of specific unseen tasks <cit.>.
Instruction tuning focusing on specific tasks trains not all but only informative tasks with instruction format. This strategy is based on the insight that not all tasks are helpful to specific tasks, and some tasks could even lead to performance degradation due to negative transfer during multi-task training <cit.>. However, selecting relevant tasks for training presents a significant challenge. First, manually reviewing through the vast array of datasets for instruction tuning is not feasible. Additionally, discerning the relevance of certain tasks in the training dataset to the target task is often ambiguous.

To address this challenge, studies have been conducted to automatically quantify the relevance between tasks <cit.>. These methods are primarily divided into two categories. The first is assessing pairwise task transferability by training models on one task and then evaluating their performance on another <cit.>. The second is calculating similarity scores between tasks using a small sample of the test dataset <cit.>. However, the former approach, which measures pairwise transfer for every task, can be exceedingly time-consuming and compute-intensive. Moreover, the latter approach has limitations in that it necessitates constructing data (instances) for unseen target tasks, contradicting the zero-shot setting condition in instruction tuning.

In this study, we explore a straightforward yet effective method for choosing relevant tasks for instruction tuning. We focus on the feature of instruction tuning, where the instructions[Following <cit.>, instruction means prompt, template, and task description without instance.] define the characteristics of each task. In this vein, we introduce an Instruction-based Task selector (InsTa) that leverages instruction similarity scores to determine task relevance and select informative tasks for training. Through experiments, we discover that this simplified instruction-only selection method adequately identifies related tasks and significantly improves performance. Moreover, by further aligning InsTa through training on the instruction style and format of a meta-dataset, it is able to closely understand the instructional nuances, achieving significant performance improvements.

A notable finding from our study is that task selection based exclusively on instruction surpasses previous methods <cit.> that incorporate both instructions and instances. Moreover, instruction information alone shows a high correlation with task similarity as determined by complex pairwise transfer methods <cit.> and even achieves slightly better average zero-shot performance. This indicates that the instruction-based approach for selecting related tasks is not only easily applicable but also highly effective.
 
In summary, our contributions are as follows: 
 
    
  * We introduce InsTa which simplifies task selection for instruction tuning by using only instruction similarity scores, obviating the need for instances.
    
  * By aligning InsTa with the instruction styles and formats of meta-datasets, we significantly improve performance, demonstrating the importance of understanding instructional nuances.
    
  * Our study reveals that using only instructions not only surpasses prior methods that employing both instructions and instances but also parallels the trends of pairwise transfer methods without their associated complexities, demonstrating our method's effectiveness.




§ RELATED WORK



 §.§ Instruction Tuning: Generalist or Specialist

Instruction tuning shows remarkable zero-shot performance on unseen tasks by training models on various tasks integrated with corresponding instructions. This approach can be broadly categorized into two main streams: instruction tuning as a generalist, which aims to perform well across various unseen tasks <cit.>, and instruction tuning as a specialist, which focuses on excelling in specific tasks rather than achieving proficiency across all tasks <cit.>, as denoted by previous research <cit.>.

According to <cit.>, instruction tuning as a generalist can be categorized based on its objectives: generalizing to "unseen tasks" and generalizing to "unseen instructions". Early studies propose the approach of generalizing to unseen tasks, which involves training on various natural language processing (NLP) tasks and evaluating on unseen tasks <cit.>. However, task-based instruction tuning presents the difficulty of defining every task that humans use. In response, a methodology that trains LLMs on diverse instructions without clear distinctions for specific tasks has been proposed, aiming to generalize to unseen instructions, rather than unseen tasks <cit.>.

Recent trends have seen the emergence of research focusing on enhancing the zero-shot capabilities of specific tasks through instruction tuning <cit.>. Instead of learning all tasks, the model selectively learns tasks related to the target task in instruction format, aiming to perform well on specific target tasks. Notably, <cit.> demonstrate that training exclusively on tasks related to a specific target task outperforms the instruction tuning as a generalist in terms of performance. 



 §.§ Quantifying Task Relationship 

Research on understanding which tasks can be helpful for other tasks has been extensively conducted across various fields. <cit.> calculate intermediate task transfer scores to discern the relationships between tasks, aiming to determine beneficial tasks to train the model before fine-tuning on specific target tasks. Additionally, <cit.> measure tasks that are helpful in parameter-efficient tuning through prompt transfer.

The research on identifying task relationships has been extended to the field of instruction tuning as well. <cit.> measure pairwise task transfer between every task pair to identify helpful source tasks on specific target tasks. On the other hand, <cit.> have calculated task similarity using only a few data samples from training and evaluation tasks instead of training and evaluating every task individually.

Previous task selection approach, which automatically measure task similarities, are either time-consuming due to the need to train and evaluate every task pair or still rely on having data samples for the target tasks, even in reduced quantities. Our method simplifies this process by evaluating task relationships solely through instructions, eliminating the need for laborious measurements across every task or generating test data, provided the tasks are adequately described through instructions. For more detailed descriptions of our method's practicalities compare to prior works, please refer to Appendix <ref>.



§ INSTRUCTION-BASED TASK SELECTOR (INSTA)

To enhance the zero-shot capability of LLMs for specific target tasks, we select informative tasks that positively impact the performance of the target task. Our task selection method exclusively relies on instruction information to assess the relevance between tasks, which is efficient as it removes the necessity of measuring correlations between all training and evaluation task pairs. 
Furthermore, unlike previous methods, our approach has the advantage of easy applicability with just the instruction (task description) of the target task, without the need for constructing data samples for the target task.



 §.§ Formulation


The meta-dataset 𝐌 consists of multiple task clusters 𝐂 and each task cluster comprises several tasks 𝐓 with the same task type. Each task 𝐓 includes various instructions 𝐈 and corresponding instances. We aim to find tasks related to the target task 𝐓̅. To identify tasks related to the target task 𝐓̅, we measure instruction-based task similarity score as follows:

    𝐒𝐜𝐨𝐫𝐞(I^T̅_i,I_j^T) = cos(𝐸(I^T̅_i),𝐸(I_j^T))

where I^T̅_i denotes the i^th instruction of the target task T̅, and I_j^T denotes the j^th instruction of some arbitrary task T ∈ M chosen for similarity assessment. For measuring similarity, we employ cosine similarity, and for the embedding function E, we utilize the Sentence Transformer <cit.> as an off-the-shelf embedding model, following <cit.>. For more specific details, please refer to the Appendix <ref>.




 §.§ Aligning Instruction-based Task Selector with Meta-dataset


Off-the-shelf embedding model often lacks the capability to accurately identify related tasks through instructions, since they are not trained on the unique instruction styles present in meta-datasets. To mitigate this issue, our approach includes an additional aligning process that fine-tunes our selector model on the distinctive instruction styles of each meta-dataset. For training, we select a random instruction from a given task as a positive sample and designate instructions from different task clusters as negative samples.[The rationale behind not utilizing task instructions of the same task cluster as positive or negative samples is twofold: in the former case, even though tasks may have the same task type, they can be differentiated strictly, and in the latter case, false negatives may occur, even within same task type.] The training objective is as follows:

    L(I_i, I_j, y) = ( y - 𝐒𝐜𝐨𝐫𝐞(I_i, I_j) )^2

where y ∈{0,1} denotes the truth label of a given instruction pair (I_i, I_j). y = 1 indicates that I_j is in same task; otherwise, y = 0. The similarity score is measured using Equation <ref>.




 §.§ Multi-task Selection for Instruction Tuning

To efficiently perform instruction tuning for a specific target task, we select the most relevant training tasks by employing an InsTa model as detailed in Section <ref>.
In this process, for each instruction I_i^T̅ of the unseen target task T̅, we compute the similarity scores with every instruction I_j^T across all tasks T in the training set, as defined by Equation <ref>. Based on these computed scores, we then select the top-k tasks T that exhibit the highest degrees of similarity. This selection mechanism is encapsulated by the following formula:

    k-argmax_T{𝐒𝐜𝐨𝐫𝐞(I_i^T̅, I_j^T)  , ∀i, j}

where the "k-argmax" operation in the formula signifies selecting the top-k tasks that have the highest scores.




§ EXPERIMENTAL SETUP



 §.§ Dataset

We conduct experiments on two representative instruction tuning meta-datasets: P3 (Public Pool of Prompts) <cit.> and NIV2 (SuperNaturalInstructions V2) <cit.>. P3 is a meta-dataset comprised of 12 task clusters. It contains a total of 35 tasks across 8 task clusters for training and 11 tasks across 4 task clusters for held-out evaluation. Note that each task in P3 includes 11.7 instructions on average. Conversely, NIV2 encompasses 72 task clusters of English tasks. The training set comprises a total of 756 tasks across 60 clusters, while the held-out evaluation tasks include 119 tasks across 12 task clusters. In the case of NIV2, each task consists of a single instruction. We additionally evaluate BIG-Bench <cit.> and BIG-Bench Hard (BBH) <cit.> as supplementary evaluation datasets. For more detailed information, please refer to the Table <ref> and Appendix <ref>.

 



 §.§ Task Selector Setup







  
P3  consists of an average of 11.7 instructions per task. In P3, there are some instructions that are designed to diversify task types in a single dataset["generating documents from its summary" for summarization task.], and we exclude such instructions from selector training since they may hinder the selection of relevant tasks. Furthermore, instructions for P3 tasks include unique placeholders[{{-}} represents placeholder in the instruction. An example instruction from the WiC task: "Does the word {{word}} have the same meaning in these two sentences? Yes, No?\n{{sentence1}}\n{{sentence2}}.]. These placeholders could act as misleading shortcuts and may negatively influence training. Therefore, we have standardized them to {{text}} and {{candidate}} for input snippets and label space, respectively. Please refer to the Appendix <ref> for more detail about P3 instruction formulation.


  
NIV2  instructions feature human-crafted, human-readable Task Definition, Positive Task Examples, Negative Task Examples, and Explanation. We utilize the Task Definition for training the task selector. Moreover, since NIV2 has only one instruction per task, there are no positive samples for training the task selector. To address this, we generate instructions for all NIV2 tasks using GPT-4 <cit.>. Specifically, we generate a paraphrased one and employ it as a positive sample for training the task selector. For more information about a query used for GPT-4 API calls and generated instructions, please refer to the Appendix <ref>.

 





 §.§ Training Setup

Following previous studies <cit.>, we employ all tasks except for those held out for evaluation as our training set. Unlike traditional approaches that train on all tasks, our strategy specifically trains only the top-k tasks deemed most informative for each target task.

For the P3, we select the top-5 highest scoring training tasks out of 35 tasks for each target task. For each training, we randomly sample 50k instances for each training task, totaling 250k instances. Unlike the P3, each task in NIV2 has only one instruction, and data instances are limited to 6.5k. Considering this difference, we select top-70 tasks out of 756 tasks for each target task and randomly sample up to 5k instances for each task, which ends up with 350k instances in total. These training quantities are significantly smaller than the 2M instances of T0 <cit.> and 5.7M instances of Tk-Instruct <cit.>, which correspond to the instruction-tuned models on P3 and NIV2, respectively. 

We use the T5 LM-adapted model(3B) <cit.> as our base model and train for 3 epochs with a constant learning rate of 1e-4 and 5e-5 for P3 and NIV2, respectively. We configure validation from training datasets and select the model that shows the best performance in the validation. Our experiment is in a true zero-shot setting, where no samples from the held-out task are used for checkpoint selection. Please refer to the Appendix <ref> for more detail.


  



  
Models

For the P3 dataset, we use the following baselines. GPT-3(175B) is an autoregressive LM that has shown remarkable ability in following demonstrations provided in its instructions <cit.>. T0-11B / T0-3B have same architecture as T5 but trained on millions of samples from 35 different P3 tasks <cit.>. T5(3B) is text-to-text pretrained LM without instruction tuning <cit.>. T5(3B) + Random is trained on 5 random tasks from P3. T5(3B) + Pairwise Transfer is trained on top-5 tasks with the highest transferability scores demonstrated in <cit.>, representing pairwise transfer task selection. T5(3B) + PE w/RoE denotes a model that selects relevant task by measuring cosine similarity between training and evaluation data samples <cit.>. 

For NIV2, we set  Tk-Instruct-3B as our baseline, which has the same architecture as T5-3B but trained on millions of samples from 756 different NIV2 tasks. 

Finally, to demonstrate the effectiveness of instruction-based task selection, we introduce the following model variants:

    
  * T5(3B) + InsTa: Model trained on selected top-k tasks by Instruction-based Task Selector (Off-the-shelf embedding model).
    
  * T5(3B) + InsTa_Aligned-P3/NIV2: Models trained on selected top-k tasks by further aligned InsTa on instructions from P3 and NIV2, respectively.









 §.§ Evaluation Setup

For P3 evaluation, following <cit.>, we apply rank classification and measure the performance of the model for each target instruction. Then, we calculate the average performance of instructions per task. Note that each target task in P3 has 10.09 instructions on average. For NIV2 evaluation, we follow the same evaluation protocol as in <cit.> and report ROUGE-L <cit.> score. We adopt greedy decoding with a maximum generation length of 256. 



§ RESULT




 §.§ P3 Results


Table <ref> shows experimental results on the 11 unseen datasets from P3. Compared to T0-3B, which is fully instruction-tuned on all 35 tasks from P3, the T5(3B) + Random model shows inferior performance, but both T5(3B) + InsTa model and T5(3B) + InsTa_Aligned-P3 model outperform 10 tasks out of 11 tasks, with each exhibiting a performance gap of 4.83% and 7.10% on average, respectively. They also show superior performance compared to T5(3B) + PE w/ RoE, highlighting the instruction itself is sufficient enough to choose informative tasks without the use of data samples. Moreover, T5(3B) + InsTa_Aligned-P3 outperforms T5(3B) + InsTa across 10 tasks, suggesting further aligning task selector on instructions from the meta-dataset increases the precision of task selection. Additionally, our approach exhibits marginally superior performance compared to T5(3B) + Pairwise Transfer, suggesting that instruction-only task selection can effectively identify related tasks without the need for exhaustive pairwise evaluations. Notably, we observe a strong correlation between pairwise task transferability and our task selection approach, further elaborated in Appendix <ref>.






























 §.§ Big-Bench Results


Table <ref> evaluates the performance of instruction-based task selection on the 14 Big-Bench tasks employing P3 held-in tasks in Section <ref> as the training dataset. We compare our method to baselines T0-3B and T5(3B) + Cos PE of <cit.>. Our model, T5(3B) + InsTa_Aligned-P3, surpasses T0-3B in 12 out of 14 tasks, achieving an average performance increase of 5.94%. Tasks like Movie D. and Syllogism, where results are comparable to random guesses, indicate informative tasks are insufficient in P3. When compared to Cos PE <cit.>, our T5(3B) + InsTa_Aligned-P3 model shows improvements in most tasks, with an average increase of 2.07%. Notably, it also surpasses T0-11B in a majority of tasks. These findings demonstrate that instruction-based task selection enables effective training even with a small number of relevant tasks.





 §.§ NIV2 Results


Figure <ref> illustrates the experimental results for 33 tasks within NIV2[Though NIV2 has 12 task clusters and 119 tasks for evaluation, we randomly select up to 3 tasks per task cluster due to computational cost.]. We adhere to [Def + Pos(2)] setting in NIV2, which includes Task Definition and two Positive Task Examples in the instruction, but note that we only use Task Definition for task selection. The precise input format utilized for training and evaluation is elaborated in the Appendix <ref>.

The experimental result reveals our T5(3B) + InsTa_Aligned-NIV2 surpasses the baseline in most tasks, highlighting the efficacy of our method in NIV2 meta-dataset. We observe notable improvements in performance for specific tasks, such as Task614, Task1385, and Task1388. This suggests that training all tasks could lead to performance degradation in certain tasks. However, training informative tasks selected by our method can alleviate this problem.





 §.§ Big-Bench Hard Results


To further validate the effectiveness of our approach, we assess our T5(3B) + InsTa_Aligned-NIV2 model on 27 BBH tasks. This model is trained on the top-70 tasks chosen from 756 NIV2 tasks. In contrast to Section <ref>, 
we adopt the Tk-Instruct-3B [Def] setting as our baseline due to the lack of Positive Task Examples in the BBH datasets. Moreover, given the absence of Task Definition for the BBH tasks, we utilize the GPT-4 API to generate definition for each task.

Consistent with results highlighted in Section <ref>, we observe performance enhancements across most tasks. Tasks such as Movie Recommendation, which exhibit suboptimal performance with Tk-Instruct-3B, demonstrate enhanced performance when only informative tasks are learned. This outcome underscores the robustness of T5(3B) + InsTa_Aligned-NIV2 in mitigating negative transfer by exclusively learning relevant tasks.






§ FURTHER ANALYSIS

For further analysis, we conduct experiments to compare instruction-based versus data sample-based task selection, examine the impact of instruction refinement in the P3 dataset, and assess the effects of varying the number of related tasks selected (k). Additionally, changes in task selection performance related to the number of instructions and experiments on learning different formats of meta-datasets and their impact on performance are detailed in Appendices <ref> and <ref>.



 §.§ Instruction-based vs. Data Sample-based Task Selection


We conduct experiments shown in Table <ref> to compare our instruction-based approach with the traditional method <cit.> of selecting related tasks using data samples (integration of instruction and instance). The T5(3B) + DSTa model identifies relevant tasks similar to T5(3B) + InsTa, with the key distinction being the use of data samples for similarity comparison.

While the T5(3B) + DSTa model outperforms in specific tasks like Hellaswag, it falls short in others, including RTE and WSC. Conversely, T5(3B) + InsTa consistently enhances performance across the majority of tasks relative to T0-3B, and achieves an impressive average performance enhancement of 5.08% over T5(3B) + DSTa. This outcome suggests that instances within data samples might obstruct the extraction of representative task features, diminishing the task selector's ability to identify related tasks effectively. Please refer to Appendix <ref> for more experimental details.





 §.§ Impact of Instruction Refinement


As mentioned in Section <ref>, our approach operates based on the premise that the instruction accurately describes the characteristics of task, necessitating the process of instruction refinement. Table <ref> demonstrates the impact of such instruction refinement on performance. The term "Unfiltered" denotes the training conducted without instruction refinement, while "Filtered" indicates the use of refined models.

Unfiltered models, while showing performance improvements in most tasks compared to T0-3B, encounter performance degradation in certain tasks, notably StoryCloze, due to the selection of irrelevant tasks. Conversely, models that use filtered and refined instructions accurately select related tasks and generally demonstrate improved performance over unfiltered models. This result emphasizes the significance of instruction quality in task selection. Moreover, the notable performance enhancement seen in T5(3B) + InsTa_Aligned-P3 indicates the substantial role of instruction quality, particularly when additional training for the alignment is held.




 §.§ Scaling Relevant Tasks




We explore how the model performance is affected by the number of selected tasks during instruction tuning. Figure <ref> illustrates the average performance of T5(3B) + InsTa_Aligned-P3 and T5(3B) + InsTa_Aligned-NIV2, across different numbers of selected tasks (k). The performance of the top-35 tasks for P3 and top-756 tasks for NIV2 corresponds to the fully instruction-tuned models, representing the scores of T0-3B and Tk-Instruct-3B, respectively. On average, we observe a progressive improvement in performance up to five tasks for P3 and seventy tasks for NIV2, after which it declines, affirming the adverse impact of non-relevant tasks during training. For more analysis and detailed results, please refer to Appendix <ref>.



§ CONCLUSION

In this study, we discover that selecting informative tasks for instruction tuning can be effectively achieved by exclusively using task instructions. Our experiments reveal that this method, particularly when aligned with the meta-dataset's instructions, surpasses traditional methods which depend on data samples to determine task relevance. Beyond its robust performance, our approach's most significant advantage is its simplicity: as long as the instruction (task description) of the target task can be described, our method can be applied. This approach marks a significant shift away from traditional methods that require exhaustive pairwise comparisons or the labor-intensive creation of data samples for new tasks. By adopting an instruction-only strategy, our method simplifies and automates the task selection process for instruction tuning, providing a more efficient and practical approach to building models in real-world scenarios.




§ LIMITATIONS

While highlighting effective task selection method in instruction tuning, we do not perform experimental results over the different sizes of model parameters other than the T5(3B) parameter model due to computational cost. For example, a language model bigger than 11B parameters may be less susceptible to negative transfer due to model capacity, or the effectiveness of our task selection method might stand out even more. Moreover, we only use the encoder-decoder architecture model in the paper, we leave the investigation on the decoder model, such as LLaMA 7B/13B, to our future work.

In this study, our focus is on two prominent meta-datasets, P3 and NIV2. Nonetheless, it is worth noting the existence of various other instruction tuning meta-datasets, such as FLAN-T5 <cit.> and the CoT (reasoning) collections <cit.>. Extending our methodology to incorporate these additional meta-datasets will be an intriguing prospect for future research and exploration.












§ PREVIOUS TASK SELECTION METHODS IN NLP


Task selection is actively researched in two main fields. One is task selection from the perspective of intermediate-task transfer learning, and the other is task selection within the realm of instruction tuning, which is the focus of our research.



 §.§ Task selection in Intermediate-Task Transfer Learning

Within the scope of intermediate-task transfer learning, the goal of task selection is to enhance the performance by further training the model on related intermediate tasks before fine-tuning it on a target task <cit.>. Therefore, it presupposes the availability of a labeled dataset for the target task..

<cit.> used BERT as a feature extractor to identify related tasks by comparing embeddings with an auxiliary task pool. In contrast, <cit.> developed a more efficient method, training a task discriminator with just 500 samples from each dataset, thereby reducing data needs. However, creating 500 labeled samples for a target task can still be burdensome, especially in an instruction-tuning setting. Moreover, in line with <cit.>, this approach entails a high computational cost for training task discriminator and inference on all auxiliary instances.

<cit.> opted for a different approach, directly measuring pairwise transfer for all datasets instead of using samples' embedding. To reduce computational costs, they employ adapter structures. Nonetheless, this method still incurs significant costs in terms of target data construction and computational overhead because it requires training and inference across all models.





 §.§ Task selection in instruction tuning

In task selection in instruction tuning, the objective is to select and train related task for improving zero-shot performance for target task. Unlike intermediate-task transfer learning, where samples are composed solely of instances, data samples comprise instructions plus instances. A model trained with such instructions can tackle new tasks given new instructions, utilizing previously learned instructions as a basis.

<cit.> propose a sample-based task selection methodology in instruction tuning. They encode all instances from a vast multitask data pool using a T5-3B and then build a search index from the resulting representations. A key advantage of their approach is that there's no need to train an embedding generator with the target task's data samples, unlike task selection in intermediate-task transfer learning. However, there is still a limitation of requiring 1000 unlabeled data samples for the target data. <cit.> adopted a method similar to <cit.> to retrieve experts related to the target task for instruction tuning by encoding 32 data samples for each expert to calculate similarity. 

<cit.> applied a similar methodology of <cit.> to instruction tuning, performing pairwise transfer across a variety of tasks. Notably, <cit.> conducted pairwise transfer for all tasks in the P3 dataset, and these results were utilized as a baseline in their paper.



 §.§ Baseline for Task Selection

As noted by <cit.>, task selection in intermediate-task transfer learning has several disadvantages compared to the task selection methodologies used in instruction tuning settings. These methodologies are primarily applied to classification tasks, require a large number of labeled samples for the target task, and involve high computational costs for training a model to generate task embeddings. For these reasons, we exclude these studies in our baselines. Additionally, we used <cit.>'s methodology as the baseline for the sample-based task selection instead of <cit.> because <cit.>'s approach requires fewer data samples, making implementation easier. The characteristics and practical applicability of each methodology are summarized in the Table <ref>




§ OFF-THE-SHELF EMBEDDING MODEL


We employ the Sentence Transformer <cit.> as an off-the-shelf embedding model in our study. Specifically, we utilize the "sentence-transformers/bert-large-nli-stsb-mean-tokens" checkpoint, which constitutes a 340 million parameter sentence-bert model trained on a combination of Natural Language Inference (NLI) data and the Semantic Textual Similarity (STS) benchmark data. To quantify the similarity between two given instructions, we measure the cosine similarity of their extracted embedding representations derived from this model.

We further train the Sentence Transformer using instructions from the meta-dataset to learn the style of the instructions. Besides the instructions, we sample 500 instances from MRPC, PAWS, and QQP datasets each and append them to training samples. These datasets are paraphrase identification datasets, ensuring that the model retains its generality throughout the training process.

During training, we employ a learning rate of 1e-6 for the P3 dataset and 1e-5 for the NIV2 dataset. The remaining training configurations align with those described in <cit.>. Specifically, we conduct training for 5 epochs and select the checkpoint corresponding to the best validation performance for subsequent analysis and evaluation.



§ TRAINING DETAILS


We truncate input and target sequences to 768 and 256 tokens, respectively. We train all models with a batch size of 256 using Adafactor Optimizer for both P3 and NIV2 instruction tuning. We train the model using 16 NVIDIA A100 GPUs (each with 40GB). Each training for top-k P3 and NIV2 instruction tuning requires less than 1 hour per epoch, and ends up 2-3 hours for 3 epochs.

Note that P3 and NIV2 meta-datasets are license free for research purpose and open-sourced with the code. 




§ EXAMPLES AND FORMULATION OF INSTRUCTIONS OF P3 AND NIV2.



 §.§ Examples of P3 and NIV2 Instructions.


<cit.> introduces natural language prompts for all datasets to enable zero-shot experimentation. They name it as prompt, template, and instruction, and we only use term instruction for this paper. The instruction they define consists of an input template and a target template, along with a collection of associated meta-data. The instructions are in functions mapping a data instance into natural language for the input and target sequences. For example, in the case of an NLI dataset, the example includes fields for Premise, Hypothesis, Label, and input instruction would be 


If {{Premise}} is true, is it also true that {{Hypothesis}}? 


The target instruction can be defined with the label choices {{Choices[label]}} but we only used input instruction without replacing the placeholder with actual data instance to find similar tasks.

In contrast with P3, NIV2 defines Instruction schema and divides the component as Definition, Positive Examples, Negative Examples. Definition defins a given task in natural language of how input text is expected to be mapped to an output text. Each task has a single Definition, and we only use Definition part as an instruction for each task and utilize it for task selection. The example below is Definition part of task1640, an Answerability Classification task. Note that, unlike P3 instruction, NIV2 instruction comprises of natural language without any placeholder.


Given a paragraph from a wikipedia article about some topic, and a question related to the topic, determine whether
the question is answerable from the paragraph. If the question is answerable, answer “True”, otherwise, answer
“False”.




 §.§ P3 Instruction Formulation


The original P3 instructions contained unique placeholders, which could potentially act as misleading shortcuts. In order to preserve the integrity of the instruction's original meaning, we have replaced these placeholders with the terms "{{text}}" and "{{candidate}}". In the classification task, if the options are provided within the instruction, we substitute them with "candidate"; otherwise, we use "text" for all replacements. Since we utilize only the input instruction and not the output label, we recommend examining the examples in the Table <ref> to understand the appearance of the original P3 instruction and how it has been modified.



 §.§ NIV2 Input Formats


<cit.> demonstrate various instruction composition using Task Definition, Positive Task Examples, Negative Task Examples, and Explanation. Out of various composition settings, we use [Def + Pos(2)] setting for Section <ref> and [Def] setting for Section <ref>. Figure <ref> and Figure <ref> represent input encoding for the above settings, respectively.




  

  



§ GPT-4 API GENERATION



 §.§ NIV2 Instruction Generation


To reproduce positive example for NIV2 instructions, we use GPT-4 API to generate similar instruction to original ones. Table <ref> represents API query used for instruction generation, and Table <ref> represents examples of generated instruction from GPT-4.



 §.§ Big-Bench Hard Instruction Generation


The BBH tasks don't have refined Task Definition like NIV2 datasets. In order to select relevant tasks using NIV2 datasets, and to compare the performance with Tk-Instruct, we generate instruction for BBH tasks using GPT-4. Table <ref> represents API query used for instruction generation, and Table <ref>, <ref> and <ref> show generated 27 instructions for each task.



§ PAIRWISE TRANSFER TASK SELECTION VS. INSTRUCTION-BASED TASK SELECTION


<cit.> measure pairwise transfer relationships on T5(3B) in all task pair of P3 meta-dataset. They measure the value of its average score on different instructions for every dataset, and they include instructions only related to the original tasks for evaluation. The T5(3B) + Pairwise Transfer model in Figure <ref> selects top-5 tasks that scored the highest transferability per target tasks. Figure <ref> represents top-5 selected tasks from T5(3B) + InsTa_Aligned-P3 and all the transferability scores. It can be verified that T5(3B) + InsTa_Aligned-P3 selects relevant tasks in accordance with pairwise transferability result in many tasks.



§ EXPERIMENTAL DETAIL FOR DATA SAMPLE-BASED TASK SELECTION


In our experiments, we adopt the data sample-based task selection approach as detailed in the experimental settings of <cit.>. To ascertain the similarity between tasks, previous works randomly sample 32 instances from each dataset, which are then paired with the corresponding instructions. Following this, embedding information are extracted using a dense retriever, and cosine similarities are calculated through matrix multiplication between the embedding vectors of the training and target tasks. Consistent with prior studies, we select 32 instances for all instructions in our training datasets, measure the similarity, and prioritize tasks that feature the highest scoring instructions. Apart from the task selection methodology, the instruction tuning procedures remain identical to those used in training the T5(3B) + InsTa model.





§ ANALYSIS ON SELECTING TOP-K RELEVANT TASKS


In our experiments, we utilize the top-k tasks selection approach, following previous work <cit.>, while acknowledging its potential risks. This method may occasionally include tasks with marginal relevance or exclude significant ones. Figure <ref> presents the performance of 11 unseen datasets in P3 as k varies. The experimental results show that training the model with all 35 tasks reveals a performance, but note that the optimal k for each dataset varies across datasets. WiC and Winogrande demonstrate optimal results when the k is 10, while other datasets perform best when the k is 5. 

Although we have attempted to use the selector's probability score to determine the cutoff, variations in the data used for training the selector and the type of models that could serve as selectors have introduced inconsistencies. These issues are recognized but beyond the scope of this study. Thus, we do not address this directly in our research and intend to investigate more precise measures of task relevance in future work.





§ SCALING THE NUMBER OF INSTRUCTIONS IN ALIGNMENT


Table <ref> shows how performance changes in the P3 meta-dataset based on the number of instructions learned during task selection. The same instructions are used across task selection, model training, and inference, but what varies is the number of instructions used for InsTa alignment. For training selector model using instructions for alignment, we scale the number of instructions by average of 2, 4, and 6.54, and 6.54 represents the aligned model InsTa_Aligned-P3 used in the entire paper.

Experimental results demonstrate an increase in the robustness of task selection as the number of instructions increases. We hypothesize that by learning a variety of instructions, the task selector becomes proficient in handling different styles of instruction formats. This enables it to effectively identify relevant tasks when presented with various formats of instructions for new tasks.



§ LEARNING DIFFERENT FORMATS OF META DATASETS AND ITS IMPACT ON PERFORMANCE


In the previous section, we explore how task selection performance varies with the number of instructions in the same meta dataset. In this section, we conduct experiments to see if learning meta datasets with different instruction formats together would lead to performance degradation compared to learning each meta dataset separately. Performance is measured on the held-out tasks of the P3 dataset, and the T5(3B) + InsTa_Aligned-(P3+NIV2) is further aligned using InsTa on instructions from both the P3 and NIV2 datasets. 

Our results, summarized in Table <ref>, indicate that the average performance T5(3B) + InsTa_Aligned-(P3+NIV2) of is on par with that of P3 alone, with some tasks even showing improved results. This evidence suggests that integrating the unique instructional style of the NIV2 dataset does not substantially impair the task selection capabilities in P3. It implies that training the task selector on diverse meta-datasets enables it to adapt to a broader range of instruction formats and tasks without declining performance.


















