Make Your LLM Fully Utilize the Context
Shengnan An Zexiong Ma Zeqi Lin Nanning Zheng Jian-Guang Lou



Atmospheric Retrievals of the Phase-resolved Spectra of Irradiated Brown Dwarfs WD-0137B and EPIC-2122B
    [
    April 29, 2024
=======================================================================================================






While many contemporary large language models (LLMs) can process lengthy input, they still struggle to fully utilize information within the long context, known as the lost-in-the-middle challenge.

We hypothesize that it stems from insufficient explicit supervision during the long-context training, which fails to emphasize that any position in a long context can hold crucial information.

Based on this intuition, our study presents information-intensive (In2) training, a purely data-driven solution to overcome lost-in-the-middle.


Specifically, In2 training leverages a synthesized long-context question-answer dataset, where the answer requires (1) fine-grained information awareness on a short segment (∼128 tokens) within a synthesized long context (4K-32K tokens), and (2) the integration and reasoning of information from two or more short segments.
Through applying this information-intensive training on Mistral-7B, we present FilM-7B (FILl-in-the-Middle).
To thoroughly assess the ability of FilM-7B for utilizing long contexts, we design three probing tasks that encompass various context styles (document, code, and structured-data context) and information retrieval patterns (forward, backward, and bi-directional retrieval).
The probing results demonstrate that FilM-7B can robustly retrieve information from different positions in its 32K context window.
Beyond these probing tasks, FilM-7B significantly improves the performance on real-world long-context tasks (e.g., 23.5→26.9 F1 score on NarrativeQA), while maintaining a comparable performance on short-context tasks (e.g., 59.3→59.2 accuracy on MMLU).
Github Link: https://github.com/microsoft/FILM/tree/maingithub.com/microsoft/FILM.






































§ INTRODUCTION




    .9
    To a great mind, nothing is little.
    —Arthur Conan Doyle




Long-context large language models (LLMs) have recently received significant attention within the open-source community <cit.>.
The training context windows of many contemporary LLMs have been expanded to tens of thousands of tokens, thereby enabling these models to process extensive context as input.
This extended training context window can enhance many real-world downstream tasks such as long-context question answering <cit.> and summarization <cit.>.





However, recent studies have revealed that these long-context LLMs struggle to effectively and robustly utilize all the information provided in the context, known as the lost-in-the-middle challenge <cit.>.
It implies that while the LLM can comprehend the information at the beginning and end of the long context, it often overlooks the information in the middle.
This challenge could significantly hinder the development of long-context LLMs, as they even often fail to pass simple probing tasks such as Needle-in-the-Haystack and passkey retrieval <cit.>.
Consequently, a pressing research question arises:
how can we make long-context LLMs fully utilize the information in the long context?






We hypothesize that the root cause of lost-in-the-middle stems from the unintentional bias hidden in the general training data.
In auto-regressive pre-training, the loss on predicting the next token is more likely to be influenced by a few nearby pre-tokens rather than long-distance tokens <cit.>.
For supervised fine-tuning and alignment, the system message, which strongly influences the generation of the response, is typically presented at the beginning of the context <cit.>.
As a result, the general training process may inadvertently introduce a position bias, suggesting that important information is always located at the beginning and end of the context.






Based on this hypothesis, our work introduces information-intensive (In2) training to explicitly teach the model that the crucial information can be intensively present throughout the context, not just at the beginning and end.

In2 training is a purely data-driven solution that utilizes a synthesized long-context question-answer dataset.
The long context (ranging from 4K to 32K tokens) is concatenated from many short segments (∼128 tokens), and the question-answer (QA) pairs ask for the information contained in one or more segments which are randomly placed in the long context.
Specifically, we generate two types of questions, requiring (1) fine-grained information awareness on exactly one short segment, and (2) the integration and reasoning of information from two or more segments.
These QA pairs are generated by prompting GPT-4-Turbo <cit.> with the designed instructions and the raw segments.









By applying this information-intensive training on Mistral-7B <cit.>, we present FilM-7B (FILl-in-the-Middle).
To thoroughly assess the long-context information awareness of FilM-7B, we design three probing tasks encompassing various context styles (document, code, and structured-data context) and information retrieval patterns (forward, backward, and bi-directional retrieval).


The probing results (Figure <ref>) demonstrate that In2 training significantly overcomes the lost-in-the-middle problem for the backbone model.
Moreover, it can enhance the open-source model to achieve comparable or even more robust performance compared with proprietary LLMs such as GPT-4-Turbo.







Beyond these probing tasks, the performance of FilM-7B on real-world long-context tasks also exhibits significant improvements (e.g., 23.5→26.9 F1 score on NarrativeQA <cit.>).
This demonstrates that training on synthesized long-context data can be generalized to real-world scenarios.
Moreover, FilM-7B maintains a comparable performance on short-context tasks compared with the vanilla backbone model (e.g., 59.3→59.2 accuracy on MMLU <cit.>).
This indicates that the short-context capability of FilM-7B is not compromised during training.






The main contents of this paper are organized as follows.
Section <ref> introduces our In2 training with details on the data construction and training process.
Section <ref> introduces the design of our long-context probing tasks and the comparison with some existing probing tasks.
Section <ref> shows the experimental results on three probing tasks, nine real-world long-context tasks, and eight short-context tasks.
Section <ref> provides further insights for the long-context training strategies.
Section <ref> discusses the related work.












§ INFORMATION-INTENSIVE TRAINING


This section introduces the construction of the dataset for In2 training and the detailed training process of our model FilM-7B.





 §.§ Training Data Construction




  
Overview.
The In2 training aims to explicitly teach the model that any position in a long context can contain crucial information.
To achieve this goal, we construct a long-context question-answer training dataset 𝔻={ℒ_i, q_i, a_i}, where the answer a_i to the question q_i requires the information contained in some short segments that are randomly placed in the whole long context ℒ_i.

Figure <ref> illustrates an overview of the data construction process.
Specifically, the training data 𝔻 is constructed based on a general natural language corpus ℂ.
Given a raw text 𝒞_i∈ℂ, we first generate a question-answer pair (q_i, a_i) using a powerful LLM, then synthesize a long context ℒ_i that includes the necessary information from 𝒞_i and other randomly sampled texts from ℂ.
We generate two types of question-answer pairs that require (1) the awareness of fine-grained information in the long context, and (2) the integration and reasoning of information appearing at different positions in the long context.
We take the  subset from the C4 corpus <cit.> as ℂ, and take GPT-4-Turbo <cit.> as the LLM to generate QA pairs.









  
Fine-grained information awareness.
We consider a 128-token segment as the minimum information unit of the context[The raw texts in  have an average length of ∼600 tokens with the Mistral tokenizer.].
Given a raw text 𝒞_i, we first randomly extract a 128-token segment s_i from it, then generate the q_i, a_i and ℒ_i accordingly,

    (q_i, a_i)∼Prompting(s_i, I_f;LLM),  ℒ_i=⊕{Shuffle(s_i, [r_j])},

where (q_i, a_i) is sampled by prompting the powerful LLM with the segment s_i and the instruction I_f, ⊕{·} represents the concatenation of the contained segments, and [r_j] are randomly sampled from 128-token segments in ℂ.

Note that I_f instructs the LLM to make the question-answer pair highly specific to the information provided in s_i.





  
Integration and reasoning of information.
Beyond utilizing each single segment, we consider to generate question-answer pairs for information contained in two or more segments.
Following the setting of the minimum information unit above,

we split a full text 𝒞_i into a set of 128-token segments [s_i], then generate the q_i, a_i and ℒ_i accordingly,

    (q_i, a_i)∼Prompting([s_i], I_r;LLM),  ℒ_i=⊕{Shuffle([s_i], [r_j])},

where I_r instructs the LLM to generate a multi-hop question-answer pair that requires the information within at least two segments in [s_i].
All segments in [s_i] and [r_j] are jointly shuffled, so the required segments may appear far apart in the context.




  
Context length balance and data mixture.
To prevent length bias during In2 training, we ensure the length of the long context ℒ_i is evenly distributed from 4K to 32K tokens.
Such a length balance strategy can be implemented with reject sampling on [r_j], according to Equation <ref> and <ref>.
To alleviate catastrophic forgetting on short-context capabilities, we retain ∼10% question-answer pairs with the original texts 𝒞_i instead of converting them into a longer context, and add some general instruction-tuning data from the OpenOrca <cit.> dataset.

Overall, our dataset for In2 training contains 1.1M long-context data for the fine-grained information awareness (∼63%), 300K long-context data for the integration and reasoning of information (∼17%), 150K short-context question-answer data (∼9%), and 200K general instruction-tuning data (∼11%).
Appendix <ref> contains the handcraft instructions for data generation.
Appendix <ref> illustrates some examples of our constructed long-context QA data.
Appendix <ref> describes the filtering strategy to avoid data contamination for evaluation.





 §.§ Training Details


Using the training data constructed above, we further fine-tune the Mistral-7B-Instruct-v0.2[https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2.] <cit.> to get our FilM-7B (FILl-in-the-Middle).
We perform In2 training in the instruction-tuning paradigm:
the long contexts and questions are used as instructions, and the loss on the answer parts are used to update the model.
Appendix <ref> contains the system template used for formatting the training data.

For hyper-parameters, we set the global batch size as 128 and conduct one-epoch training with ∼14K training steps.
We use the cosine learning rate decay with a 1e-6 maximum learning rate and 3% warm-up steps.
The training process is conducted on 16 nodes of 8x80G A100 GPUs with the full sharding strategy and cpu offload strategy implemented by pytorch FSDP <cit.>.
The entire training process consumes ∼300 GPU days.







§ LONG-CONTEXT PROBING


In this section, we first show the preliminary evaluation of FilM-7B on the Needle-in-the-Haystack and discuss about the inadequacies of this probing task.
Subsequently, to comprehensively evaluate the long-context information awareness of FilM-7B, we introduce Various Long-context (VaL) Probing. This includes three tasks that cover various context styles (document, code, and structured-data context) and information retrieval patterns (forward, backward, and bi-directional retrieval).





 §.§ Near-Perfect Performance on Needle-in-the-Haystack: Are We There Yet?


The Needle-in-the-Haystack[https://github.com/gkamradt/LLMTest_NeedleInAHaystackhttps://github.com/gkamradt/LLMTest_NeedleInAHaystack.] task is widely used to assess how robustly a model utilizes information positioned in the long context.
It reveals that even some powerful proprietary LLMs, such as GPT-4 and Claude 2.1 <cit.>, struggle to fully exploit the information within the long context.




We use the Needle-in-the-Haystack task to preliminarily evaluate the long-context capability of FilM-7B.
Appendix <ref> demonstrates that FilM-7B has achieved near-perfect performance on this task.
This result is not surprising as recent open-source LLMs, such as LongAlign <cit.> and InternLM2 <cit.>, have also shown near-perfect performance on this task.





However, we argue that the near-perfect performance on Needle-in-the-Haystack may overestimate the long-context capabilities of LLMs, based on the following two considerations:


  * Needle-in-the-Haystack employs a document-style context, which LLMs could be quite familiar with due to the pre-training on natural language corpora.

  * The forward retrieval pattern in Needle-in-the-Haystack may simplify the difficulty of information seeking in the long context.

The “forward retrieval” means that the information being retrieved directly follows the retrieval keyword in a long context.
For example, the default question used in Needle-in-the-Haystack is "What is the best thing to do in San Francisco?" and the answer is contained in "The best thing to do in San Francisco is eat a sandwich and sit in Dolores Park on a sunny day."
The retrieved information "eat a sandwich and ..." just follows the retrieval keywords "best thing to do in San Francisco".
According to the mechanism of induction head <cit.>, such a following-up copying is an easily learned pattern for LLMs, thus less challenging for evaluating long context utilization.

Given these considerations, we suggest that performances on Needle-in-the-Haystack may not adequately reflect the long-context capabilities of LLMs.
Therefore, we propose VaL Probing for a more comprehensive evaluation involving various context styles and retrieval patterns.






















 §.§ VaL Probing


Our retrieval-based VaL Probing considers three context styles (document, code, and structured-data context) and three retrieval patterns (forward, backward, and bi-directional retrieval).
Each context in VaL Probing contains ∼32K tokens, and each task contains ∼3K examples. Figure <ref> briefly illustrates the contexts and retrieval instructions in VaL Probing.







  
Document Sentence Retrieval (Bi-Direction).
The contexts consist of numerous natural language sentences, and the instruction aims to retrieve a single sentence containing a given piece.
The sentences are sampled from the abstracts of papers on arXiv[https://info.arxiv.org/help/api/basics.htmlhttps://info.arxiv.org/help/api/basics.html.].
This task follows the bi-directional retrieval pattern, as the expected retrieval results contain words both before and after the given piece in the context.
The evaluation metric is the word-level recall score.



  
Code Function Retrieval (Backward).
The contexts consist of Python functions, and the instruction aims to retrieve the function name for a given line of code within the function definition.
The raw code functions are sampled from the StarCoder <cit.> dataset[https://huggingface.co/datasets/bigcode/starcoderdatahttps://huggingface.co/datasets/bigcode/starcoderdata.]. 
We randomly select three lines of definitions for each function.
This task follows the backward retrieval pattern, as the function name always precedes the definition.
The evaluation metric is the exact-match accuracy.



  
Database Entity Retrieval (Forward).
The contexts contain lists of structured entities, each with three fields: ID, label, and description.
The query aims to retrieve the label and description for a given ID.
The entities are sampled from Wikidata [https://www.wikidata.org/wiki/Wikidata:Data_accesshttps://www.wikidata.org/wiki/Wikidata:Data_access.]. 
This task follows the forward retrieval pattern, as the label and description follow the ID.
We take a relaxed exact-match accuracy as the metric: a 1 score is given if either the label or the description is exactly matched in the response, otherwise a 0 score.











































§ EXPERIMENTS AND ANALYSIS


We assess the long-context capability of FilM-7B on both probing tasks and real-world long-context tasks. Moreover, we investigate if the performance in short-context scenarios is affected.






 §.§ Experimental Setup




  
Models.
We mainly compare FilM-7B with long-context open-source models that have been trained with ≥32K context windows, including the Mistral <cit.>, LongChat <cit.>, ChatGLM <cit.>, LongAlign <cit.>, LongWanjuan <cit.>, Yi <cit.> and InternLM2 <cit.>.
We utilize the instruct/chat versions of these models as most of our evaluation tasks are under the zero-shot instruction-following paradigm.

We also draw comparisons with popular proprietary LLMs such as GPT-3.5-Turbo <cit.> and GPT-4-Turbo <cit.>.
All models and tasks employ greedy decoding.
For probing tasks, we primarily compare FilM-7B with LongAlign and InternLM2 series, as these models have shown near-perfect performances on Needle-in-the-Haystack.






  
Real-world long-context tasks.
We take 9 tasks from the LongBench <cit.> collection to evaluate the long-context capability on real-world scenarios.

These tasks encompass long-document question answering (NarrativeQA <cit.>, Qasper <cit.> and MultiFieldQA (MultiFQA) <cit.>, multi-document multi-hop reasoning (HotpotQA <cit.>, 2WikiMultihopQA (2WikiMQA) <cit.> and MuSiQue <cit.>), and long-context summarization (GovReport <cit.>, QMSum <cit.> and MultiNews <cit.>).


We employ the middle truncation strategy in LongBench to limit the input within 32K tokens.

We report ROUGE-L <cit.> for summarization tasks and F1 scores for other tasks.
The evaluation metrics are computed using the official evaluation scripts [https://github.com/THUDM/LongBenchhttps://github.com/THUDM/LongBench.].




  
Short-context tasks.
We select 8 short-context tasks commonly used for evaluating the general capabilities of models.
These include MMLU <cit.>, BoolQ <cit.>, RACE-High (RACE-H) <cit.>, CommonsenseQA (CSQA) <cit.>, ARC-Challenge (ARC-C) <cit.>, HellaSwag <cit.>, GSM8K <cit.>, and MATH <cit.>.

We use 5-shot for MMLU, 8-shot for GSM8K, 4-shot for MATH, and 0-shot for other tasks.
We utilize the [https://github.com/EleutherAI/lm-evaluation-harnesshttps://github.com/EleutherAI/lm-evaluation-harness.] for the evaluations on MMLU, BoolQ, RACE-H, ARC-C and HellaSwag, and use the evaluation scripts from <cit.> for other tasks.







 §.§ Main Results




  
FilM-7B significantly mitigates the lost-in-the-middle problem.
Figure <ref> presents the probing results for both FilM-7B and the backbone model, Mistral-7B-Instruct-v0.2.
In all three probing tasks within Val Probing, the vanilla Mistral model experiences substantial information loss at the middle positions in the long contexts.
In contrast, our FilM-7B model consistently exhibits robust performance across different positions within the whole context. 
This stark comparison illustrates that the lost-in-the-middle problem can be effectively addressed using our In2 training.






  
FilM-7B achieves performance comparable to, or even outperforming, that of GPT-4-Turbo.
Figure <ref> illustrates the comparison between FILM-7B and GPT-4-Turbo on our probing tasks.
Beyond a qualitative comparison between the performance curves of two models, we quantify the long-context performances on VaL Probing using two metrics:



  * Average score (Avg). We compute the average performances across the entire context length, reflecting the overall long-context utilization.

  * Min-max gap (Gap). We calculate the differences between the maximum and minimum performances in Figure <ref>. A smaller performance gap signifies greater robustness across different positions.

Table <ref> presents the quantified performances on VaL Probing.
It reveals that FilM-7B has comparable performance with GPT-4-Turbo on the database probing task, and exhibits better robustness in document and code probing tasks.
These results indicate a great potential for the development of open-source long-context models to close the gap with proprietary models.













  
VaL Probing presents a more challenging test suite for long-context models.
Figure <ref> and <ref> show the probing results of LongAlign and InternLM2, two state-of-the-art long-context models.
Despite their extended training context windows, these models still encounter the lost-in-the-middle problem.
This is particularly noteworthy given their near-perfect performance on the Needle-in-the-Haystack task. 
This comparison suggests that VaL Probing provides a more challenging evaluation for long-context models.

In particular, the results on document and database tasks in VaL Probing demonstrate clear comparisons with Needle-in-the-Haystack.
Compared to Needle-in-the-Haystack which uses forward retrieval on natural language context, the document task employs natural language context but with bi-directional retrieval, and the database task uses forward retrieval but with structured-data context.
These comparisons highlight that both context styles and retrieval patterns significantly contribute to the hardness of the probing tasks.















  
Training on synthesized long-context data effectively generalizes to real-world scenarios.
Table <ref> contains the results on various real-world long-context tasks.
It shows that FilM-7B also significantly improves the performance of the backbone model in real-world long-context scenarios.
Moreover, it also achieves SOTA-level performances on these tasks among ∼7B size open-source models.
Notably, the long contexts used in In2 training are all synthesized from short segments. 
These improvements suggest that the long-context capabilities learned from the synthesized data can be successfully applied to real-world tasks.








  
FilM-7B maintains the performance on short-context tasks.
Figure <ref> illustrates the performances of FilM-7B and the vanilla backbone model on short-context tasks.
It reveals that the overall performances on short-context tasks are almost comparable with minor variances. 
These results confirm that FilM-7B does not compromise the short-context capabilities of the backbone model.



 §.§ Training Strategy Analysis

Experimental results in Section <ref> demonstrate the feasibility of In2 training.
We aim to explore further into enhancing the effectiveness and efficiency of In2 training, particularly from the perspective of training strategies.
We are specifically interested in investigating the impact of the following two training strategies: applying the sliding window and adjusting the position encoding.
Considering the high cost of training, the following experiments use 20% of all training examples.



  
Models using sliding windows cannot effectively capture the long distance information.
Our experiments involving Mistral models, as shown in Figure <ref>, reveal that the performance of Mistral-7B-Instruct-v0.1 is awful when the information is positioned at a long distance. 
It's worth noting that Mistral-7B-Instruct-v0.1 employs the sliding window strategy while Mistral-7B-Instruct-v0.2 does not.
Consequently, we are interested in determining whether our In2 training can still alleviate the lost-in-the-middle problem under the sliding window strategy.
We conduct the following two experiments with a 4K sliding window during training:

    
  * Apply the sliding window in both pre-training and In2 training. We take the Mistral-7B-Instruct-v0.1 as the backbone model and conduct In2 training with the same window size (4K).
    
  * Apply the sliding window only during the In2 training. We take the Mistral-7B-Instruct-v0.2 as the backbone model and additionally apply a 4K sliding window during In2 training.

Figure <ref> illustrates the performances of models with sliding windows.
It shows that in both two settings with sliding windows, the performances drop dramatically when the distance between the retrieval question and information is longer than the sliding window size.
It reveals that the sliding window strategy greatly hurts the long-context capability of models.



  
Training with higher information intensity requires a larger RoPE base θ.
The training stage in Section <ref> follows the RoPE settings configured for the backbone model. 
Previous studies on context extension suggest that training with an extended context length necessitates a larger RoPE base θ <cit.>.
In the case of our In2 training, the context length remains unchanged, but the information intensity is significantly increased. 
As a result, we are interested in exploring whether the RoPE settings should also be adjusted to further enhance the In2 training.
Table <ref> shows the results with increasing the RoPE base θ from 1.0 × 10^6 to 1.0 × 10^8.
It shows that increasing the default RoPE base θ of the backbone model leads to better performances on VaL Probing.
We suggest to use a 10 times of the default RoPE base θ to conduct In2 training.






























§ RELATED WORK




  
Long-context LLMs.
Recent research has significantly contributed to the exploration of training large models with extended context windows <cit.>.
There are primarily two directions in the development of long-context LLMs.
(1) Data engineering, which emphasizes the construction of long-context data for training the LLMs. 
This includes data balancing <cit.>, data order arrangement <cit.>, instruction data collection <cit.>, and data quality measurement <cit.>. 
Our In2 training can be categorized into this field.
(2) Effective and efficient training, which investigates methods to optimize the training of a long-context model.
This encompasses the design of position encoding <cit.>, batching strategy <cit.>, parameter-efficient training <cit.>, and the development of new model architectures <cit.>.










  
Long-context evaluations.
Existing benchmarks for evaluating long-context models can be divided into two categories.
(1) Real-world benchmarks that assess general long-context capabilities (e.g., long-context QA, summarization, and language modeling), such as NarrativeQA <cit.>, LongBench <cit.>, ZeroSCROLLS <cit.>, L-Eval <cit.>, Loogle <cit.>, ∞Bench <cit.>, and a series of work on perplexity evaluation <cit.>.
(2) Probing tasks that provide a more concise reflection of the long-context utilization across different context lengths and positions. These include Needle-in-the-Haystack, passkey retrieval <cit.>, synthesized document QA <cit.>, Discovery <cit.>, RULER <cit.>, and the VaL Probing proposed in this study.
Among these probing tasks, our VaL Probing is the first to explicitly incorporate a variety of retrieval patterns.












§ CONCLUSION

This work introduces In2 training to overcome the lost-in-the-middle problem.
By applying In2 training on the open-source model, our FilM-7B exhibits significant improvements on probing tasks and real-world long-context tasks while does not compromise the short-context performance.






§ ACKNOWLEDGMENTS

Shengnan An and Nanning Zheng were supported in part by NSFC under grant No. 62088102.
Thank you to arXiv for use of its open access interoperability.



colm2024_conference






This is the Appendix of the paper: Make Your LLM Fully Utilize the Context.



§ DATA FILTERING STRATEGY
To avoid data contamination for the evaluation stage in Section <ref>, we apply a pre-filtering strategy during sampling the raw texts for constructing the dataset of In2 training.
Specifically, during sampling 𝒞_i for generating data, if the sampled 𝒞_i has a 10-gram overlap with any example in all of our evaluation data (including probing tasks, real-world tasks and short-context tasks), it will not be used for neither generating question-answer pairs nor serving as the random segments [r_j].




§ TRAINING EXAMPLES FOR IN2 TRAINING




Fine-Grained Information Awareness (1)data_fine_1
    

    Segment:
    
    Last summer I took a cross-country road trip with a group of people driving different types of electric vehicles.
    It was an amazing experience, and our trip, our adventures, and our misadventures were portrayed in the documentary Kick Gas.
    That experience gave me the idea to take my own road trip to experience the freedom of traveling alone on an electric motorcycle.
    My goals were simple: ride the great motorcycle roads of America, meet other electric motorcycle riders, and visit friends  ...

    Question:

    What were the primary goals of the individual's solo road trip on an electric motorcycle?
    
    Answer:

    The individual's primary goals for the solo road trip on an electric motorcycle were to ride the great motorcycle roads of America, meet other electric motorcycle riders, and visit friends.



Fine-Grained Information Awareness (2)data_fine_2
    

    Segment:
    
    Another addition to the kernel is the ar5523 driver, which was started over five years ago for the Atheros USB chipset of the same name. 
    Extensions for supporting more chips and WLAN adapters were added to a number of other drivers; the brcmsmac WLAN driver, for example, now supports the BCM43224 Broadcom chip, while the rt2800usb RaLink driver supports the Sweex LW323 USB WLAN adapter ...

    Question:

    What new addition to the kernel supports the BCM43224 Broadcom chip?
    
    Answer:

    The new addition to the kernel that supports the BCM43224 Broadcom chip is the brcmsmac WLAN driver.



Fine-Grained Information Awareness (3)data_fine_3
    

    Segment:
    
    The Arlington County Board plans to vote Saturday afternoon on giving Amazon $23 million and other incentives to build a headquarters campus in Crystal City, but only after hearing scores of northern Virginia residents and advocates testify for or against the project.
    The five-member board is expected to support the plan, which was announced amid much hoopla on Nov. 13. The proposed county incentives are part of an agreement in which Amazon would occupy significant office space and bring at least 25,000 high-paying jobs to Arlington in coming years.
    Opponents hope to postpone the vote until after additional public hearings, where they want representatives of the online retail giant to answer questions directly from anyone in the community.

    Question:

    What is the Arlington County Board's plan regarding Amazon's proposed headquarters in Crystal City?
    
    Answer:

    The Arlington County Board is planning to vote on providing Amazon with $23 million and other incentives to construct a headquarters campus in Crystal City. However, this vote will only take place after listening to the opinions of numerous residents and advocates from northern Virginia. The board, which consists of five members, is anticipated to endorse this plan. The proposal includes Amazon occupying a significant amount of office space and introducing at least 25,000 high-paying jobs to Arlington in the forthcoming years.




Integration and Reasoning of Information (1)data_reason_1
    

    Segment 1:
    
    Cloud Cruiser's utilization analytics application enables users to manage and optimize public, private and hybrid cloud usage and spending ...

    Segment 2:

    This reduces the risk of organizations investing too much or too little in IT, eliminates unused capacity and frees up valuable IT resources for new value-adding projects ...

    Segment 3:
    
    As a Cloud Cruiser customer, we have seen firsthand the value that Cloud Cruiser's technology creates by enabling HPE Flexible Capacity to meter and bill for usage of on-premise IT infrastructure in a pay-as-you-go model ...

    Question:

    What benefits does the acquisition of Cloud Cruiser bring to HPE and its Flexible Capacity business?
    
    Answer:

    The acquisition of Cloud Cruiser enriches HPE's Flexible Capacity business by providing a utilization analytics application that allows users to manage and optimize their cloud usage and spending. Cloud Cruiser's software also facilitates the metering and billing for customers' IT consumption. This enables HPE to offer its customers an on-premise physical IT infrastructure at the cost of cloud services, effectively a rental agreement. Moreover, it reduces the risk of organizations either under or over investing in IT, eliminates unused capacity, and frees up IT resources for new projects ...



Integration and Reasoning of Information (2)data_reason_2
    

    Segment 1:
    
    Julie Annette Wasden, 45, of 38 Roles St., Haines City, was charged with retail theft at Beall's at 5998 Cypress Gardens Blvd ...

    Segment 2:

    Katherine D. Collazo Santos, 18, of 108 Hilltop Drive, Winter Haven, was charged with retail theft at Macy's at 700 Third St., S.W. ...

    Question:

    Who was charged with retail theft at Beall's and who was charged with the same crime at Macy's?
    
    Answer:

    Julie Annette Wasden and Katherine D. Collazo Santos



Integration and Reasoning of Information (3)data_reason_3
    

    Segment 1:
    
    A court presided by Magistrate Joseph Mifsud explained in its judgement of 17 August, 2016 in The Police v Eebis Getu, that the charges of a false criminal report and calumnious accusations may be factually similar, however, from a legal point of view they are two separate and distinct charges.
    The accused, Eebis Getu was charged with having filed a false police report and with not obeying a legitimate order. She admitted these charges ...

    Segment 2:

    In passing judgement the court took into consideration that the accused admitted to the charges immediately and also that she did what she did in order to be with her husband in Malta.
    Magistrate Mifsud referred to what Pope Francis said last June, where today2̆019s information technology brings suffering of others instantly, but we also become immune to tragedies and sufferings ...

    Question:

    What were the two charges Eebis Getu admitted to, and what was her reason for committing these actions according to the court's judgement?
    
    Answer:

    Filing a false police report and not obeying a legitimate order; to be with her husband in Malta.









§ PERFORMANCE ON NEEDLE-IN-THE-HAYSTACK




Figure <ref> shows the performance of FilM-7B on Needle-in-the-Haystack.
It shows that FilM-7B has achieved near-perfect performance on Needle-in-the-Haystack within its 32K context window.




§ PROMPTS FOR DATA GENERATION AND TRAINING


Prompt For Equation <ref>prompt_fine
    Generate one question and the answer from the given context.
    The question should be highly specific to the information provided in the context.
    It should not be a general question that suits any context.

    Rules to follow when generate the question:

    1. The question should be fully answerable from information present in given context.

    2. Make sure the question is clear and unambiguous.

    3. Phrases like 'based on the provided context', 'according to the context', etc, are not allowed to appear in the question.

    Rules to follow when generate the answer:

    1. The answer must use the information provided in the context.

    2. Do not just copy words from the context. Answer the question in your own words.


    ### Context ###:

    s_i


    ### Question ###:

    {completion}



Prompt For Equation <ref>prompt_reason
    Generate one question and the answer from the given context.
    The context contains several pieces.
    Answering the question should require the reader to make multiple logical connections or inferences using **at least two pieces**.

    Rules to follow when generate the question:

    1. The question should be fully answerable from information present in given context.

    2. Make sure the question is clear and unambiguous.

    3. Phrases like 'based on the provided context', 'according to the context', etc, are not allowed to appear in the question.

    Rules to follow when generate the answer:

    1. The answer must use the information provided in the context.

    2. Do not just copy words from the context. Answer the question in your own words.


    ### Context ###:

    # Piece 1: s_i^1

    # Piece 2: s_i^2

    ...


    ### Question ###:

    {completion}



Training Templateprompt_training
    

    Input:

    [INST] Below is a context and an instruction.
    Based on the information provided in the context, write a response for the instruction.


    ### Context:

    ℒ_i


    ### Instruction:

    q_i [/INST]


    Output:

    a_i






