Knowledge Augmentation: In-context or In-parameter?
Representation Learning;In-Parameter Knowledge Injection;Parametric Knowledge Representation;Large Language Models
Yes
No ethics review needed.
See weakness
The experimental results show the effectiveness of In-Parameter Knowledge Injection method on 1B, 3B, and 8B LLMs. 

The idea is novel. Embedding knowledge through model parameters will ideally represent how a LLM “understand” certain external knowledge.
This paper introduces a method named In-Parameter Knowledge Injection to integrate external knowledge into the large language models. Different from the in-context learning methods that adopt natural language to represent external knowledge, the in-parameter method represents knowledge through parameters, thus avoiding length constraints and becoming more compatible with the foundation LLM.
1. As shown in Figure 3, knowledge can be effectively represented and understood through either natural language or parameters. So, which kind of knowledge should be integrated through parameters? More explorations and investigations are recommended here. 

2. The method demands additional pre-training or post-training costs. I suggest incorporating an additional baseline method where a copy of the LLM is adopted to represent the knowledge. The baseline will also demonstrate the contribution of knowledge encoding phrase.

3. Will the In-Parameter Knowledge Injection method generalize to LLMs with a larger scale (e.g., Llama 3.1 70B Instruct, Llama 3.2 11B instruct, etc.)? It’s unclear about the exact contribution of the method, since the model scale is relatively small.
Yes
No ethics review needed.
1. How might in-parameter knowledge injection perform in tasks beyond fact-based reasoning, such as creative writing or dialogue generation?

2. Can the authors elaborate on how their work differentiates from previous studies on knowledge augmentation and knowledge editing in language models?

3. What challenges might arise when scaling in-parameter knowledge injection to larger models or datasets, and how could these be addressed?
• Systematic Comparison: The paper offers a clear and structured comparison between in-context and in-parameter knowledge injection methods, elucidating the trade-offs in different scenarios.

• Experimental Design: The progression from simple to complex tasks effectively demonstrates how each method scales, offering insights into their applicability across various task demands.
The paper investigates two methods of knowledge augmentation in language models: in-context knowledge injection and in-parameter knowledge injection. In-context augmentation involves adding external information directly to the model’s input prompts, while in-parameter augmentation temporarily embeds this knowledge into the model’s parameters. Through a series of tasks with increasing complexity—ranging from fact extraction to comparative and multi-step reasoning—the study evaluates the effectiveness of both methods. The findings indicate that in-parameter injection performs better on complex reasoning tasks, whereas in-context methods are more effective for simpler fact extraction. The authors provide insights into the advantages of each approach based on task complexity and computational efficiency.
• Limited Novelty: The in-parameter knowledge injection method is essentially LoRA with additionally synthetic augmentation. The paper does not sufficiently acknowledge this overlap or explain how its approach differs from or improves upon existing parameter-efficient fine-tuning methods.

• Insufficient Differentiation from LoRA: Given the similarities to LoRA, the paper should have provided a detailed comparison, highlighting any unique contributions or advantages. 

• Scope of Evaluation: The experiments focus on a limited range of tasks. Expanding the evaluation to include more diverse or real-world applications could enhance the robustness and generalizability of the conclusions. 

• Lack of Theoretical Advancement: The paper does not offer new theoretical insights into knowledge augmentation or parameter adaptation in language models, limiting its contribution to an empirical comparison that may already be addressed in existing literature.
Yes
No ethics review needed.
1. How will the proposed solution hurt the model's general performance?
2. Can you present a more detailed analysis of the QA extraction quality? Especially when we do not have a strong external model, can we use the same model to do that, and how will that influence the final performance?
1. The proposed method is sound, and the introduction is clear.
2. The experiment setting is comprehensive.
This paper studies different ways of injecting new knowledge into the LLMs. Specifically, it proposes a new way of first extracting QA pairs from contexts and then training the model with LORA using the QA pairs as a new in-parametric knowledge injection solution. Following that, this paper compares the proposed solution with the widely used in-context solution on different tasks. Experiment results show that on simpler tasks, the in-context solution is still stronger, but on more complex tasks that require complex reasoning of the updated knowledge, the proposed in-parameter solution might be helpful.
The proposed solution has several bottlenecks: (1) the quality of extracted QA pairs is crucial for the effectiveness of the knowledge injection. For example, the context might contain many details, and it is unlikely that the generated QA pairs could cover all of them, so the proposed solution will lose those details. (2) LORA is a way for efficient tuning. There is no guarantee that the model learns all the knowledge in the extracted QA pairs; (3) since LORA directly changes all the parameters, this might hurt the model's performance on other tasks.
Yes
No ethics review needed.
1.	What is the difference between in-parameter knowledge injection and knowledge editing?
2.	What is the difference between IP and LLM-based data augmentation + LoRA? If IP is not an online method where each test example is augmented with different LoRA weights?
3.	In Section 3.3, the authors mention the comparison of the complexity. Is there any comparison of IC and IP on efficiency, e.g, with FLOPs or time?


Typos:
1.	Line 50, no year in the citation; Line 516-517, wrong citation formats
1.	The topic is important and interesting. There is indeed a limitation with in-context fixed language-based knowledge. 
2.	The experiments are comprehensive, with an evaluation of various models and settings. A good number of details of the decisions and prompts are provided in the appendix. The test set suite is also good. All the tasks included are indeed relevant and require external knowledge.
3.	This paper is well-written, the paragraphs and figures are clear and easy to follow.
The paper discusses the differences in two kinds of ways to augment language models with external knowledge: in-context learning or in-parameter augmentation. The authors propose to use LoRA parameters tuned from QAs that are synthetically generated by language models to augment the models. Experiments on 1B-sized models show good performance on various knowledge-intensive tasks such as fact extraction and comparative reasoning.
1.	More work in the literature can be discussed in Section 2. The discussion on in-context/in-parameter knowledge injection can be improved with more details in the literature, e.g., a similar setting of parameter updates with knowledge updates is defined in the paper Fast Model Editing at Scale (Mitchell, et al., 2022). Also, much previous work on parameter-efficient fine-tuning (PEFT) is relevant but not discussed, e.g., Adapter.
2.	From my understanding of the paper, the LoRA weights are tuned from QAs from the whole knowledge base. If so, what is the difference between the proposed in-parameter method and LLM-based data augmentation? This seems to limit the novelty of the method part. Also, it may influence the fairness of comparing with in-context learning. How are the exemplars chosen? Given the current framework, it can be extended to a novel method if, for each test example, you consider different LoRA weights associated with different data points, groups, and external knowledge source K. This paper may provide further intuition along this direction: https://arxiv.org/pdf/2110.04366.
3.	It is quite surprising to me that in-context learning yields such bad performance on comparative tasks. It is possible that the model sizes might limit the reasoning capability. It would be good if the authors could provide the ICL performance of larger models (e.g., 70/405B Llama) as a reference or extend the IP experiments on these models.
This paper propose a novel in-parameter knowledge injection method and  compare both in-context and in-parameter knowledge injection method for generative language models across a wide range of tasks.
@inproceedings{
anonymous2024knowledge,
title={Knowledge Augmentation: In-context or In-parameter?},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=sl4hOq9wm9},
note={under review}
}
Generative language models rely on knowledge augmentation to enhance their ability to complete a variety of tasks by incorporating relevant external information. The most common approach is in-context knowledge injection, where the relevant information is appended directly to the model’s input. While straightforward and easy to implement, this approach is limited for complex reasoning tasks due to input length constraints and shallow integration of external and internal knowledge in language models. 
To overcome these limitations, we introduce an in-parameter knowledge injection method, which temporarily embeds external knowledge into the model's parameters. By injecting knowledge in this way, the language models can access and reason over the information more flexibly, bringing enhanced performance on tasks requiring sophisticated reasoning. 
We conduct deep explorations of both in-context and in-parameter knowledge injection, highlighting their respective advantages and limitations. Through comprehensive experiments across tasks of varying complexity, we demonstrate that in-parameter knowledge injection is particularly advantageous for complex tasks requiring deep reasoning, while in-context injection remains effective for simpler tasks where the answer can be directly extracted. Our findings provide practical guidance for selecting appropriate knowledge augmentation strategies based on the complexity of the task. 

We have open-sourced all the code, data, and models in the following anonymous GitHub link: https://anonymous.4open.science/r/In-parameter-Knowledge-Injection/
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Representation Learning In-Parameter Knowledge Injection Parametric Knowledge Representation Large Language Models
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/13c0a39b4453c9da6ecbd1ffcb0c3b33c8838f1b.pdf
unsupervised, self-supervised, semi-supervised, and supervised representation learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/4aa09586f9b66a026275c91131013172de4deebb.zip
Knowledge Augmentation: In-context or In-parameter?
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission