Online Detection for Black-Box Large Language Models with Adaptive Prompt Selection
online change detection;LLM security;active prompt selection
Yes
No ethics review needed.
Generalizability: Could the authors provide more experimental results on the generalizability of the algorithm across different types and scales of LLMs?
Adversarial Attacks: Have the authors considered the impact of adversarial attacks on the detection algorithm, and how could the robustness against such attacks be improved?
Real-time Performance: Could the authors provide data on the performance of the algorithm in real-time or near-real-time environments?
Computational Efficiency: Could the authors further discuss possible methods for optimizing the computational efficiency of the algorithm in resource-constrained settings?
Innovation: The paper proposes an online change detection approach tailored for black-box LLMs, which is a relatively new research direction and shows high innovation.
Methodology: The combination of CUSUM-type detection statistics and UCB strategy enhances the accuracy and efficiency of change detection.
Experimental Design: The validation of the method on both synthetic and real-world data, including different change scenarios, strengthens the paper's credibility.
Practical Application: The method is significant for ensuring the stability and reliability of LLMs in commercial and safety-critical applications.
The paper presents a novel online change detection method for black-box Large Language Models (LLMs), combining a CUSUM-type detection statistic based on entropy and Gini coefficient with a UCB-based adaptive prompt selection strategy to effectively detect changes in LLMs. The experimental results demonstrate the method's effectiveness on both synthetic and real-world APIs, covering various change scenarios.
Generalizability: The paper could benefit from additional experiments to demonstrate the generalizability of the method across more datasets and application scenarios.
Computational Resources: While the paper mentions considerations for computational resources, a deeper discussion on optimizing the algorithm for resource-constrained environments would be beneficial.
Adversarial Attacks: The paper does not address the robustness of the detection system against adversarial attacks designed to evade detection.
Yes
No ethics review needed.
See the weakness section above
+ The paper is easy to follow
+ The author tries to propose a new topic related to LLM security
The paper proposes an online changepoint detection method aimed at identifying changes in black-box LLM APIs. The method is evaluated on synthetic data through simulated watermarking and version updates
First of all, I found the study itself a little bit weird. The author claims that the detection is for "tampering, malicious prompt injection, or unauthorized parameter updates" at the very beginning of the paper. However, as the user of the API(since it is the black-box setting, I can only assume that the user wants to detect the API change since it makes no sense that the LLM developer wants to detect LLM change in the black-box setting), I don't find it very convincing to claim the ''unauthorized'' parameter updates as a concern that we even need a detector for that. The LLM provider has the right to update their model, and those trusted providers like OpenAI and Anthropic often inform users of the updates and have checkpoints for older versions. Also, the watermark or fingerprint is a method to protect the model or response property [1] , leaking them would make it easier for intellectual leakage. And the decoding changes should not be necessary connected with the watermark. For example, having a more efficient decoding method like [3], or decoding-phase alignment[4] could also lead to that. There is not necessary proof that it is harmful and I don't understand why it is a concern that the user must have the detection against it. For those untrusted LLM API/interface providers, then it could be a risk by prompt injection. For example, custom GPT store or Coze have lots of uncensored bots that the system prompt can be changed into malicious and leading to security or privacy risks[2]. From my side, I can only agree that this could be a risk that we should take care of. However, even though the author mentions the prompt injection in the introduction and abstract, there's no further study on that. Please clarify and provide specific scenarios where detecting such changes is necessary.

Second, I don't find the detection in this setting very challenging. Since some experiment details are missing, I can only assume that the API usage is with a fixed system prompt and zero temperature. In such case, the response is expected to be deterministic. If I see the changes as shown in Fig1, I should know obviously that the API might change. I don't think there's a need for such algorithm proposed in the paper. Could you clarify the experimental setting and provide the LLM response between different LLM versions? Also, have you considered some simple methods, such as similarity in the text level? The lack of baseline methods also makes the contribution less significant.

For the false positive measurement,  I believe a more accurate method is to evaluate the same model with different temperatures or different system prompts since this is more practical in the real-world. For example, the ChaGPT web interface's system prompt and temperature are unknown to the user(let' put aside why this is a risk). Using generating a different answer can sometimes have a huge difference in the response. It is a more practical case and the false positives should be take care with if we really need a detector here. How would your detector perform in this scenario?

Minor weakness: no board impact nor limitation discussed in the paper, also, how the author generated those prompts in the table are unknown

[1] Instructional Fingerprinting of Large Language Models

[2] On the (In)Security of LLM App Stores

[3] Fast Inference from Transformers via Speculative Decoding

[4] Decoding-time Realignment of Language Models
Yes
No ethics review needed.
1. What is the threat model or detector's operation surface here? Does the detector have access to API and thus to log probabilities produced by it?
- The method is running aggregation of simple statistical metrics and is easy to compute.
- Strong empirical results for such simple metrics in detecting a change in a given model with little delay.
- The experiment covers both white-box and black-box API models.
This paper offers a CUSUM statistical method to detect if there is any change in underlying language models based on prompts. Towards this, it introduces a statistic based on entropy and Gini coefficient. The different methods introduced in the paper are effective in black-box settings and adaptive. The paper also discusses the detection of watermarking being enabled by model providers.
- The prerequisites of the detector are unclear. The assumptions and level of access/information are missing and therefore the precise motivation/utility behind detection is unclear. The change in model is well advertised in "real-world APIs" and is publicly available information.
- If the detector has access to log probabilities from the model API, then it can easily hash them for each prompt and observe changes in them. The tok-K log probabilities are generally available for frontier SOTA models like GPT4, Gemini, and Claude. This simple baseline (for the detection of change in the model, not watermarking explicitly) is missing. 
- The experiments only cover single instances of changes in the model. It would be great to see the True Positive Rate @ low False Positive/Alarm Rate aggregated over multiple different changes in API models. This can be difficult in black-box API case but can be easily done in a simulated case. 
- For experiments, there are no baselines identified/used making it hard to evaluate effectiveness of method and the task. The related work section is poor.
Yes
N/A
No ethics review needed.
- Could you clarify the calculation methods for NTE and NTG? Specifically, how are the tokens merged to estimate the distribution?
- Will different decoding strategies affect the robustness of the detection? For instance, is the model change less likely to be detected with nucleus sampling than with greedy decoding?
- The paper addresses the important issue of detecting online changes in black-box LLMs.
- The proposed approach is practical and applicable across different type of models, as it comes with minimal assumptions of model itself.
- The UCB-based adaptive prompt selection enables cost-effective detection by focusing on the most sensitive prompts.
- The method is robust, maintaining good ADD with a low false positive rate (high ARL).
This work proposes a detection algorithm for online black-box LLM changes. The detection leverages a CUSUM-type statistic that incorporates entropy and the Gini coefficient. To enable effective detection, it adopts a UCB-based adaptive prompt selection strategy to identify the most change-sensitive prompts. In evaluations, the algorithm demonstrates effectiveness in identifying various backbone changes and model watermarking.
- While the authors discuss hyper-parameter selection, further ablation studies on individual hyper-parameters (e.g., C and N) would help clarify their specific effects, rather than assessing them as a whole.
- Due to computational limitations, most experiments are conducted on relatively small models. An analysis of larger models (e.g., Vicuna-7B) would provide more insights into real practice. For example, how ARL and ADD will change with larger model sizes? Will this method still be robust with larger models?
- There are limited experiments on real-world APIs with LLM updates. Additional results on version changes will be helpful. For example, the author can test different versions of gpt-4o or gpt-4 with different dates from OpenAI API.
This paper proposed a novel online detection algorithm to detect changes happened in the LLM, with an adaptive prompt selection strategy to enhance detection power.
@inproceedings{
anonymous2024online,
title={Online Detection for Black-Box Large Language Models with Adaptive Prompt Selection},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=fwHVclv0ij},
note={under review}
}
The widespread success of large language models (LLMs) has made them integral to various applications, yet security and reliability concerns are growing. It now becomes critical to safeguard LLMs from unintended changes caused by tampering, malicious prompt injection, or unauthorized parameter updates, etc. Early detection of these changes is essential to maintain the performance, fairness, and trustworthiness of LLM-powered applications. However, in black-box settings, where access to model parameters and output probabilities is unavailable, few detection methods exist. In this paper, we propose a novel online change-point detection method for quickly detecting changes in black-box LLMs. Our method features several key innovations: 1) we derive a CUSUM-type detection statistic based on the entropy and the Gini coefficient of the response distribution, and 2) we utilize a UCB-based adaptive prompt selection strategy for identifying change-sensitive prompts to enhance detection. We evaluate the effectiveness of the proposed method using synthetic data, where changes are simulated through watermarking and model version updates. Our proposed method is able to detect changes quickly while well controlling the false alarm rate. Moreover, for real-world data, our method also accurately detects announced changes in LLM APIs via daily online interactions with APIs. We also demonstrate strong evidence of unreported changes in APIs, which may be of independent interest.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
online change detection LLM security active prompt selection
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/4671498a0e3d98e46e8cb9af8f0704050cb84eab.pdf
generative models
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Online Detection for Black-Box Large Language Models with Adaptive Prompt Selection
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission