On the Sequence Evaluation based on Stochastic Processes
stochastic representation;stochastic process;Brownian bridge;text coherence;human-AI differentiation
Yes
No ethics review needed.
See above
1. The SP Encoder's performance on out-of-domain datasets highlights its robustness and generalizability.
2. The theoretical analysis is solid.
This paper explores the structural and temporal characteristics encoded in the stochastic representation of latent trajectories and their applications in NLP tasks through theoretical and empirical studies. It introduces a flexible coherence evaluation metric (SPM) that is not influenced by individual article properties like text length. To verify this, the authors designed a Mixed Shuffle test based on the established Shuffle test. They also discovered that their metric, which evaluates the fit of target stochastic processes, can help distinguish between human-written and AI-generated data, indicating that the stochastic representation encodes useful properties for human-AI discrimination.
1. The paper does not adequately address the relationship with BBScore, another metric that uses a Brownian Bridge-Based approach for sequence evaluation. This omission limits the perceived novelty of the proposed method. 
2. The introduction and methodology sections are not well organized, making it difficult to follow the flow of ideas and understand the proposed approach. 
3. The experiments only use GPT-2 as the backbone model, which limits the generalizability of the findings. To provide a more comprehensive evaluation, the proposed methods should be tested on a wider range of representative large language models (LLMs), such as LLaMA and others.
Yes
No ethics review needed.
- How does SPM perform in real-time applications, given the computational demands of likelihood-based methods?
- How does SPM compare to transformer-based coherence evaluation metrics in terms of both performance and computational efficiency?
- The paper introduces a novel SPM that effectively captures temporal and structural dependencies, showcasing a new direction in long-sequence evaluation.
- Leveraging the Brownian Bridge model adds theoretical robustness, enhancing coherence evaluation.
Robust Encoder Design: The SP Encoder, with its negative log-likelihood loss, is shown to outperform contrastive learning approaches in out-of-domain tasks, adding versatility.
- The metric’s success in distinguishing human from AI-generated text is a practical application with high relevance.
- SPM is demonstrated to work well across tasks, including mixed shuffle tests, with flexibility across varying text lengths.
This paper introduces a new method for evaluating the coherence of long text sequences using a stochastic process model called the Brownian Bridge. The main metric, the Stochastic Process Metric (SPM), captures both time-related and structural details, helping to measure text coherence and distinguish between human and AI-generated text. The paper also introduces an SP Encoder that uses a negative log-likelihood objective, which performs well on out-of-domain tasks and improves text sequence analysis. The results suggest that SPM and the SP Encoder could be valuable for text coherence evaluation and human-AI discrimination tasks.
Overall, this paper is good, with no significant weaknesses. It may be further improved by addressing the following considerations:

- The reliance on stochastic processes and likelihood estimation might increase computational demands, which is not thoroughly addressed.
- SPM’s struggles with capturing local coherence perturbations suggest that it may overlook smaller, context-specific coherence challenges.
- The SP Encoder’s success partly relies on domain-specific parameters, which might limit its application across highly diverse domains without retraining.
Yes
No ethics review needed.
(just the ones above)
* The technical presentation of this work is strong. It provides a clear explanation of the methods used and how they can be adapted to a text evaluation task. The argument for non-diagonal \Sigma and to attempt full MLE is reasonable, and the motivation for using this approach is valid. 

* While not a novel method per se, this line of research is underexplored compared to other papers in the language modeling space. It would be good to see more creative uses of these methods in practice. 

* The zero-shot performance of human-AI detection seems promising. It is good that a well motivated probabilistic model can discriminate systems based on purely statistical properties (although would be curious to see the results on more recent LLMs)
This paper explores the use of Brownian Bridges as a type of language model encoding. Specifically the model is used to represent the temporal dynamics in embedding space. The hope is that non-typical generations that lack structural coherence will be easy to discriminate from generations that have a more typical coherence pattern. To do this the authors fit a BB model on to the output of GPT-2, and then evaluate other documents (synthetic and natural) in an unsupervised manner. 

The use of BB models has been previously explored, most notably in "Language modeling via stochastic processes" (Wang, 2022). The authors distinguish this work from the previous work by switching to a non-identity covariance term and by fitting the model using MLE versus a contrastive approximation.
* The applications in the work are primarily of moderate-to-low interest as applications in NLP. While the Shuffle test and Entity Grid based models are of critical historical importance in NLP they are not actively used in modern systems. The results given are similar to other approaches for these tasks and serve primarily to validate the approach as opposed to improve metrics. 

* The zero shot detection task from detect-GPT is a bit more promising, but it was unclear to me why this method was only applied on a small subset of the data sets that were used in that paper. 

* Results comparing the main contribution of the work MLE based SP seem to be mixed-to-negative? It would be useful to have a better sense of how the authors see these results and if they justify the more costly procedure.

* The main thing lacking from the work seems to be any mention of text generation. Given that Wang 2022 is able to generate it is unclear to me why these results are not included or studied. 

* It seems as if there is a contrastive style triplet approximation being used during MLE training. Given that step, it is less clear to me what the advantages are as an algorithm with contrastive loss.
@inproceedings{
anonymous2024on,
title={On the Sequence Evaluation based on Stochastic Processes},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=A53m6yce21},
note={under review}
}
Generative models have gained significant prominence in Natural Language Processing (NLP), especially in tackling the complex task of modeling and evaluating long text sequences. This task is crucial for advancing various downstream applications, such as text generation and machine translation. Recent methods that utilize stochastic processes to capture the intrinsic dynamics of sequences have shown superior performance in generative modeling. However, the accurate encoding of both temporal and structural dependencies from text datasets, as well as leveraging this encoded information for sequence evaluation, remains an open area of research. In this paper, we propose a novel approach to learn the stochastic dynamics of long text sequences, utilizing a negative log-likelihood-based encoder that outperforms contrastive learning methods. We also introduce a likelihood-based evaluation metric for long-text assessment, which measures sequence coherence and can be applied to downstream tasks such as Human-AI discrimination. Our encoder preserves sequence coherence effectively and performs robustly on out-of-domain datasets. Additionally, the proposed evaluation metric captures both temporal and structural information comprehensively. Theoretical analysis demonstrates the superiority of our metric in sequence evaluation, and experimental results highlight its flexibility and exceptional performance across a variety of tasks, showcasing its utility in diverse NLP applications.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
stochastic representation stochastic process Brownian bridge text coherence human-AI differentiation
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/9609a00a3c253f12befb674acdd62749c0ee9b6d.pdf
applications to computer vision, audio, language, and other modalities
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/b1c827d7d4b2a5275a977f7fbcfdb65775288da2.zip
On the Sequence Evaluation based on Stochastic Processes
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission