LOB-Bench: Benchmarking Generative AI for Finance - with an Application to Limit Order Book Markets
finance;generative models;time series;state-space models;benchmark
Yes
No ethics review needed.
1) The framework does not account for hidden liquidity, which is critical for LOB modeling. The model can be improved in this area. 
2) Why did you focus on autoregressive models, (C)GANs, and agent-based models? Have you considered other deep learning architectures for LOB data generation, such as transformers or diffusion models?
3) Could you clarify why you selected the L1 and Wasserstein-1 distances as the primary metrics for distributional comparison? How do these metrics handle the complexities of financial time series data, such as fat-tailed distributions or volatility clustering?
4) The benchmark evaluates market impact metrics, but how do you account for outliers or extreme events in real-world LOB data? Are these adequately captured in your current framework?
5)  The conditional evaluation approach looks at conditional distributions of specific LOB statistics. How would this framework perform when dealing with very high-dimensional data, and does it scale well for more complex scenarios?
6) The paper uses the LOBSTER dataset for evaluation. Given its limitations (e.g., limited asset coverage, certain missing features), do you think the results would generalize to other datasets or more diverse market conditions?
1) The paper proposes unconditional and conditional evaluations, allowing for a nuanced comparison of generative models. It accounts for model drift and evaluates market impact metrics, making the framework robust.

2) The use of widely adopted LOBSTER data for empirical evaluations makes the benchmark relevant for practitioners in the financial domain.

3) The open-source code enhances the reproducibility and practical unity of the work.
The paper presents the LOB-Bench, a benchmark designed to evaluate the quality and realism of generative models for Limit Oder Book(LOB) data. It provides a theoretical framework and Python package that systematically compares generative models like autoregressive models, (C)gAN, and agent-based models. It uses a range of metrics like spread, order book volumes, order imbalance, and adversarial scores to asses the distributional differences between generated and real LOB data. Empirical results demonstrate that the autoregressive generative approach outperforms traditional models, offering a standardized evaluation for financial data generation.
1) The paper explores generative models; some more, like advanced deep learning architectures, could be explored.
2)  The evaluation highlights that errors accumulate over longer sequences. The benchmark does not fully address how to mitigate this. 
3) Certain model results with rate events like market orders have data sparsity. and generating additional data might not be practical.
Yes
No ethics review needed.
1. As mentioned above, this paper does not demonstrate that LOB-Bench addresses autoregressive gaps and distribution shifts effectively. Can you provide a detailed analysis or case studies that illustrate the benchmark's performance in managing these specific challenges?
2. Could you demonstrate how LOB-Bench differentiates between generated and real LOB data in certain scenarios where traditional methods such as cross-entropy fail to?
3. How does the proposed benchmark influence the development and refinement of generative AI models? Are there examples where insights derived from LOB-Bench can directly lead to improvements in model architecture or training methods?
1. This paper highlights the essential demands for quantitative metrics to evaluate generative AI for finance and addresses this critical problem by introducing several innovative quantitative metrics, which are well-motivated and useful for the further development of generative AI in finance.
2. This paper effectively employs visualizations to show the quantitative results, making them intuitive and comprehensive for a broad audience.
This paper proposes a unified benchmark framework, LOB-Bench, designed to quantitatively evaluate generative AI models with a focus on limit order book (LOB) data. Unlike prevailing qualitative measurements, this framework provides numerical results to analyze the accuracy of how the LOB generative models resemble the real data. Compared to current quantitative metrics in the literature, such as cross-entropy, LOB-Bench is claimed to better deal with issues like distribution shifts and autoregressive gaps within data. LOB-Bench is applied to evaluate a range of generative models. The experiment section presents some interesting findings. Specifically, the results indicate that autoregressive GenAI’s excel in learning LOB data more effectively than traditional methods.
1. This paper provides a non-anonymous code link on line 097, which violates the double-blind protocol of the conference. The authors should take care to remove or anonymize the link for the review process. 
2. This paper displays several signs of incomplete editing that require further proofreading. Specific issues include a non-functional GitHub link on line 033, and repeated sentences between lines 097-099. Before submitting, the authors should conduct a thorough proofreading pass, paying particular attention to consistency in links and removing any redundant text. 
3. Although the paper introduces a new benchmark for evaluating generative AI models, it does not adequately compare this benchmark with existing alternatives. Furthermore, the experiments fail to convincingly demonstrate the benchmark's claimed ability to effectively address distribution shifts or autoregressive gaps within the generated data. What are some existing benchmarks against which the proposed method can be compared against? Specific experiments that clearly demonstrate superiority in dealing with distribution shift or autoregressive gaps in the generated data would make the method more convincing.
Yes
* **Responsible research practice (data release)**: the used GOOG stock in the paper is not freely available undermining the authors' claims and the reproducibility of their results

* **Other reasons (anonymity violation)**: The authors provided, in the end of the introduction, a "clean link" to their GitHub repository where I can pinpoint the authors' affiliation and their identity.
Yes, Responsible research practice (e.g., human subjects, data release) Yes, Other reasons (please specify below)
See weaknesses, and the following:

1. Have you tried the KL-divergence for $\mathbb{D}$ in Eq. 1? Why is L1 more suitable? Can you run some experiments please to see this effect?
2. Does Figure 2 measure the L1 loss on all these dimensions?
3. Fig 15 $\rightarrow$ So, you trian a GAN on real and generated data from *lobs5* and you achieve an AUCROC of 82\%? This means that the generated orders are easily identified as fake, undermining what the authors claim in their original works. How come? What's the AUCROC for *coletta* and the *baseline*?
4. I might have missed it but what's the *baseline* model here?
The paper presents some interesting ideas about evaluating autoregressive models by binning the generated order into some aggregation function $\Phi$ to avoid accumulating errors in time. The quantitative evaluation metric is the L1 distance between $p(x)$ and $\hat{p}(x)$. What this means is still a mystery to me and I'd appreciate to have more clarification from the authors here.
The gist of the paper is to evaluate the realism of generated LOB orders. In more detail, the authors want to numerically assess the quality of the generated data instead of just using stylized facts, classic LOB evaluation format, nor only cross-entropy, usually used in generative approaches. The authors want to evalute pre-trained models in *sampling regime*.
### Discussion over the meaning of L1 as a qualitative evaluation metric
I appreciate the authors taking the effort of running all stylized facts experiments and stress-testing 2 SoTA and 1 baseline method. However, the authors claimed in the beginning that just having stylized facts, already thoroughly explored by Vitrienko et al. [5], isn't sufficient. While they support this claim by introducing the L1 as a qualitative metric, it lacks substance and overpresenting these stylized facts overshadows the "novelty" of the benchmark. Anyhow, to the best of my understanding, the L1 should quantitatively say how good a LOB generation model performs. Nevertheless, I argue that it'd be simpler and more than enough to train a model -- even a simple one -- on a financial downstream task -- say mid-price prediction -- on "fake" generated orders and measure its MSE. Then, at test time, one would measure the model's MSE on real orders and see how well the model trained on generated orders generalizes over the real data. This would be a sufficient indicator to assess the goodness and realism of the generated data. Furthermore, you wouldn't even need to bin your data and go through unnecessary hoops just for the sake of the benchmark. So, the question raised here is *Why is the literature in LOB generation "dying" to have your contribution? What are you bringing to the table that future works are going to benefit from?*

### Shortsightness and lack of discussion within the benchmark
As a benchmark paper, I would expect the authors to give insights and directions for future autoregressive/generative LOB models. This is lacking which undermines its usefulness. Again *What is it that future researchers would benefit from this benchmark?*

### Weaknesses
1. **Why did you share your GitHub repository not anonymized? Now, I know your identity and have reported it to the ACs**.
2. Lines 69-84 represent the authors' contribution and the writing is sloppy and all over the place. So are the authors trying to evaluate how good SoTA methods predict the mean-price on the newly generated orders? Doesn't this just cover the cross-entropy aspect that GenAI literature uses for evaluation? Where's the realism here?
3. It is unclear from lines 86-89 why model derailment is due to true data seeding. Care to discuss?
4. There's missing information about *coletta* and limited results for GOOG (lines 390-394). What missing info?
5. In Figure 6, it is evident that the trend of MO$_0$, and, especially, MO$_1$ aren't captured by *lobs5*. How do you justify this?

Minors:
1. There might be a missing reference in line 76 where you stat "Our aggregator functions are closely inspired by metrics used in literature,
such as spread etc.". How are they inspired by the literature? Which work specifically are you inspired from?
2. Who said that GANs are worst-case aggregators? Why is that?
3. It would be better to introduce already the models you're using in the benchmark in line 86. Why only 2 models if you're proposing a benchmark? There are plenty of other generic time series generation models that you can try to adapt for LOB [1-4]. 
4. Reference sections/equations correctly:
* e.g., line 265 $\rightarrow$ Equation~\eqref
* e.g., line 307 $\rightarrow$ Section~\ref
5. Lines 97-98 and 98-99 are the same sentence. This is where the authors also reveal their identity.
6. Nobody calls cross-entropy *xent*. Please amend this.


[1] Lim H, Kim M, Park S, Park N. Regular time-series generation using SGM. arXiv preprint arXiv:2301.08518. 2023 Jan 20.

[2] Li J, Wang X, Lin Y, Sinha A, Wellman M. Generating realistic stock market order streams. In Proceedings of the AAAI Conference on Artificial Intelligence 2020 Apr 3 (Vol. 34, No. 01, pp. 727-734).

[3] Shi Z, Cartlidge J. Neural Stochastic Agent-Based Limit Order Book Simulation: A Hybrid Methodology. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems 2023 May 30 (pp. 2481-2483).

[4] Hultin H, Hult H, Proutiere A, Samama S, Tarighati A. A generative model of a limit order book using recurrent neural networks. Quantitative Finance. 2023 Jun 3;23(6):931-58.

[5] Vyetrenko S, Byrd D, Petosa N, Mahfouz M, Dervovic D, Veloso M, Balch T. Get real: Realism metrics for robust limit order book market simulations. In Proceedings of the First ACM International Conference on AI in Finance 2020 Oct 15 (pp. 1-8).

## **Even in case the authors didn't reveal their identity, I would still have rejected this paper**
LOB-Bench offers a rigorous framework and open-source Python package for standardized evaluation of generative limit order book data models, addressing evaluation gaps and enhancing model comparisons with quantitative metrics.
@inproceedings{
anonymous2024lobbench,
title={{LOB}-Bench: Benchmarking Generative {AI} for Finance - with an Application to Limit Order Book Markets},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=XsYJ6yvgEC},
note={under review}
}
We present LOB-Bench, a benchmark designed to evaluate the quality and realism of generative message-by-order data for limit order books (LOB). We enable a rigorous and comprehensive model comparison by providing both a theoretical framework and an open-source Python package. Addressing the lack of consensus on evaluation paradigms in the literature, where qualitative comparison of stylized facts is prevalent, our work offers a crucial building block for advancing generative AI for financial data. LOB-Bench provides a standardized method to numerically assess the quality of various model classes that generate limit order book data in the widely used LOBSTER format. It provides a range of quantitative characteristics and uses a simple parametric benchmark model as a baseline. Potential model classes include autoregressive models, (C)GANs, and agent-based models. Our framework measures distributional differences in conditional and unconditional statistics between generated and real LOB data, supporting a flexible multivariate statistical evaluation. The benchmark features commonly used LOB statistics such as spread, order book volumes, order imbalance, and message inter-arrival times, along with adversarial scores derived from a neural network trained to differentiate between real and generated data. Additionally, LOB-Bench evaluates "market impact metrics" by computing cross-correlations and price response functions for specific events in the data. We present empirical benchmark results for a generative autoregressive state-space model and results for a (C)GAN and parametric LOB model. We find that the autoregressive GenAI approach beats traditional model classes.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
finance generative models time series state-space models benchmark
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/70519a37d960acf3711e308646a87e9af5c696c6.pdf
datasets and benchmarks
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
LOB-Bench: Benchmarking Generative AI for Finance - with an Application to Limit Order Book Markets
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission