SysCaps: Language Interfaces for Simulation Surrogates of Complex Systems
surrogate models;multimodal text and timeseries models;language-interfaced regression
Yes
No ethics review needed.
I would be interested in seeing the performance compared between time-series-centric models and current generic architectures.
- The authors provide a clear explanation of their setup, flow of data across multiple components and their evaluation and analysis.
- (I do not feel sufficiently well-acquainted with this domain to evaluate the predictive contribution or performance of the models.)
This paper describes a set of lightweight models to model complex energy systems, using an LLM to generate prompts and a encoder and bidirectional time-series model to predict energy consumption.
- I presume the authors' choice of models is due to resource constraints and aiming for a lightweight setup, but it feels like it has multiple components when it could be a simpler setup with fewer model components. For instance, a BERT-type model could also be used for time-series prediction (as opposed to only text encoding). Similarly the two-step process of generating prompts using a separate LLM and then encoding that prompt with an encoder could be avoided by just using the LLM directly and fine-tuning it.
Yes
No ethics review needed.
In addition to the points in the weakness:

1. Did the authors try to just templatize the sentences rather than generating them using an LLM, how would that impact performance (i.e. rather than telling an LLM to adhere to some constraint-based template, just have a sketch sentence and fill attribute values in the given sentence)?
2. Why wasn't RFE performed for the Wind Farm Wake modeling dataset, would performing RFE improve performance ?
3. Would the model not further improve if the SysCaps were generated using synonyms for the attributes, did the authors see the LLM generate different synonyms for the building or wind farm dataset? 
4. Do the authors believe that training on the subset of data where the caption quality assessed by the classifier model, would improve the overall model performance?
The paper is well-written and easy to follow. The authors motivate the problem well and empirically show improvements over 2 real-world datasets. Further, SysCaps can be used by non-expert users to understand the features of surrogate systems. The Design space exploration is insightful to show the features learned by the model.
The paper discusses the important challenge of building surrogate models for the prediction of simulation data. They specifically motivate the problem for complex energy systems(CES). These surrogate models often model system features as one hot vectors. The authors propose using text based descriptions to model these so-called surrogate systems with time series data. The text data is encoded as a dense embedding obtained from language models. The embedding is then fed to a bidirectional sequence encoder along with the time series data. 

The paper discusses the generation of the text pertaining to the attributes of such systems and proposes an automatic evaluation strategy for the same. 

For generating the captions the authors prompt an LLM with an in-context learning-based prompt that tunes the style and number of sentences. To evaluate the SysCap quality the authors train a multi-class classifier to check the attributes covered in the description generated by the LLM, using the text embedding. 

The authors show how including SysCaps along with time series data leads to improved performance against baselines that perform onehot encoding over attributes. The authors further show how training a custom embedding model can aid in improving time series prediction over just using a time series-based model. They further empirically prove how the embeddings are more robust to synonyms and missing data.
1. The paper claims the technique uses a pretrained text encoder for generating the embeddings, but then in section 5 mentions that the models are actually finetuned. This should be explicitly mentioned in the claims that the paper makes rather than just mentioning that a pretrained embedding is used. 
2. Further, the authors do not compare with the "said-pretrained" embeddings but only finetuned embeddings, and other SOTA embedding models for text encoding. 
3. The paper also claims that they train a system to evaluate the caption quality, the parameters of the said multiclass classifier are omitted from the paper.
4. The paper claims that for the CES building energy consumption dataset, the SysCaps-kv configuration works best, and for the turbine configuration the SysCaps-nl, there should be some discussion regarding the insights drawn from both cases and why the performance for both techniques are different. 
5. The authors claim that SysCaps would be useful for non-expert users, but lack the discussion if LLM-based explanations (complementary to the work done) can also aid in explaining the system attributes for surrogate models.
Yes
No ethics review needed.
* Q1) What's the advantage of the proposed approach using LLMs over more traditional template-based natural language captions? (e.g. "The building is <x> squared feet.", etc.)
* Q2) In Figure 1, the key-value template has only a colon to separate the key and the value. Have you tried adding a space in between? I expect 
* Q3) For the one-hot encodings, how do you deal with numeric inputs?
* Q4) In the results in Table 3, why did you expect longer captions to have larger error? I would have had the opposite intuition as shorter captions are more likely to miss important attributes.
* S1: Provides extensive empirical evaluation of the proposed system
* S2: The presentation is clear.
This paper studies the use of natural language captions as inputs to surrogate models that simulate "complex energy systems". These natural language captions describe the features of the system being simulated. The task is to predict a timeseries of some variable of interest that depends on these features and some other independent variable that is fed as a time series. The paper introduces an architecture that fuses the textual description with the time series data to achieve this goal.
The viability of the approach and its robustness to out-of-distribution perturbations are validated with a relatively extensive empirical evaluation, including different ablations of the system (such as one-hot encoding of the features, or no features), variations on the caption lengths or replacing words with synonyms.
* W1: The LightGBM baseline is underspecified. This baseline is the only one that stands as a reference point that is not an ablated version of the proposed model. However, as I understand it, LightGBM is a framework but not necessarily a model, so I don't really to which model this system is really being compared against.
* W2: Not very clear what is the added value of the proposal of using LLMs against simply using a template-based natural language description.
* W3: Despite the system is motivated on the potential intuitiveness of language interfaces to non-experts, no particular study is conducted on that front.
Augmenting surrogate models of complex systems with natural language interfaces.
@inproceedings{
anonymous2024syscaps,
title={SysCaps: Language Interfaces for Simulation Surrogates of Complex Systems},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=1R5BcYS8EC},
note={under review}
}
Surrogate models are used to predict the behavior of complex energy systems that are too expensive to simulate with traditional numerical methods. 
Our work introduces the use of language descriptions, which we call "system captions" or SysCaps, to interface with such surrogates. 
We argue that interacting with surrogates through text, particularly natural language, makes these models more accessible for both experts and non-experts.
We introduce a lightweight multimodal text and timeseries regression model and a training pipeline that uses large language models (LLMs) to synthesize high-quality captions from simulation metadata. 
Our experiments on two real-world simulators of buildings and wind farms show that our SysCaps-augmented surrogates have better accuracy on held-out systems than traditional methods while enjoying new generalization abilities, such as handling semantically related descriptions of the same test system.
Additional experiments also highlight the potential of SysCaps to unlock language-driven design space exploration and to regularize training through prompt augmentation.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
surrogate models multimodal text and timeseries models language-interfaced regression
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/de2706eabd3bc68eb7bfdb2d8584b607b5992841.pdf
applications to physical sciences (physics, chemistry, biology, etc.)
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/07f94bc944995380d0b4303a019399392a63b03c.zip
SysCaps: Language Interfaces for Simulation Surrogates of Complex Systems
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission