NIAQUE: Neural Interpretable Any-Quantile Estimation - Towards Large Probabilistic Regression Models
deep probabilistic regression;large regression models
Yes
No ethics review needed.
See weaknesses.
- To my knowledge, the benchmark is novel because it focuses on multi-dataset probabilistic regression
- NIAQUE design decisions are explained and also include some theoretical analysis
- NIAQUE is more interpretable while having competitive performance when compared with standard approaches
- Experimental results sweep over parameters and include confidence intervals
This paper introduces NIAQUE, a new model for probabilistic regression that learns to approximate the inverse of the posterior distribution. It also introduces a new benchmark LPRM-101 with 101 datasets for multi-dataset probabilistic regression. The authors provide theoretical analysis for NIAQUE and also show that it is competitive with (or better than) prior baselines (trees, transformers) on the new benchmark. Additionally, the authors argue that NIAQUE improves over existing baselines by being more interpretable.
- While the paper introduces a new benchmark, it lacks detail or tooling that would allow others to use it. I would like to better understand how easy it would be for another researcher to use it and build off this work.
- This is a multi-dataset benchmark, but I would like to understand how the best existing single dataset approaches perform, or if the best approaches are the ones in Table 1. For example, are the top submissions to the Kaggle datasets' respective competitions significantly different from Transformers or XGBoost? How do those numbers compare with the "local" approaches in Table 1?
Yes
No ethics review needed.
My main concern is minimal gains from multi-task transfer. 
* One way to address this: is it possible to evaluate this model in the standard meta-learning setup, with entirely held-out datasets, as in [1, 2]? If these results are strong, it might better justify NIAQUE even when in the "held-in" dataset setting, NIAQUE-local and NIAQUE-global perform similarly. In the meta-learning setting, Neural Processes are the relevant baseline. If NIAQUE is competitive with them, it would encourage me to raise my score.

Other medium-priority concerns:
* In Table 1, why is Transformer-local not included? How does it perform?
* The architecture appears to resemble Neural Processes [1, 2], another neural architecture for multi-task learning / meta-learning. Could you contextualize NIAQUE with regard to them?
* The nomenclature of "co-training" seems to clash with the old-school pseudo-label style setting as in [3]. Could you instead use the established nomenclature for this setup (multi-task learning, pretraining, or meta-learning depending on the context?).

[1] Garnelo et al., 2018. Neural Processes. https://arxiv.org/abs/1807.01622

[2] Garnelo et al., 2018. Conditional Neural Processes. In ICML. https://proceedings.mlr.press/v80/garnelo18a.html

[3] Blum and Michell, 1998. Combining Labeled and Unlabeled Data with Co-Training. In COLT. https://www.cs.cmu.edu/~avrim/Papers/cotrain.pdf
* Good motivation: The problem setting of multi-task learning across tabular regression datasets is a promising and important one, so the paper is well-motivated.
* NIAQUE compares reasonably well to strongest baseline: The NIAQUE architecture appears to have decent empirical results, performing slightly better than the Transformer with ~2x reduction in wall-clock training time (L254).
* Strong benchmark and set of evaluation datasets: The contributed LPRM-101 benchmark is expansive.
* The method (the neural network architecture and quantile regression objective) seem sensible and novel.
* The consistency result for NIAQUE appears to be correct.
The authors focus on the problem of multi-task probabilistic regression over tabular datasets. They contribute a benchmark of 101 regression datasets drawn from public sources, and develop a neural network architecture trained with quantile regression for this setting. They demonstrate that their architecture, NIAQUE, outperforms tree-based and Transformer baselines in both point and distributional metrics. The authors also provide a theoretical result demonstrating the consistency of their any-quantile training algorithm.
* Minimal gains from multi-task transfer: the core pitch of the paper is to contribute an architecture that works well for multi-task learning, but the gains from learning across datasets appear to be pretty marginal (comparing NIAQUE-local and NIAQUE-global in Table 1).
* Unclear presentation: Some parts of the presentation of NIAQUE were confusing or unclear. How exactly is the model used for prediction if the encoder takes a variable number of inputs? At test time, do you concatenate each query point with the training dataset (or a subsample of training points) and feed them to the encoder to obtain the encoder representation? If so, have you considered various heuristics for selecting these points (e.g., nearest neighbors to the query input)?
Yes
No ethics review needed.
* Q1) I am wondering how NIAQUE's performance varies with different quantile choices.
* The authors provided the LPRM-101 benchmark which allows for rigorous evaluation across diverse regression tasks and data sources.

* The proposed NIAQUE seems a new probabilistic regression that learns to approximate the inverse of the posterior distribution during training.

*
The current manuscript introduces NIAQUE, a neural network model for probabilistic regression that performs any-quantile estimation across multiple datasets.  NIAQUE takes a training batch of the variable number of observations and generates a quantile of uniformly at random in the [0,1] range for each training sample. The quantile is used as the input of the quantile decoder to modulate
the observation representation and as the supervision signal in the quantile loss.

Besides this, the paper also introduces the LPRM-101 Benchmark which is a new benchmark with 101 diverse regression datasets from different sources like UCI, PMLB, OpenML, and Kaggle, aimed at assessing the performance of models on probabilistic regression tasks. The proposed method enables to handle a range of quantiles in a single framework. Its encoder-decoder design effectively manages variable data formats across datasets, making it flexible for multi-dataset.
* W1: co-training across datasets could introduce biases that are absent in single-dataset models. This is especially relevant if certain datasets dominate the training process, potentially leading to skewed representations and reduced performance on minority datasets. It would be interesting to see the performance of your model in individual datasets when trained jointly vs. separately.  

* W2: While the any-quantile framework is flexible, the performance of the model may vary depending on the choice of quantiles.The lack of specific guidance on the choice of quantiles may prevent users from achieving optimal results in different use cases.

* W3:NIAQUE relies on specific probabilistic assumptions for feature importance and distributional modeling, which may not be suitable for all regression tasks, particularly those with non-standard distributional characteristics. This could limit the model’s applicability to a narrower range of tasks than intended.
Yes
No ethics review needed.
Please refer to the weaknesses section above.
1. This paper aims to narrow the gap between current neural networks and probability regression models, which is still under explored.
2. It proposes a complete working pipeline from building benchmark, proposing new model with theoritical analysis, and empirical analysis.
3. Overall, the draft is easy to read and follow.
This paper aims to narrow the gap between deep probabilistic regression models with current powerful deep neural networks. It proposes a large unified model to handle regression tasks across different datasets. Correspondingly, it introduces a framework called NIAQUE, to address probabilistic regression problem. Compared with several baseline methods, the proposed model achieves better empirical results.
1. The problem statement is still not clear, why use transformer model to handle such tasks, what kind of real-world applications to illustrate the model practices? more discussions and intuitions are needed to support the motivation of this work.
2. Some table and figure formats are not well prepared such as figure 3. Necessary descriptions are needed.
3. Based on the table 1, there is not clear difference of final results compared with previous works, what is the benefit of proposing such a learning pipeline? how to illustrate the proposed model superiority?
Deep probabilistic regression model, which we call NIAQUE, learns a meaningful cross-dataset representation and scores favourably against strong tree-based baselines and Transformer
@inproceedings{
anonymous2024niaque,
title={{NIAQUE}: Neural Interpretable Any-Quantile Estimation - Towards Large Probabilistic Regression Models},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=5x65bI0aY8},
note={under review}
}
State-of-the-art models in computer vision and natural language processing largely owe their success to the ability to represent massive prior knowledge contained in multiple datasets by learning over multiple tasks. However, large-scale cross-dataset studies of deep probabilistic regression models are missing, presenting a significant research gap in this area. To bridge this gap, in this paper we propose, analyze, and evaluate a novel probabilistic regression model, capable of solving multiple regression tasks represented by different datasets. To demonstrate the feasibility of such operation and the efficacy of our model, we define a novel multi-dataset probabilistic regression benchmark LPRM-101. Our results on this benchmark imply that the proposed model is capable of solving a probabilistic regression problem jointly over multiple datasets. The model, which we call NIAQUE, learns a meaningful cross-dataset representation and scores favourably against strong tree-based baselines and Transformer.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
deep probabilistic regression large regression models
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/c19476d2e109276372ba979dfce4c354c1d3c25f.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
NIAQUE: Neural Interpretable Any-Quantile Estimation - Towards Large Probabilistic Regression Models
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission