Regularized DeepIV with Model Selection
Nonparametric estimator;instrumental variables;model selection;causal inference.
Yes
No ethics review needed.
* is it really fair to say that your algorithm is more computationally tractable when it is based on MLE?
* With the (unfortunate) exception of the introduction, I found the paper mostly well-written and clear.

* The paper studies an interesting problem, proposes a natural solution, and proceeds to analyze said solution. While I am not familiar with the immediately preceding related work (in IV), this seems clean to me.
This paper studies a two stage procedure for regression in the scenario where the errors are not conditionally independent. They first learn a conditional density to make use of instrumental variables and consequently solve a square loss erm problem weighted by the learned conditional density. They show that this procedure attains mostly standard nonparametric rates.
* The organization of the paper is hard to follow and the introduction is way too terse. As someone well-versed in nonparametric statistics but not necessarily IV methods, I had to skip ahead to section 4 to really understand what was going on.  Stating that you are trying to solve some fixed point equation in the introduction is not conducive to most people's understanding of the problem you are solving. 

* My overall feeling is that the result is somewhat incremental. To my understanding the main difficulty lies making standard guarantees for MLE in Hellinger^2 compatible with the square loss. I could not entirely follow why this is so challenging and would encourage the authors to further explain why this is the case (for instance, in the very last paragraph of section 1, you mention this difficulty but do not really expand on it, nor do you reference the lemmata which might be useful for understanding this difficulty).
Yes
No ethics review needed.
please see above
RDIV offers several key advantages over existing methods. It addresses three significant limitations of prior literature: it eliminates the need for unique IV regression identification, avoids reliance on the often unstable minimax computation oracle, and supports model selection procedures.
This paper addresses the problem of nonparametric instrumental variable (IV) regression, a framework with wide applications across fields such as causal inference, handling missing data, and reinforcement learning. The objective in IV regression is to solve the conditional moment equation, 𝐸 [ 𝑌 − ℎ ( 𝑋 ) ∣ 𝑍 ] = 0, where 𝑍  serves as the instrument. The authors introduce RDIV, a regularized variant of the DeepIV method, marking the first work to provide rigorous theoretical guarantees for DeepIV. RDIV enhances generalization by incorporating Tikhonov regularization. Methodologically, RDIV follows a two-stage approach. The first stage involves learning the conditional distribution of covariates, while the second stage refines the estimator by minimizing a Tikhonov-regularized loss function.
It is unclear how the method compares for example to recently developed methods (see arxiv:2405.19463; to appear at NeurIPS 2024) that completely avoids minimax formulations, as well as avoiding the need for two-stage procedures.
Yes
No ethics review needed.
How is Tikhonov regularization related to a function space parametrized by the neural network? It seems not straightforward to relate it to weight decay.

Is there a computational gain when minimax optimization is no longer needed?
The paper is well-written, and the results are motivated well. I didn't go through the proofs, but the explanations after each result are insightful, and ease the reading.  The theoretical contribution is solid.
The paper studies the nonparametric instrumental variable regression with Tikhonov regularization (RDIV), and proves that RDIV allows model selection procedures and matches the SOTA convergence rate. I agree with the author's claim that this work is the first attempt to provide rigorous guarantees for DeepIV. With Tikhonov regularization, the model selection procedure achieves the oracle rate and iterative RDIV matches the SOTA rate.
The numerical experiments are only based on simulated data. It would be better to have some results from real data to demonstrate the strength of the proposal.
Yes
No ethics review needed.
* To convert the MLE guarantee into an $L_2$ guarantee, the authors assumed a minimum density on the conditional density. What’s the benefits/drawbacks compared with the conditional mean embedding based methods (although it also requires some assumptions like HS operators).
* The authors discussed lots of the aspects for non-parametric clearly.
This manuscript is basically a technical paper, discussed about two-stage non-parametric IV and model-selection in the second stage when equipped with an additional $L_2$ regularization.
* I feel the problem studied in this paper is with limited novelty. The transformation between one-stage and two-stage algorithms and analysis is in general only a technical problem and have been discussed in different places like DualIV, and the $L_2$ regularization itself makes the model-selection easier (with the strongly convexity).
@inproceedings{
anonymous2024regularized,
title={Regularized Deep{IV} with Model Selection},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=0gqCIaBRQ9},
note={under review}
}
In this paper, we study nonparametric estimation of instrumental variable (IV) regressions. While recent advancements in machine learning have introduced flexible methods for IV estimation, they often encounter one or more of the following limitations: (1) restricting the IV regression to be uniquely identified; (2) requiring minimax computation oracle, which is highly unstable in practice; (3) absence of model selection procedure. In this paper, we analyze a Tikhonov-regularized variant of the seminal DeepIV method, called Regularized DeepIV (RDIV) regression, that can converge to the least-norm IV solution, and overcome all three limitations. RDIV consists of two stages: first, we learn the conditional distribution of covariates, and by utilizing the learned distribution,  we learn the estimator by minimizing a Tikhonov-regularized loss function. We further show that RDIV allows model selection procedures that can achieve the oracle rates in the misspecified regime. When extended to an iterative estimator, we prove that RDIV matches the current state-of-the-art convergence rate. Furthermore, we conducted numerical experiments to justify the efficiency of RDIV empirically.  Our results provide the first rigorous guarantees for the empirically well-established DeepIV method,  showcasing the importance of regularization which was absent from the original work.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Nonparametric estimator instrumental variables model selection causal inference.
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/36421fa6453920abc8f29f0ef6d69cd1560c852a.pdf
learning theory
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Regularized DeepIV with Model Selection
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission