Non-negative Tensor Mixture Learning for Discrete Density Estimation
Tensor factorization;EM algorithm;Low-rank approximation
Yes
No ethics review needed.
Is there a specific general mathematical expression for the many-body approximation?
1. A unified methodology has been developed to deal with a variety of low-rank structures, including CP, Tucker, Train decompositions, and  their combinations, forming mixtures of low-rank tensors. 
2. A mixture of low-rank tensor modeling procedure is developed  to empirically demonstrates inferential robustness and improved generalization.
3. Both theoretical analysis and numerical comparisons are provided to show the merits of the proposed methodology.
This work presents an expectation-maximization (EM) based unified framework for nonnegative tensor decomposition that optimizes the Kullback-Leibler divergence, and further establishes a general relationship between low-rank decomposition and many-body approximation. The proposed framework offers a unified methodology for a variety of low-rank structures, including CP,
Tucker, and Train decompositions, and their combinations. A series of experiments are carried out to illustrate the merits of the developed methodology.
1. The convergence analysis seems overly coarse and somewhat redundant, as it only demonstrates that negative cross-entropy increases with iterations—an expected result given the maximization objective. More importantly, the analysis should address how negative cross-entropy could converge to the true value.
2. Please note that the computational cost of the proposed algorithm increases significantly with the size of the tensor. Kindly provide the specific order of computational complexity and compare it with the complexity orders of related algorithms.
3. The compared baselines are not SOTA. Please consider some recent methods for numerical comparisons. 
4. Please provide a detailed description of the convergence conditions in Algorithm 1.
Yes
No ethics review needed.
In my experience, adding a noise (or constant tensor) term to a non-negative tensor decomposition can significantly improve model fit, and the empirical results shown in Table 2 mostly demonstrate this phenomena. How much of the empirical improvements in negative log-likelihood are from the adaptive noise term? I view the ability to seamlessly learn the noise term from the data as an advantage of this method, although I am concerned it is the only substantial advantage of this method in practice. 
When would it be useful to include a Tucker component in practice? EMTucker is computationally expensive, scaling as R^D. While EMTuckerN beats existing methods in Table 5, it generally performs worse than EMCPTrainN.
The contribution of closed-form maximization updates for the specific EM algorithm is a strength. The computational efficiency, in some instances of the method (such as EMCPTrain), are useful. The unifying framework across low-rank tensor decompositions is parsimonious, and leveraging the parsimony of mixture models is a neat idea. I particularly like how the authors show that one may learn the weights of their mixture model, removing the need to choose between low-rank structures in advance of training.
This paper aims to parsimoniously unify a variety of low-rank tensor decomposition structures and address discrete density estimation using a novel mixture of decompositions method. The method permits mixtures of CP, Tucker, and tensor-train decompositions to model non-negative data. The authors derive a computationally efficient expectation-maximization (EM) algorithm for performing inference in their model. The inference algorithm allows for learning of the mixture weights of the components, which are tensor decompositions. In addition to establishing a unifying framework for low-rank tensor decompositions, the authors demonstrate their method’s effectiveness in estimating discrete mass functions by comparing a specific instance of their method, EMCPTrain, to a large body of existing ones. Their method achieves marginally better negative-log-likelihood per sample than existing methods.
The approach, while unifying in theory and a neat idea, yields very incremental empirical results at best. The empirical gains are modest. When more baselines are taken into account, as in Table 6, the gains are further reduced. The comparisons and evaluations are not well-organized, making the aggregate contribution difficult to evaluate across the many tables in Section 5 and Appendix C. There are some section of the paper that came across as unclear that first time I read it. In particular, the paper refers to “many-body approximation” many times in the first two sections of the paper without a clear definition. A formal definition in Section 1 would help clarify how the paper aims to leverage the many-body approximation representation.
Yes
No ethics review needed.
Questions/Comments:

1.	Writing and Organization: It is important to introduce the low-rank tensor structures when discussing it in the introduction. 

2.	It is commented that “it always converges regardless of the choice of the low-rank structure assumed in the model.” EM convergence is hard to establish unless it is well initialized. Hence, it is not easy to claim convergence for the proposed method

3.	While there is a technique introduced to reduce the computational complexity of tensor train decomposition as presented in Section 4.2, computational complexity of Tucker remains the same which becomes dominant especially the case of K>1 mixture cases. The convergence speed curves should be presented with time in x axis to understand where it stands with respect to the baselines. Due to the computational complexity, it is also challenging to utilize the mixture model in practice. Once may need to choose the component of the mixtures (the type of tensor decomposition) , rather than allowing the model to learn by itself. A detailed discussion on the limitation of the approach (in the main section) would help here. 

4.	It is unclear how does the adaptive noise term guarantees the ``convexity” and convergence as claimed in Section 4.4. What do you mean by convexity here? How do you learn the noise parameter here? Do you learn different noise parameter for different low-rank tensor models?

5.	Experiments are limited to showing negative log likelihood. While it shows the dynamics of the algorithm, it does not show how does the approach learn the ground-truth. Simulation studies would help understand how well the method learns the true discrete density. In real data experiments, missing value prediction should be a better approach to understand the applicability of the approach.

6.	Minor typos: “distribution underlying the data” in Page 5, notation $\eta^k$ is confusing with $\eta$ raised to kth power.
Strengths:

1.	The proposed tensor decomposition method is a general framework that can handle different types of tensor decomposition models like CP, Tucker and tensor train, innovatively utilizing the many body approximation technique from the (Ghalamkari et al.,2023). The closed-form updates of the parameters in the M-step of the EM algorithm is also attractive as it avoid tuning of any hyperparameters, which is often practically inconvenient
The work deals with a nonnegative tensor decomposition algorithm for discrete density estimation. The framework is based on expectation maximization algorithm from KL divergence minimization perspective. Unlike the existing KL divergence-based nonnegative tensor decomposition algorithms for this problem that used gradient-based iterative steps for parameter updates, the proposed approach leverages the insights from the many-body approximation problem. Consequently, the updates of the parameters in the M-step boils down to closed form expressions, reminiscent of the updates derived in (Huang & Sidiropoulos,2017; Yeredor & Haardt, 2019) .
Weakness:

1.	The paper organization has lot of room for improvement. From the start of the paper, a clear description of problem is missing. Section 4 introduces the problem statement, while section 3 starts describing the approach (many-body approximation). Then, it is confusing to connect and understand that some of the terms have been obtained from the previous step of the algorithm.

2.	The problem formulation and the application of the solution itself is debatable. The tensor ${\cal T}$ is an empirical distribution tensor which is of the dimension of the categorical features. The entire solution of discrete density learning depends on the accuracy of the empirical tensor, which is hard to make sure due to curse of dimensionality. 

3.	The problem is hardly scalable as it deals with the many-body approximation tensor ${\cal M}$ whose dimension is much larger than that of even the empirical tensor ${\cal T}$.

4.	The experiments and major baselines are lacking. In discrete density estimation, there are works using second and third order marginals, that can mitigate the curse of dimensionality problem to an extent.

a.	Kargas, Nikos & Sidiropoulos, N.D. & Fu, Xiao. (2017). Tensors, Learning, and “Kolmogorov Extension” for Finite-Alphabet Random Vectors. IEEE Transactions on Signal Processing. 66. 10.1109/TSP.2018.2862383.

b.	S. Ibrahim and X. Fu, "Recovering Joint Probability of Discrete Random Variables From Pairwise Marginals," in IEEE Transactions on Signal Processing, vol. 69, pp. 4116-4131, 2021, doi: 10.1109/TSP.2021.3090960

Discussions and comparisons with these baselines would help readers understand the strength of the approach (if any).
EM-based unified framework for non-negative tensor decomposition that can accommodate a mixture of low-rank tensors and adaptive noise terms
@inproceedings{
anonymous2024nonnegative,
title={Non-negative Tensor Mixture Learning for Discrete Density Estimation},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=mbo4YnWCHd},
note={under review}
}
We present an expectation-maximization (EM) based unified framework for non-negative tensor decomposition that optimizes the Kullback-Leibler divergence. To avoid iterations in each M-step and learning rate tuning, we establish a general relationship between low-rank decomposition and many-body approximation. Using this connection, we exploit that the closed-form solution of the many-body approximation can be used to update all parameters simultaneously in the M-step. Our framework offers not only a unified methodology for a variety of low-rank structures, including CP, Tucker, and Train decompositions but also their combinations forming mixtures of tensors. The weights of each low-rank tensor in the mixture can be learned from the data, which eliminates the need to carefully choose a single low-rank structure in advance. We empirically demonstrate that our framework provides superior generalization for discrete density estimation compared to conventional tensor-based approaches.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Tensor factorization EM algorithm Low-rank approximation
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/f1b53946885130db7f0a0a21065d02fce74687cf.pdf
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/530c358327ce089d9da7b70d7e43c9c99e269bc4.zip
Non-negative Tensor Mixture Learning for Discrete Density Estimation
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission