Diffusion-Nested Auto-Regressive Synthesis of Heterogeneous Tabular Data
Tabular data synthesis;autoregressive models;diffusion models
Yes
No ethics review needed.
- For continuous columns, why use a diffusion loss?  You could also use z to parameterize any simple continuous distribution, e.g. a Gaussian.  Given that the diffusion process can be computationally expensive, do you think that it is a necessary component of the method?  I understand that you do an ablation in which you compare against discretizing the continuous variable, but this also seems like an unnatural approach compared to simply passing it through a continuous distribution. 
- In your embedding/masking, how do you distinguish between a continuous value being equal to 0 and you masking out that value?  To me, it seems that both would lead to the same result after embedding the data.  Is there a particular reason you chose to use zero for missing values instead of an embedding for a [mask] token, as typical in masked language models? 
- Details on the diffusion aspect are generally missing.  Is the denoising network the same for different continuous columns or is the denoising network column-specific?  How many steps of diffusion are generally used?  
- Table 2 — since some of these values are quite close, could you provide some standard deviations?  Also, if there are ties with your method in the table, you should probably also bold these for consistency and fairness to other approaches.  
- Figure 6 — It would be interesting to see standard deviations for these bars.  I wonder how sensitive the results are to different initializations / different MCAR masks.  The authors mention that they are “significantly outperforming” other methods, but I think standard deviations are needed to justify this somewhat bold claim.  It would also be interesting to see how the percent missing from the data affects the results (currently it is fixed at 30%)? 
- For conditional sampling (Algorithm 5), is there a reason why you need to generate step-by-step?  During training, it seems the model learns to predict all masked columns from unmasked ones in one step during training.  Do you get better performance by generating step-by-step at test time?
- This paper tackles an interesting and important problem -- how to learn the generative distribution of tabular data.  Tabular data is indeed challenging because of the potentially unstructured and diverse nature of the columns at hand.  As the authors mention, there has been a lot of recent interest in developing deep generative models for this domain.
- The authors' proposed solution is an interesting one.  They provide comparisons in the experiments against several other methods in the literature and generally show superior performance on the statistical fidelity metrics (Table 1).  
- I appreciate the extensive set of experiments, such as machine learning efficiency, the various visualizations, and the ablation studies. These help give a holistic view of the relative performance of their method compared to the literature.  
- The code is available and generally looks well-organized and a useful resource for the community.
The authors develop a generative modeling method for tabular data.  Their method learns a conditional generative model that takes masked input and predicts the missing columns.  At inference time, they use this model autoregressively to generate data one column at a time -- for both unconditional data generation and missing values imputation. The model comprises of a linear embedder, a transformer encoder, and either a diffusion decoder or categorical classifier -- depending on whether the particular column is continuous or discrete in nature.  The authors perform experiments on various tabular datasets and use various metrics to compare the samples generated by their method against those generated by other methods in the literature.  They also perform ablation studies to understand the relative importance of various components of their approach.
- Nitpick: The authors sometimes use overly promotional language, e.g. line 71 — “through two ingenious design features”, line 84 —  “TABDAR offers several unparalleled advantages” — I recommend against the use of overly promotional words such as “ingenious”, “unparalleled advantages” in the paper.
 - Nitpick: Eq. (5) — Define $x^{<i}$.  For a reader familiar with autoregressive modeling notation, it is obvious what this means, but for others it may not be.  E.g. it would be good to clarify what this means for $i = 1$.
- One point that was confusing to me at first was that the model is not trained as an autoregressive model -- it is trained as a masked language model.  If I understand Eq. (8) / Algorithm 3 correctly, you are only masking out a particular set of columns at each training step. 
 Thus, you are not really learning an autoregressive distribution, this is more like a conditional distribution like masked language modeling in which you predict p(masked | unmasked).  I understand that you are using the learned conditional distribution in an autoregressive way at test time / generation time -- it would be good to clarify this point in the text.  
- There are also some questions about the method I have, listed below.  I am happy to give a positive score for this paper, but it is contingent on the questions below being addressed.
Yes
No ethics review needed.
See previous section.
- Good results, and extensive experimental results.
The authors propose a Diffusion-nested Autoregressive model (TABDAR) that can handle both discrete and continuous tabular data. Continuous features are modeled by nested diffusion models, and based on the assumption that table columns are permutation invariant, bi-directional attention is utilized to support generation of the columns in any order. TABDAR is compared to several recent methods for hetrogeneous table generation, and is shown to perform comparably or better in terms of statistical fidelity (Table 1), generation efficiency for machine learning tasks (Table 2), and Distance to Closest Record (DCR) scores (Table 3, Figure 5). Ablations suggest that both the use of diffusion losses and random sampling of order improve performance (Table 4), and that the results are quite stable wrt depth and embedding dimension (Table 5 and 6). Imputation results are also competitive (Figure 6).
- The TabSyn methods results stated in the paper are different (worse) that those reported in the TabSyn paper. These differences need to be fully explained and accounted for before the paper can be considered for publication.
- In addition, since TabSyn and TABDAR perform similarly and address the same problem, the advantages and differences between the two approaches should be discussed. 
- As a method that simply integrates diffusion modeling into bidirectional transformers to handle continuous tabular entries, the novelty of the paper seems on the lower side.
- The use of bidirectional attention is based on the assumption that columns are permutation invariant, while in practice, one expects that columns will sometimes have dependencies, and these would always be based on reading order (e.g. for English, left to right). Also, bi-directional attention is generally not equivalent to arbitrarily ordered causal attention, as suggested in Figure 2, due to the use of position embeddings. Based on these observations, It's not clear that bi-directional attention necessarily the best approach.
Yes
No ethics review needed.
See above.
The authors present state-of-the-art empirical results (by some margin). The proposed method is based on interesting observations and to the best of my knowledge, the way they combine multi-modal signals is novel.
This work proposes to combine autoregressive sequence modeling for discrete (token) generation and diffusion for continuous signal.
- The authors claim that columns are permutation invariant, which I tend to agree with. However, combining this observation with autoregressive generation is impossible. From the manuscript, I couldn't figure out if the method combines lower triangular mask and permutations of the tokens. Can the authors clarify?
- Superscript and subscripts - it seems like the authors used mixed notations in different sections (I think?). E.g., in equation (1), items in a sequence are subscript and then $<i$ is in superscript; later, the authors use the subscript for diffusion time and superscript for items (section 3.1, equation 2, equation 3, equation 5, and more). It feels like a mess. Please let me know if I didn't understand correctly.
- The authors claim that it is straightforward to do conditional sampling using their method, but I find it hard to understand how it is actually done. I.e., condition on continuous representation in discrete phase and vice versa.
- Learning a permutation invariant representation is well known in transformers, and to the best of my knowledge, the classical way to do it is by removing the positional embeddings. Can the authors explain why they chose a different path (which seems less effective)?
- Equation 6 - is it correct?
- Results - I am not familiar with the benchmarks used, but I am not sure what I can learn about the proposed method compared to prior works when the improvements are so small (as it seems that the tasks are roughly solved).
Yes
I don't see ethical concerns will arise from this paper.
No ethics review needed.
A.	I think the data generation time will take long because of the autoregressive nature of the model. I think it would be great if authors can compare the generation times with other baseline methods.

B.	In line 85, the authors claim they use two types of generative models: one is diffusion, and another one is categorical prediction. But categorical prediction is not a generative model. 

C.	Authors should mention they use one-hot-encoding for preprocessing categorical data. I noticed they mentioned it in Appendix, but they should be mentioned in main paper to avoid confusion.

D.	In Figure 3, the role of diffusion model is not emphasized. 

E.	Are multiple diffusion models used for each of the continuous variable? If it is true, for each 

F.	In Tables 5 & 6, are the values in the tables averaged results over the two datasets? 

G.	Is it safe to say the model is robust to the hyperparameters? The results are based only on two datasets. I think authors need to work on more datasets for validating this claim. 

H.	Are the P(P(x|z))s in Eq. (6) and (7) typos?
Strong experimental results.
This paper suggests a new tabular data synthesizer by combining bi-directional transformer and diffusion model.
Please see the questions below.
This paper nests a diffusion model into an autoregressive modeling framework, such that it can generate heterogeneous tabular dat aof mixed numerical and categorical features.
@inproceedings{
anonymous2024diffusionnested,
title={Diffusion-Nested Auto-Regressive Synthesis of Heterogeneous Tabular Data},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=kkGIbmpCHU},
note={under review}
}
Autoregressive models are predominant in natural language generation, while their application in tabular data remains underexplored. We posit that this can be attributed to two factors: 1) tabular data contains heterogeneous data type, while the autoregressive model is primarily designed to model discrete-valued data; 2) tabular data is column permutation-invariant, requiring a generation model to generate columns in arbitrary order. This paper proposes a Diffusion-nested Autoregressive model (TabDAR) to address these issues. To enable autoregressive methods for continuous columns, TabDAR employs a diffusion model to parameterize the conditional distribution of continuous features. To ensure arbitrary generation order, TabDAR resorts to masked transformers with bi-directional attention, which simulate various permutations of column order,  hence enabling it to learn the conditional distribution of a target column given an arbitrary combination of other columns. These designs enable TabDAR to not only freely handle heterogeneous tabular data but also support convenient and flexible unconditional/conditional sampling. We conduct extensive experiments on ten datasets with distinct properties, and the proposed TabDAR outperforms previous state-of-the-art methods by 18\% to 45\% on eight metrics across three distinct aspects.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Tabular data synthesis autoregressive models diffusion models
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/c052f18106f32baea942738a91948476b2ce3db7.pdf
generative models
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Diffusion-Nested Auto-Regressive Synthesis of Heterogeneous Tabular Data
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission