Scalable Multi-phase Word Embedding Using Conjunctive Propositional Clauses
Tsetlin Machine;Word Embeddings;Natural Language Processing (NLP)
Yes
No ethics review needed.
In addition to addressing the items raised in the Weaknesses section, the authors could consider the following suggestions to further enhance the paper:
1. Include a more detailed comparison between TM-AE and two-phase TM-AE embeddings to clarify the specific benefits and limitations of each approach.
2. Evaluate the quality of two-phase TM-AE embeddings on additional word embedding tasks such as analogy, categorization, and outlier detection.
3. Provide a few examples of two-phase TM-AE embeddings to illustrate their characteristics more concretely.
4. Demonstrate the potential of two-phase TM-AE embeddings in more complex NLP tasks by incorporating them into Transformers or other state-of-the-art NLP or large language models, showcasing applications beyond classification tasks.
5. Include a discussion of possible future research directions to give readers insight into how this work could be further developed.
6. Provide the code for two-phase TM-AE training and evaluation to facilitate reproducibility and further research.
This paper has several notable strengths:
1. It introduces a relatively novel approach by employing the Tsetlin Machine automaton to generate word embeddings, distinguishing it from more conventional methods.
2. The paper provides a thorough explanation of the CoTM architecture, TM automaton, TM-AE training process, and the unique two-phase TM-AE training schema.
3. The authors highlight a computational efficiency feature: by skipping the first phase of TM-AE when updating a single word, one can reduce the computational load.
4. Multiple human-annotated datasets, including RG65, MTURK287, MTURK717, and WS-353, are used to rigorously evaluate the quality of the generated word embeddings.
5. The proposed two-phase TM-AE embeddings are comprehensively compared to established models such as GloVe, Word2Vec, FastText, BERT, and ELMo, offering insight into its relative strengths and weaknesses.
This paper presents a novel two-phase method for generating word embeddings using the Tsetlin Machine (TM) and Coalesced Tsetlin Machine (CoTM) structures. The authors describe how the CoTM automaton process and the TM autoencoder (TM-AE) operate, introducing a two-phase training mechanism for the TM-AE. The proposed approach's effectiveness is evaluated on semantic similarity and sentiment classification tasks, comparing the performance of the two-phase TM-AE embeddings with other word embedding frameworks, including GloVe, Word2Vec, FastText, BERT, and ELMo. Results indicate that the two-phase TM-AE embeddings outperform GloVe, Word2Vec, and FastText on specific tasks and metrics. Additionally, the authors highlight that this approach represents the first scalable, transparent, end-to-end framework for generating word embeddings based on propositional logic through a TM-based method.
The paper has several areas where it could be improved:
1. Although the model reportedly required 6 months of training on a DGX H100 machine, the paper lacks an analysis of the computational time and space complexity of the proposed method, particularly in comparison to other approaches like GloVe, Word2Vec, and FastText.
2. While the paper emphasizes the scalability and transparency of the two-phase TM-AE approach, further explanation and analysis of these properties would strengthen the argument for its advantages. For example, present empirical scalability results with increasing vocabulary sizes, or provide examples of how the logical structure of embeddings can be interpreted.
3. The quality of the two-phase TM-AE embeddings could be better demonstrated by including additional evaluation tasks beyond semantic similarity, such as analogy, categorization, and outlier detection.
4. The hyper-parameters used in the sentiment classification MLP model and TM classifier are not documented, and there is no discussion of the hyper-parameter fine-tuning process for the TM classifier.
5. In Section 4.1, the comparison of TM-AE and two-phase TM-AE on similarity tasks shows that TM-AE performs better on two out of three metrics. This result raises questions about the benefit of the two-phase approach for this task. To address this, you may provide a more in-depth analysis of why the two-phase approach underperforms on these metrics, or what potential trade-offs or benefits it might offer despite these results.
6. In Section 4.2, the paper omits the performance of the TM-AE model, which would provide valuable context for assessing the gains offered by the two-phase TM-AE.
Yes
No ethics review needed.
I have a question regarding the Tsetlin machine-based word embedding approach. This method does not seem to account for word order, relying more on co-occurrence features to represent semantics. As a result, it may be less effective at capturing dependency, structure, and other linguistic characteristics. This could be why it performs reasonably well only on tasks like sentiment classification that do not require deep structural semantics. What is the authors' view on this issue?
A novel tsetlin machine based incorporating two-phase training to discover contextual embeddings of input sequences.
This paper presents a scalable two-phase word embedding method using the Tsetlin Machine (TM) framework for NLP tasks. Phase 1 generates interpretable word embeddings by learning contextual knowledge for each target word, capturing it in logical clauses. In Phase 2, these embeddings are used to create representations for word sequences, supporting applications like sentiment analysis. Evaluations show competitive performance, particularly in sentiment analysis and similarity assessments on human-annotated datasets. The approach provides a transparent, reusable alternative to traditional embeddings.
- The main motivation is not clearly expressed; the introduction does not outline the differences and advantages over previous methods, only presenting an overview of prior work. The CoTM method is also not original, thus the interpretability mentioned at the beginning is also not a contribution of this paper.

- The introduction mentions the issue of long training time, but the experiments section does not seem to analyze the efficiency issue.

- The experiments are conducted only on a sentiment analysis dataset for a text classification task, which is of limited persuasiveness.
Yes
No ethics review needed.
See the Weaknesses
- The two-phase approach to word embedding with TM is novel, especially using conjunctive propositional clauses, which adds interpretability.
- The use of Tsetlin Machines enables a more transparent and interpretable embedding process compared to deep learning models.
- Practical Application in Sentiment Analysis: The embeddings show utility in real-world tasks, as demonstrated through sentiment analysis with data augmentation.
The paper introduces a scalable two-phase word embedding approach using TM and conjunctive propositional clauses. Phase 1 captures knowledge for individual words, while Phase 2 constructs embeddings for word sequences. Experiments show that the model achieves competitive cosine similarity on benchmark datasets but lags in Spearman correlation. Additionally, the model performs moderately well in sentiment analysis compared to BERT and ELMo. This approach demonstrates TM’s potential for interpretable NLP applications, though further refinement is needed.
- The results don’t look good; the model performs poorly on Spearman and Kendall correlations, which suggests it struggles to capture the ranking of word pairs.
- I guess Algorithm 1 doesn’t seem to be a direct contribution of this paper, so it might be better left out of the main text or put into Appendix.
- The paper could use more polish. For example, Tables 1 and 2 are missing top and bottom lines, which affects readability.
- The evaluation is pretty narrow, focusing mostly on basic similarity metrics and sentiment analysis, without exploring a broader range of NLP tasks that would really show the model’s robustness.
- For Table 2, why the socres for all models looks quite similiar? Also, I didn't see the TM outperform the other models. It even lag behind a lot for some embeddings. It may happens very randomly but it becomes hard to convince the advantages of TM.
@inproceedings{
anonymous2024scalable,
title={Scalable Multi-phase Word Embedding Using Conjunctive Propositional Clauses},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=ZOrqCRDKh6},
note={under review}
}
The Tsetlin Machine (TM) architecture has recently demonstrated effectiveness in Machine Learning (ML), particularly within Natural Language Processing (NLP). It has been utilized to construct word embedding using conjunctive propositional clauses, thereby significantly enhancing our understanding and interpretation of machine-derived decisions. The previous approach performed the word embedding over a sequence of input words to consolidate the information into a cohesive and unified representation. However, that approach encounters scalability challenges as the input size increases. In this study, we introduce a novel approach incorporating two-phase training to discover contextual embeddings of input sequences. Specifically, this method encapsulates the knowledge for each input word within the dataset’s vocabulary, subsequently constructing embeddings for a sequence of input words utilizing the extracted knowledge. This technique not only facilitates the design of a scalable model but also preserves interpretability. Our experimental findings revealed that the proposed method yields competitive performance compared to the previous approaches, demonstrating promising results in contrast to human-generated benchmarks. Furthermore, we applied the proposed approach to sentiment analysis on the IMDB dataset, where the TM embedding and the TM classifier, along with other interpretable classifiers, offered a transparent end-to-end solution with competitive performance.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Tsetlin Machine Word Embeddings Natural Language Processing (NLP)
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/42d0cc7198efbb27370fb2fb7e9399822b2e325a.pdf
other topics in machine learning (i.e., none of the above)
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Scalable Multi-phase Word Embedding Using Conjunctive Propositional Clauses
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission