CTGC: Cluster-Aware Transformer for Graph Clustering
Graph Clustering;Graph Transformer
Yes
No ethics review needed.
See weaknesses.
1. Nice presentation with clear figures.

2. Technique sound. Lots of experiments are provided.

3. The paper is well-written and organized.
Graph clustering faces challenges with long-range dependencies and suboptimal embeddings due to a two-stage approach. CTGC addresses these by integrating transformers for better dependency modeling and introducing mechanisms to enhance task awareness, leading to improved clustering quality on real-world datasets.
1. From my idea, the proposed method seems to integrate Transformer frameworks in node-level clustering, which lacks novelty. Maybe you should emphasize the importance of why you use transformer instead of other frameworks. In other words, the motivation should be more clear.

2. As for the long-range dependencies, I would recommend the authors add case studies to intuitively show that the Transformer framework indeed helps with it, rather than only relying on the performance.

3. Maybe larger datasets should be used for comparison.

4. If possible, I would like to see the performances of the proposed method on graph-level clustering.
Yes
None
No ethics review needed.
What is the time complexity of the CTGC algorithm, particularly for the momentum cluster-aware attention and cluster-aware regularization mechanisms? How does it compare to traditional clustering methods?
1.  CTGC effectively models long-range dependencies in graph clustering, overcoming the limitations of traditional GNN methods that primarily focus on local node aggregation.

2. The paper includes numerous experiments that validate the effectiveness of the proposed method, providing strong empirical support for its claims.

3. The article is well-structured, making it easy to read and follow, which enhances the understanding of the proposed methods and results.
This paper presents CTGC, a novel approach integrating transformers into graph clustering to effectively capture long-range dependencies. The method enhances representation learning through momentum cluster-aware attention and cluster-aware regularization, resulting in improved clustering performance on seven real-world graph datasets compared to existing SOTAs.
The paper states, “To the best of our knowledge, we are the first to introduce a transformer for graph clustering to alleviate long-range dependency modeling.” However, existing studies have already applied transformers in graph clustering, which questions the novelty of this approach. For example[1],[2]. Including these recent works in the baseline would enhance the credibility of the claims.

[1] Li Y, Li L, Liu X, et al. Influence maximization for heterogeneous networks based on self-supervised clustered heterogeneous graph transformer[J]. Pattern Recognition, 2024, 154: 110595.

[2] Weng W, Hou F, Gong S, et al. Attribute graph clustering via transformer and graph attention autoencoder[J].
Yes
No ethics review needed.
See above.
- Alleviating the long-distance dependency problem in graph clustering is a research challenge.
- The paper is written in a very clear and accessible manner.
The paper introduces an approach to graph clustering by incorporating the transformer architecture, which is particularly excel at capturing long-range dependencies. The authors propose two key mechanisms: momentum cluster-aware attention and cluster-aware regularization, to enhance the model's task awareness during representation learning.
- The author claims to be the first to introduce the graph transformer, but this statement isn't entirely accurate. Several prior studies[1,2] on graph clustering have likely also incorporated graph transformers.
- How does the transformer's ability to capture long-range dependencies translate to improved clustering performance compared to traditional GNNs, and what theoretical justifications exist for this choice?
- The long-distance dependencies in graph clustering are constrained by the over-smoothing problem. Does the method proposed by the authors effectively address the over-smoothing issue in GNNs?
- Could the authors elaborate on the design choices behind the momentum cluster-aware attention mechanism, and how it contributes to the model's ability to learn more effective representations?
- Given the computational complexity of transformer models, how does the proposed model scale with larger and more complex graphs?
- The authors' method is very similar to [3]. Could you explain the differences between them?

[1] Transforming Graphs for Enhanced Attribute Clustering: An Innovative Graph Transformer-Based Method

[2] Attribute graph clustering via transformer and graph attention autoencoder

[3] Less is More: on the Over-Globalizing Problem in Graph Transformers
@inproceedings{
anonymous2024ctgc,
title={{CTGC}: Cluster-Aware Transformer for Graph Clustering},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=bew8KfCw7g},
note={under review}
}
Graph clustering is a fundamental unsupervised task in graph mining. However, mainstream clustering methods are built on graph neural networks, thus inevitably suffer from the difficulty in long-range dependencies capturing. Moreover, current two-stage clustering scheme, consisting of representation learning and clustering, limits the ability of the graph encoder to fully exploit task-related information, resulting in suboptimal embeddings. In this work, we propose CTGC ($\textbf{C}$luster-Aware $\textbf{T}$ransformer for $\textbf{G}$raph $\textbf{C}$lustering) to mitigate these issues. Specifically, considering the excellence of transformer in long-range dependencies modeling, we first introduce transformer to graph clustering as the crucial graph encoder. To further enhance the task awareness of encoder during representation learning, we presents two mechanisms: momentum cluster-aware attention and cluster-aware regularization. In momentum cluster-aware attention, previous clustering results are adopted to guide the node embedding production with specially designed cluster-aware queries. Cluster-aware regularization is designed to fuse the cluster information into bordering nodes through minimizing the overlap between different clusters while maximizing the completeness of each cluster. We evaluate our method on seven real-world graph datasets and achieve superior results compared to existing state-of-the-art methods, demonstrating its effectiveness in improving the quality of graph clustering.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Graph Clustering Graph Transformer
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/cc02840d43c8c3ba1b2932fa9c153374d07efc7b.pdf
learning on graphs and other geometries & topologies
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
CTGC: Cluster-Aware Transformer for Graph Clustering
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission