GeST: Towards Building A Generative Pretrained Transformer for Learning Cellular Spatial Context
Generative model;Transformer;Spatial Transcriptomics
Yes
No ethics review needed.
- What is the impact of tissue preparation methods and batch effects on GEST's performance? 

How does the model,
- handle rare cell types or spatially isolated cells that may not have sufficient neighboring context? 
- perform across different tissue types beyond the mouse brain?
- compare to graph-based approaches for ST data, such as spaGCN?
- The authors adopted a clever strategy to tokenize continuous gene expression profiles into discrete cell states. In particular, this helps mitigate error accumulation in autoregressive generation, a common issue when dealing with continuous data in transformer models.

- The model demonstrates strong performance across multiple tasks, including unseen cell generation, niche clustering/annotation, and in-silico spatial perturbation analysis. This versatility showcases the model's potential as a foundation for various spatial transcriptomics applications.
The authors present a generative pre-trained transformer model designed for spatial transcriptomics. The authors propose strategies to tackle common challenges in applying transformer models to ST data, including a serialization strategy, cell quantization method, and spatial attention mechanism.
- The cell quantization strategy presented in the paper is not significantly different from previous strategies employed by existing methods for ST. For example, this problem of discretizing spatial data is addressed before by [1] Wen et al., [2] Yarlagadda et al, [3] Schaar et al.

- The evaluation is focused mainly on mouse brain datasets - which are known to have organized spatial structures of various distinct cell types. Evaluating model on more challenging datasets like from cancerous tissues will help solidify the work.  While spatial serialization introduces an ordinal structure, its application might overlook the full potential of irregular spatial patterns within tissues, limiting the model’s adaptability across different spatial configurations.

- The multi-level cell quantization and hierarchical loss approach are suited for well preserved mouse brain tissues. But in practice, ST data have several artifacts due to poorly preserved tissues and not very clean - transformer models for ST tend to perform relatively poorly compared to their CNN counterparts for modeling hierarchical information in the tissues.
 
- The reliance on a vocabulary to tokenize gene expression may lead to loss of subtle gene-level variations, potentially limiting the granularity of predictions, especially for rare cell subtypes.

- The model’s design does not fully account for dynamic gene-gene interactions within perturbed cells during in-silico simulations, which could lead to oversimplified, and often incorrect, biological interpretations.

-  The Spatial Attention mechanism is computationally expensive, and not optimized for long-range dependencies in large tissue sections, which may lead to biased local predictions without sufficient contextual global information. The pre-training is computationally intense and require multiple GPUs - and the authors should report how the model generalizes to non-brain tissues without sufficient available ST data.

- Authors use RMSE and Spearman correlation for evaluation, but lacks biologically relevant validation metrics, such as alignment with known cell types or tissue architectures.

- While the authors mention error accumulation in autoregressive generation, they don't provide a detailed analysis of how this affects long-range predictions or the model's stability over multiple generation steps.

[1] Wen, Hongzhi, et al. "Single cells are spatial tokens: Transformers for spatial transcriptomic data imputation." arXiv preprint arXiv:2302.03038 (2023).

[2] Yarlagadda, Dig Vijay Kumar, Joan Massagué, and Christina Leslie. "Discrete representation learning for modeling imaging-based spatial transcriptomics data." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[3]. Schaar et al. "Nicheformer: a foundation model for single-cell and spatial omics." bioRxiv (2024): 2024-04.
Yes
No ethics review needed.
- My main question/concern is that there is no inherent order in cells located in spatial positions (as mentioned in the paper). Lines 150-160 explain a procedure to assign a "pseudo-order" to cells. This procedure contains cropping a square from the spatial data, selecting one of the anchors, and repeatedly selecting cells based on their spatial distance to the selected anchor. At least I do not intuitively understand why such a procedure should resemble "an order"?
- For evaluating the generative power of the model, in Figure. 4 metrics like RMSE and correlation are used. Was there a reason for not using the commonly used metrics for this purpose, like Wasserstein distance, MMD, EMD, etc?
- Clear writing and explanatory figures
-
I'm initially rating this paper as "5: marginally below the acceptance threshold".

Summary: the paper proposes an auto-regressive generative model for spatial transcriptomic data. A notion of "order" is introduced thereby making use of (modified version of) pipelines for sequences with incremental updates. The method is evaluate on niche clustering, niche label annotation, unseen cell generation, and spatial perturbation prediction. To facilitate the generation of the final counts, a hierarchical clustering and meta-cell vocabulary is used.
- typo (not included in score): In line 236 the sentence shouldn't spot at "$g(x)$. Instead ..."
Yes
No ethics review needed.
1. It is advisable to try data from multiple resolutions, as different technologies offer varying levels of resolution. For example, Stereo-seq can achieve subcellular resolution, which may allow the algorithm to examine the impact of organelles on the structure.
2. When conducting biological experiments involving tissue sections, these sections are often assembled from multiple pieces. This assembly process can introduce inaccuracies that may impact the reliability of neighborhood information. How should this issue be addressed to ensure data integrity?
3. As the authors mentioned, the model is expected to perform well in predicting genes with high spatial variation. It would be beneficial to validate this conclusion using cancer datasets, which are characterized by high variability. Additionally, considering cancer datasets could be crucial for addressing key questions about the tumor microenvironment.
The authors creatively employ a generative pre-trained transformer for the first time to understand spatial transcriptomics at the single-cell level, introducing innovative methods to the field.
The authors introduce an innovative generative pre-trained model (GeST) designed to learn the spatial context of cells within spatial transcriptomics. This model ingeniously converts two-dimensional spatial data into a serialized one-dimensional sequence to accurately capture and model the intricate spatial relationships between cells. This novel approach facilitates a deeper understanding of complex tissue organizations and provides a promising direction for further research in the field. Preliminary results demonstrate the model’s effectiveness in capturing relevant spatial patterns, although further validation is required to assess its performance across diverse datasets and potential limitations in handling varying spatial resolutions.
1. The introduction could be enhanced by comparing the proposed GeST model with other relevant models like GraphGT or SpaGCN, as CellPLM, which is mentioned, differs significantly in context and isn’t directly related to spatial transcriptomics.
2. The model appears to primarily apply the Vision Transformer architecture to spatial transcriptomics with minimal modifications, suggesting a lack of substantial innovation.
3. The resolution variance among different spatial transcriptomics technologies should be more thoroughly addressed, potentially by incorporating datasets from Stereo-seq, Slide-seq v2, STARmap, and 10x Visium to provide a broader validation of the model’s utility. However, it is important to note that the resolution of 10x Visium is based on spots rather than individual cells. Does the model still perform effectively under these conditions?
3. In the ablation studies detailed in Table 3, it is unclear whether changes in the number of layers and heads simultaneously affect the window size. Clarification on how these architectural modifications impact the model’s spatial resolution would be valuable.
4. Consider the possibility of conducting an ablation study where the neighborhood  information is removed, to assess its impact on the model’s performance and spatial understanding.
We develop GeST, a deep generative transformer model that is pre-trained on the task of using information from neighboring cells to iteratively generate cellular profiles in spatial contexts.
@inproceedings{
anonymous2024gest,
title={Ge{ST}: Towards Building A Generative Pretrained Transformer for Learning Cellular Spatial Context},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=8e9KpZyksc},
note={under review}
}
Learning the spatial context of cells through pre-training may enable us to systematically decipher tissue organization and cellular interactions in multicellular organisms. Yet, existing models often focus on individual cells, neglecting the intricate spatial dynamics between them. We develop GeST, a deep generative transformer model that is pre-trained on the task of using information from neighboring cells to iteratively generate cellular profiles in spatial contexts. In GeST, we propose a novel serialization strategy to convert spatial data into sequences, a robust cell quantization method to tokenize continuous gene expression profiles, and a specialized attention mechanism in the transformer to enable efficient training. We pre-trained GeST on a large-scale spatial transcriptomics dataset from the mouse brain and demonstrated its performance in unseen cell generation. Our results also show that the pre-trained model can extract spatial niche embeddings in a zero-shot way and can be further fine-tuned for spatial annotation tasks. Furthermore, GeST can simulate gene expression changes in response to spatial perturbations, closely matching experimental results. Overall, GeST offers a powerful framework for generative pre-training on spatial transcriptomics.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Generative model Transformer Spatial Transcriptomics
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/5170be1091d99abf1c15d57f873506789ab3cd99.pdf
applications to physical sciences (physics, chemistry, biology, etc.)
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/6c24fac9f5d3c0a76de8df7c8a6a0c5e031ceec0.zip
GeST: Towards Building A Generative Pretrained Transformer for Learning Cellular Spatial Context
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission