To bridge the gap between textual descriptions and crystal structure generation, we have formulated a generative model trained using information from both types of data, as illustrated in Figure MatTPUSciBert (hereinafter referred to as baseline BERT). The primary goal of this contrastive learning is to enhance the alignment between text and graph embeddings in the latent space. Specifically, the training objective is to maximize the cosine similarity for positive pairs, which consist of graph embeddings from GNNs and their corresponding textual descriptions, both derived from the same crystal structure. Simultaneously, the process minimizes the cosine similarity for negative pairs, which are not derived from the same crystal structure. This methodology ensures that positive pairs are positioned closely, and negative pairs are placed distantly in the embedding space, as illustrated in Figure .