Recently, Gruver et al. introduced CrystalLLM, a method that applies low-rank adaptations and 4bit quantization to pre-trained Llama-2 models to generate stable crystal structures from stringencoded CIF formats . Although this approach is effective, it may overlook the deeper semantic relationships within crystal structures, such as the interactions between lattice parameters, symmetry, and atomic coordinates, since it relies on partial CIF representations. In contrast, Antunes et al. proposed CrystaLLM , an autoregressive approach, which directly utilized complete CIFs for learning but does not leverage the benefits of pre-trained LLMs. Our approach, CrysText, combines the strengths of both methods. By integrating pre-trained LLMs with complete CIF-based learning, CrysText introduces an advanced approach for crystal structure generation, capturing intricate relationships within crystal structures while harnessing the flexibility and efficiency of fine-tuned LLMs.