We introduce a generative model, Chemeleon, that learns from textual descriptions alongside threedimensional structural data to sample novel chemical compositions and crystal structures. By incorporating textual descriptions into the training process, the model is better informed of the relationship between composition and structure, with support for the integration of more sophisticated text descriptions and physical properties. This approach is achieved by introducing cross-modal contrastive learning where embedding vectors from a text encoder are aligned with those from a crystal graph neural network.