We generate embeddings for the FreeSolv dataset from the hidden states in the last layer of ChemGPT models pre-trained on PubChem10M, without fine-tuning on FreeSolv. This is intended to simulate the usage of pre-trained ChemGPT models in general representation learning settings on new, previously unseen chemical spaces. Of course, the model could be fine-tuned on a target dataset to improve the quality of embeddings. We project the 1024dimensional embeddings down to two dimensions using principal component analysis (PCA) (Figure ) and t-distributed stochastic neighbor embedding (t-SNE) ((Figure ). For both dimensionality reduction techniques, the embeddings cluster based on the molecular scaffolds. Additionally, the t-SNE embeddings are clustered with respect to the target property, hydration free energy (Figure ). The structure in the latent space suggests that the learned embeddings from ChemGPT are chemically meaningful and useful for reasoning about chemical spaces outside of the pre-training set. We also show unsupervised ChemGPT embeddings for the Tox21 dataset, which is a binary classification task from toxicity measurements for 7,831 small molecules on 12 different targets. These embeddings cluster active and inactive compounds, again without finetuning on the dataset. The FreeSolv and Tox21 tasks show the relevance of unsupervised ChemGPT embeddings for both physicochemical and ADMETox tasks, where properties depend on intrinsic qualities of the molecules.