Chemical language models have the potential to streamline drug discovery by enabling direct compound generation, free from the rigidity of rule-based approaches. However, their practical utility has been limited by the difficulty of ensuring the synthesizability of generated compounds. To address this issue, we have introduced PoE chemical language models, which effectively combine multiple models trained on different datasets. Unlike previous models optimized for specific objective functions, the PoE chemical language model is designed to facilitate the exploration of ultra-large virtual chemical spaces. Our results demonstrate that chemical language models are capable of accurately modeling such vast chemical spaces and generating novel compounds within them. Furthermore, the PoE approach proved effective in controlling the compound generation process by leveraging external data sources. These observations suggest that the PoE chemical language model can serve as a scalable and flexible framework for navigating ultra-large virtual chemical spaces. While further validation and refinement of the methodology are necessary for practical implementation, we anticipate that this study will be a stepping stone toward the development of generative models that can harness the full potential of ever-growing virtual chemical spaces.