Drawing inspiration from chemists, we propose a multi-modal approach for molecule elucidation, leveraging powerful encoding models tailored to individual spectroscopy techniques. Spectro combines IR and NMR spectral information translating it to a SELFIES representation using a decoder. When trained jointly with j-IR-vis, Spectro achieves a test accuracy of 93%. The multi-modal approach allows each encoder to learn relevant information from its individual task and be easily pretrained. For IR spectra, we pretrained j-IR-vis with data that only requires knowledge of functional groups, not the entire molecular structure. For NMR, we leverage the text notation of NMR spectra, demonstrating that existing models like LLM2Vec can encode relevant molecular information. This approach could be further extended with more tailored LLM-based encoders for these representations.