Structure elucidation is at the heart of chemical sciences. For a long time, chemists have used lookup tables and deduction to propose structural candidates based on spectroscopic measurements. Recently, data-driven techniques have enabled us to predict spectra for compounds with high fidelity or to convert spectra for specific techniques into proposals for structures. However, current systems cannot be easily extended, customized, and, more importantly, synergistically combined with multiple spectroscopic techniques while providing calibrated and contextualized predictions. Here, we have shown that by combining deep learning and discrete optimization techniques, one can obtain a surprisingly powerful and flexible tool that achieves stateof-the-art performance on the structure-from-spectra elucidation task. Importantly, matching spectra is not only important for structure elucidation but also for many materials design applications, e.g., to design optoelectronic devices with desired lightadsorption properties, making our pipeline directly applicable for inverse molecular design challenges.