In this work, we report a system (spec2struct) that can flexibly incorporate information for different spectroscopic techniques and directly decode spectra into structures while providing chemists with relevant context and confidence estimates. Even though our focus was on the structure elucidation process, we demonstrate how our approach can also help prevent incorrect structural assignments -a persistent challenge in chemical research that has led to numerous literature corrections.