Headphone jack broken for music, yet works perfectly for telephone calls
It is likely music, etc is in stereo, while telephone voice audio is in mono. Mono goes to both channels, but in stereo, its separate channels. If this is the case, perhaps one of the channels is broken on your headphone plug (or jack). In other words, L or R is broken.

Of course, your options are repair, or try a different set of headphones or since this is an SE, you have the option of using the Lightning port as a headphone jack. Simply purchase a Lightning to headphone adapter from Apple, and you should get the audio back.