Hear ambient sound while listening to music with headphones on OSX
There is an app in the macOS app store called "Dumbo" that lets you monitor room audio from your mac mic when you have headphones on. It doesn't automatically lower the volume, but you can adjust the mic input level as well as the iTunes level to get the mix you want.

Here's the link to Dumbo in the US Mac App store.