MiniDisplay port to HDMI to optical 5.1 audio only has 2-channel output
According to Apple, the 2017 MacBook Air does not support audio over DisplayPort to HDMI devices.  It is only supported on models from 2010 through 2015.  

Send audio and video to HDMI displays through Mini DisplayPort

It explains what you are seeing/experiencing with the audio settings.  It is detecting your HDMI device which is capable of 8.1 audio, but it simply cannot deliver it; thus all the settings are greyed out.  

Now, when I have connected DisplayPort (or mDP) to a DisplayPort monitor with audio capabilities, they have always been stereo (left and right), never more than that.  This corroborates what you’re experiencing - only left and right channels available.  I currently don’t have an MBA to re-verify this, but this appears to be the case.

The 2018 MacBook Air (and on) with Thunderbolt 3 ports fully supports the HDMI 1.4b (minimum) spec with 8.1 audio.