Why is the Mac OS volume button grayed out while connected to an external device?
Data, including video and audio, is sent over the HDMI cable in digital form, not analog.  The only way you could adjust that would be to adjust the content of the digital data, which is a destructive operation.

Depending on the software used it is also possible that your machine is sending compressed data, such as AC3 or DTS audio, directly to the receiver which decodes it.

Generally the solution is on the receiver side, which often has a "source volume" adjustment option.