How to play AAC 5.1 audio through S/PDIF on Mac OSX?
Apparently Plex can do this according to this forum post 


  "Posted 06 March 2013 - 02:28 PM
  I recently discovered that Plex now are able to do transcode AAC to AC3 on the fly - "under preferences for system" audio, select audio output to "digital" and of course highlight AC3 and DTS capable receiver. If you select HDMI if doesn't transcode AAC to AC3. "


I don't have the hardware to test it out though