'Matrix Mode' for surround sound output?
Unfortunately, Mac OS X has fairly limited multi-channel audio support.  It can stream already-encoded Dolby Digital or DTS audio from a DVD (and programs like VLC can do this from other file formats) but it doesn't provide a means of expanding stereo sound to more channels.  Futhermore, Toslink only has enough bandwidth for uncompressed stereo or Dolby/DTS-encoded multi-channel, so you'd need to encode the audio to one of these formats on-the-fly -- this is another feature Mac OS X lacks.

It's generally up to your audio receiver to provide a means of expanding stereo sound to utilize multiple speakers.  Does your digital surround system have an option for Dolby Pro Logic or DTS Neo?  These are other popular algorithms for expanding two-channel sound to multiple speakers.