Can the AppleTV (Version 2) send output to only the optical output?
I've got an unjailbroken Apple TV (2nd gen) running Apple TV Software 5.2, and it does permit audio through the optical out, even when my TV is off or when the HDMI cable is disconnected.  I have my Apple TV connected to my LG TV via HDMI and to my Onkyo receiver via S/PDIF (TOSLINK) optical out.  I tried these scenarios:


Base case: Everything connected, TV on, AirPlay works and audio is transmitted from optical out to the receiver.  The TV also receives audio over the HDMI, though I usually keep the TV's volume muted. (FWIW, I was used the Music app on my iPad to initiate AirPlay audio.)
Next case: TV off and HDMI not connected (headless).  I unplugged my Apple TV's power cord, removed the HDMI cable, and then plugged the power cable (only) back in, forcing a reset.  After a few minutes, I was able to use AirPlay to stream audio to the Apple TV and could hear it from the receiver via the optical output.
Next case: TV off and HDMI is connected.  I unplugged my Apple TV's power cord, plugged the TV HDMI cable back in to the Apple TV, and then plugged the power cable back in, foricng a reset.  After a few minutes, I was able to use AirPlay to stream audio to the Apple TV and could hear it from the receiver via the optical output.
Next case: Unplugging/plugging the HDMI cable while AirPlay audio is streaming already.  I tried this out of curiosity to see how the Apple TV would behave, e.g. would it get stuck?  When I inserted/removed the HDMI cable, audio to the optical output stopped for a few seconds, then resumed.


As to why your Apple TV might not output audio to optical without the TV on, I have a couple of theories:


Some TV sets, when the HDMI is connected and the TV is off, could interfere with the device's normal operation.  I used to own another TV, and my cable company's set-top box would routinely complain that its HDMI output was "blocked" when that TV was off, and it wouldn't permit playback through other outputs unless I turned on the TV or disabled the HDMI.  If you can't get AirPlay to work when your TV is off,  try the headless / no HDMI case #2 above.
Were you perhaps trying to initiate AirPlay after your Apple TV was idle for a long time or else put into standby mode?  Wake the Apple TV first with the remote and then try initiating AirPlay.  (Notice in each case I tried above, I had power-cycled the Apple TV prior to changing the cable configuration, and such power-cycling guaranteed the Apple TV was awake.)
How is your receiver set up?  In my setup, I have my Apple TV's optical output going to a dedicated device input on the receiver (DIG 2), and the TV's HDMI pass-through to another device input (DIG 3/TV).  (My TV's HDMI pass-through doesn't pass anything through the HDMI cable when the TV is off, hence why I use the optical output in the first place ... that, and the TV seems to boost &amp; distort the audio.)
Perhaps the issue is media/settings specific.  Do your Apple TV's audio settings enable Dolby Digital output, and your receiver, perhaps, isn't capable of interpreting the signal?  Try turning this option off.


Additional links at Apple that may be helpful:


Apple TV: HDMI audio stops working and related questions
Apple TV (2nd generation): No audio when using Apple TV and related questions