How to turn off the dictation beep?
Tested:


There are 3 sound files for the 3 functions: start, cancel, and stop.
They are called: dt-begin.caf, dt-cancel.caf, and dt-confirm.caf (respectively). These have a mid, low, and high tone (respectively).
They are located in: /System/Library/PrivateFrameworks/AssistantServices.framework/Versions/A/Resources/
You can make a functional .caf audio file by simply changing the extension of a .aif/.aiff file to .caf.
You can convert from another format or create a .aif/.aiff in iTunes, Garageband, most audio editing programs, etc.
If you put a store-bought music CD in your computer you will often see the tracks as .aiff, not .aif files. However, they are the same format. An analogy is .jpg and .jpeg. The only difference I know of being you can set different programs to open each by default.


Power users should mostly be able to take it from here, and/or skip to Step 10.

For the rest:

Notes for novice audio conversion folks:
AIFF stands for Audio Interchange File Format.
If you put a CD in your computer the files will have .aiff
If you convert files with iTunes you will end up with .aif (missing second 'f')
If you convert files with other programs you may be able to choose, or set ahead of time, or will simply get what the devs decided you get.

To convert a bit of (optionally silent) audio in iTunes 11 to an aiff file:
Turn off Dictation: (System Preferences>System(tab)>Dictation&amp;Speech(button))


iTunes(menu)>Preferences(menu-item)>General(tab)>Import Settings...(button)>Import Using(menu)> 
change to: AIFF Encoder
click OK (button).
Find a song with a silent intro (fastest), or segment, or any sound that you like.
Get Info on the song (command-i, or right-click, or control-click, or File(menu)>Get Info). Go to the Options tab.
Edit the Start and Stop times to bracket the portion you want for the new sound (this may be 10 milliseconds of silence, up to you). The check-box(es) will automatically tick themselves if you edit the time(s). Click OK. (Do not worry, this is non-destructive and the times will be reset back to the correct ones down to the thousandth of a second if you uncheck the boxes and click OK again, I will remind you at the end to do this.)


**If you want NO sound, DO NOT simply remove the relevant files from the framework. Follow the directions herein and make a "silent" piece of audio from a quiet part of a song.
You can use any combination of the default sounds and your own.
You can use other built-in system sounds, like alerts, etc. Just go to /System/Library/Sounds and copy your favorite alert sound(s) and rename as appropriate (see the following steps).

control-click, or right-click, or File(menu)>Create New Version> and select Create AIFF Version. Then from either menu from the first sentence of this step select Show In Finder, or hit command-shift-r.
Go to Finder and open a new Finder window (command-shift-n) so you have two windows open. In the new window, navigate to the directory with the default dictation sounds (for the novice do this: by highlighting, then copying (command-c) the following line (it works, it's different from the actual directory path, given at the top of the answer, if you want you can use that one instead, the Resources in the following string is merely an in-built alias or sym-link):
/System/Library/PrivateFrameworks/AssistantServices.framework/Resources
hit command-shift-g, then command-v to paste in what you just copied, then hit return)
Back-up the original files (they are called:
dt-begin.caf, dt-cancel.caf, and dt-confirm.caf) by dragging them to a folder you create somewhere specifically for the purpose of backing up in case something goes wrong or you don't like your new custom sound files. Sometimes I like to name the folder the the path to the directory where the system files came from with modifications like this. Since you can now include the / character in filenames this is not a problem (typically). If you want the best of both worlds simply make an empty directory in your backup directory with the path-as-name next to the files you just backed up, so if there's a problem with some third-party backup program it will fail copying an empty directory, no great harm. Directory and folder are interchangeable terms herein. Once they're backed up, delete them from the Resources folder. You will have to authenticate as an admin since they are protected system files. 
Now, in the other window showing the new music file you created (in your iTunes library somewhere) rename the file to whatever sound file you are replacing. If replacing all three default sounds with the same file, duplicate it by highlighting it and hitting command-d twice, or three times if you want to keep the ORIGINAL filename/file in your iTunes library for any reason. Then rename them appropriately, as per step 7. Confirm you want to change the extension. Then hold down option to copy them to the Resources folder. You will have to authenticate again.
Cleanup time (before we forget. This is the second to last step):


backup the 3 (or 4) files you just made/renamed to a folder as well. This way you should always be deleting or REPLACING the files in the Resources folder by copying and you'll always have a backup of your custom set and the original set.
delete the renamed custom files from the iTunes library folder they were in if you didn't move them in the sub-step before this one where you backed them up.
if you don't want the original modified song file in iTunes you can move it too, or you can leave it in there for reference. If you moved it in the previous sentence or at the beginning of cleanup, then highlight it in iTunes and hit the 'Delete' key, then confirm, I would move to trash if it's there still and you don't want it.
go back into iTunes and Get Info on your original song file, and go back to Options to change the Start/Stop times back to original by unchecking the boxes and hitting 'OK' (iTunes will automatically get the correct Stop time, don't worry!), like in step 4.

Now that you've changed the files with Dictation off you can either:


a) Reboot (if Dictation was turned on since the last time you booted up, or if you don't know, or just to be as sure as possible you're having a good shot)
b) kill the 'assistantd' process owned by 'you'. (If you don't know what this means, just reboot, there's been enough hand-holding so far.)
Once your computer comes back on, log-in and go to System Preferences to Re-Enable Dictation.



Power User / Activity Monitor Notes:


DO NOT simply remove the relevant files from the framework if you want a 'silent' notification (replace them with silent sound files). This will cause a crash of the relevant processes, and spawn the ReportCrash process, and dictation will fail to recognize any audio input, and assistantd will have 'unexpected' behavior.
Three (3) Principle processes are launched when Dictation is turned on (all run by the user):


assistand is launched first, and doesn't quit on its own.
Dictation is then launched a few seconds later, and auto-quits ~6 seconds after Dictation is turned off normally (in System Preferences).
ABAssistantService is launched a few seconds after Dictation and I believe it is gathering and then passing to assistantd to upload some amount of AddressBook data, then quits. This happens if it's been ~>45 seconds since the last launch. (See this question on A.SE for more research me and others did.)

If you kill the assistantd process while Dictation is turned on in System Preferences it will just relaunch.
as a side note, multiple com.apple.audio.ComponentHelper and com.apple.audio.SandboxHelper (a duplicate of an older process owned by _coreaudiod) were spawned with user ownership when playing around a bit too heavy handedly with changing the .caf files sometimes, usually together.