Applescript Won't Invoke Shortcut Command
I suspect it wasn't accepting the custom shortcut because the shortcut's solely used in TextEdit, whereas the script was sending it to System Events.

Whether that's true or not, the following script works. It chooses the menu item rather than triggering it via shortcut:

tell application "Finder"
activate
open document file "reference.rtf" of folder "Documents" of folder "MYSTUFF" of folder "Users" of startup disk
end tell
delay 1
activate application "TextEdit"
delay 1
tell application "System Events"
tell process "TextEdit"


click menu item "Check Spelling While Typing" of menu 1 of menu item "Spelling and Grammar" of menu 1 of menu bar item "Edit" of menu bar 1

end tell
end tell