Number of recent items not working
The number of Recent Items in TextEdit or other applications is independent from the general settings in the System Preferences.
The amount of entries and the entries itself are stored in *.LSSharedFileList.plist files.

Example:  

PlistEdit Pro method
To change the number for TextEdit clear the menu in TextEdit, quit TextEdit, open com.apple.TextEdit.LSSharedFileList.plist in ~/Library/Preferences/ with an appropriate application (e.g. PlistEdit Pro (free 14-day trial)) and enter 0 as MaxAmount.



You may have to create (if the file doesn't exist) and modify com.apple.TextEdit.plist afterwards:  

touch ~/Library/Preferences/com.apple.TextEdit.plist  


to create the file com.apple.TextEdit.plist

defaults write com.apple.TextEdit NSRecentDocumentsLimit 0  


to change the preferences.

plutil and nano method 

Quit TextEdit and open Terminal.app and enter:  

touch ~/Library/Preferences/com.apple.TextEdit.plist  


to create the file com.apple.TextEdit.plist if it doesn't exist

defaults write com.apple.TextEdit NSRecentDocumentsLimit 0  


to change the preferences

plutil -convert xml1 ~/Library/Preferences/com.apple.TextEdit.LSSharedFileList.plist  


to convert the file to XML.  

Open the file with nano  

nano ~/Library/Preferences/com.apple.TextEdit.LSSharedFileList.plist


and edit the value of MaxAmount, write out your edit to file with ctrlO and exit nano  ctrlX
and finally   

plutil -convert binary1 ~/Library/Preferences/com.apple.TextEdit.LSSharedFileList.plist  


to convert it back to the original binary format.

defaults method works with 10.7, 10.8 and obviously 10.9:

defaults delete com.apple.TextEdit.LSSharedFileList RecentDocuments
touch ~/Library/Preferences/com.apple.TextEdit.plist (if the file doesn't exist)
defaults write com.apple.TextEdit NSRecentDocumentsLimit 0
defaults write com.apple.TextEdit.LSSharedFileList RecentDocuments -dict-add MaxAmount 0
killall Dock