Delete default folders: Documents, Public, etc -- permanently?
It's definitively not recommended to remove these directories because a lot of applications just assume they are there. But it's your computer of course, so you can do with it whatever you like.

Instead of hunting down all applications/processes which might recreate these folders you could change their properties to make them invisible in Finder and inaccessible for everything :

chflags hidden ~/{Downloads,Documents,Public}
chmod 000 ~/{Downloads,Documents,Public}