How do I get a folder to stop forgetting its sorting options?
"Arrange by Name" is a one-time command to arrange the icons. When an item in the folder gets deleted, or new ones added, they don't adhere to this rule, until you force another "Arrange by Name". It looks like this is the behavior you are seeing.

The solution is to choose "Sort by Name", which applies an ongoing rule.