Can I change "delete" to "archive" in the message control bezel in Mail.app?
As stated by CMW, Delete2Archive can solve this issue:
http://akgungor.com/2013/11/11/delete2archive-archive-gmail-messages-using-delete-key-os-x-mavericks-mail/

I use it an can confirm that it works, also for your problem:


When I hit delete on the keyboard it is archived (i.e. it stays in Gmails "All Mail")
When I hit the button from your screenshot, this seems to apply as well (to my surprise). So therefore your issue should be solved as well :)
When I hit the delete button in the toolbar of Mail.app, it gets actually deleted to the trash. You can customize the toolbar to include an "Archive" button as well (right click on the toolbar and choose 'customize toolbar'), see screenshot. I put it left from the delete button: