How to close a new mail without Mail.app becoming active?
When I click on the mail notification, it opens the email in a viewing window, but it's also switching you to the Mail application. When you close that window, the main viewer probably comes up because its the only other window that's open in Mail. You can't have a mail viewer window open and be on top of Safari windows, while still being in Safari. (Safari for example)

But that said, if you keep your regular mail viewer window closed when you're not directly using it, it won't pop up in front of your work when you close the viewer window. You'll still be in Mail.app at this point, but your work should still be right on your screen so the first click you do will bring that app back to the front, whatever app it is.