macOS Sierra Archive Utility compress popping up new windows
After some experimenting, I finally found a solution of sorts.

Before I get to it, just a word on the Archive Utility preferences. As far as I can tell, the preferences on the lower half of the window no longer have any effect on the Finder's archive creation process. It may have in the past, but the Finder no longer uses Archive Utility for archive creation, only for expansion. For the record, Archive Utility is located at /System/Library/CoreServices/Applications/Archive Utility.app.

The problem arises from the Finder's desire to reveal the archive it's just created. If you are in column view and are viewing the contents of a folder, and want to compress that folder and do so like in the image below, the Finder may open a new window to reveal the archive because it doesn't want to lose your folder selection in the current window.:



If instead in column view, you select a file (it must be a file) in the same column as the folder, and then right-click on the folder you want to compress, like in the image below, the Finder won't open a new window (it will change your current file selection to the newly created file).



So far, that's the closest I've come to any kind of "solution".