How did I set this annoying window margin on MacOS app and how to remove it?
I eventually figured this out and the cause is specific to iTerm2 rather than MacOS.

There is an advanced setting in iTerm2 called "Terminal Windows Resize Smoothly" which when set to "No" forces the terminal window size to snap to the current font grid. This is what caused the margin and prevented me from resizing to remove the margin. Setting this to "Yes" or holding CTRL when resizing the iTerm2 window allows the window to resize to any size rather than snap to grid.

Hooray!