Keyboard Touch Bar MacBook Pro late 2016
No, you're not the only one. 

I'm aware of people who have used BetterTouchTool to modify the Touch Bar buttons and, in some cases, move the ESC key to the right somewhat because they have the same problem.

BTT offers much more customisability then what Apple does via System Preferences. Also, you can associate buttons with your own scripts (using AppleScript) or use it to trigger Automator workflows.

You can download it for free to give it a try.

Note: I am not affiliated in any way with BTT.