Only keep the escape key on touch bar keyboard
System Preferences -&gt; Keyboard -&gt;
  Touch Bar shows: Quick Actions
  Press Fn key to: Show Control Strip


After that you only have Esc and Configure in: Extension Preferences buttons on the Touch Bar, while quick actions are accessible through pressing the hardware Fn key.

Now, the question is how to also remove that Configure in: Extension Preferences button as it still results in accidental key presses, but at least it only shows a preferences window, which is easy to quickly close with Cmd+W.