Mysterious Reply-To Header in Apple Mail
So with a bit of help the problem was tracked down to an interaction between SIP (System Integrity Protection), terminal, and Apple Mail.
Here's what was going on:

Due to SIP the Terminal app did not have access to Mail's defaults file: ~/Library/Containers/com.apple.mail/Data/Library/Preferences/com.apple.mail.plist
As a result, the defaults program was unable to read the defaults settings that Mail.app was actually using.
Instead of reporting an error when trying to access the com.apple.mail.plist it reported that the &quot;The domain/default pair of (com.apple.mail, UserHeaders) does not exist.&quot;

The big clue was that defaults read com.apple.mail returned nothing.
The fix was to allow the Terminal.app full disk access in System Preferences (in Security &amp; Privacy &gt; Privacy &gt; Full Disk Access). Then you restart Terminal.app.
After that defaults works as expected and
defaults delete com.apple.mail User-Headers
solved the problem – since I didn't have any other User-Headers set.