Would the atmosphere move rigidly with ground if you ignored wind?
I think you’re mixing up frames of reference a bit here.  That ~400 m/s speed of the surface is relative to an absolute, non-rotating frame of reference in which an observer sees the Earth as spinning on its axis.  In meteorology we typically use a rotating frame of reference that spins with the Earth, in which an observer sees the surface as moving at 0 m/s.  When we talk about wind, we talk about air motion relative to this rotating frame.  You essentially say this in different words:

… since both speeds need to be relative to the same reference (the ground)...

So, if all winds disappear (i.e., wind speed = 0 m/s) then all the air is moving with the rotating frame by definition.  There’s no meteorology or fluid dynamics invoked here, this just falls out from our chosen coordinate system and definition of “wind”, so it would be the same at 2 m or 25 km altitude.
NB Your definition of air speed changes from &quot;minus&quot; in the first para to &quot;plus&quot; in the last para; I've assumed that this was a typo.