How does QuickTime Player know when do play a video in HiDPI/Retina/@2x mode?
The specific atom is in the trak metadata. When the com.apple.quicktime.pixeldensity metadata has the value 0x30, quicktime will play it at double the DPI.
There is currently not very good support for transferring this atom when re-encoding the file. You may have success using the use_metadata_tags value for the movflags option if you are just copying the video.
Example:
# loses high dpi playback
ffmpeg -i retina_sample.mov -c copy new.mov  

# retains high dpi playback
ffmpeg -i retina_sample.mov -movflags use_metadata_tags -c copy new_high_dpi.mov

Hopefully soon issue 7045 will be addressed to fix this.