Installing Android M OTA on Android M preview Currently my Nexus 5 is on Android M preview. I know that developer preview won't get the OTA update, that's why I am trying to manually install OTA of Android M public release, but I get these errors on 50%:

failed to verify whole-file signature 
signature verification failed Installation aborted


adding more details:

no root
no custom recovery
just bootloader unlocked
no changes in system files


Does manually installing Android M final release OTA work on device running on Android M preview? Or do I need to downgrade it to Lollipop first?
Short answer: You can't.

OTA update is built for specific build model. As far as I can find, Google only provides OTA update for Android 5.1.1 Lollipop users (e.g. article on Digital Trends, article on Android Central).

In addition of that, Google has already mention that there is no OTA update for developer preview to stable version (refer to Will the Android M developer preview get an OTA update to the stable release version?).

So, either you must flash the final Marshmallow image from Google's Factory Images for Nexus Devices, or downgrade to Lollipop and install the OTA (for whatever reason, because I think it's impractical since you have unlocked bootloader anyway).