Flashing update.zip signed with OEM keys return "failed to verify whole-file signature"
You are on the right track.
The message &quot;failed to verify whole-file signature&quot;, means that the file could not be deemed valid for the provided certificate.
The missing error message indicates that this is the case as the signature could not be matched to the trusted certificate store:
https://android.googlesource.com/platform/bootable/recovery/+/2c6c23f651abb3d215134dfba463eb72a5e9f8eb/verifier.cpp#258
The Android application which runs while sideloading uses a different certificate store than the recovery:
https://source.android.com/devices/tech/ota/sign_builds#signatures-sideloading
Assuming the keys you have are valid for the recovery, you might get away with directly booting into recovery using fastboot, then selecting your signed ZIP from there.