Having a backup of the master key, how can I restore a database that was encrypted by master key?
The screenshots show a backup of the service master key for the instance but that is not the same as the database master key that the AAG wizard is asking for. See Encryption Hierarchy in SQL.

Your situation sounds similar to adding SSISDB to an AlwaysOn AG, in that a password protected DMK exists on the database, and so you must provide that password in order for the DB to be restored and accessible on the other replica.

If you don't know the password, you need to go back to the live server and alter the DMK password:

ALTER MASTER KEY REGENERATE WITH ENCRYPTION BY PASSWORD = '####';


Once this has been done, record the new DMK password in a safe place, like a password safe. Backup the DMK to this location if possible as well, so that you can recover the DMK if need be.

Once this is done, take a new backup use that to join the secondary in the AG, providing the password when prompted. An alternative to this is to use a symmetric key created by a certificate instead of a password which is protected by the DMK, and then simply backup and restore that certificate on all replicas to enable encrypted databases to be joined to the AG with relative ease. More info here.