Is it better to perform delayed jobs using queues or at app level?
This sounds on the fence with opinion based but I think there's an important thing that narrow enough the problem to be answered.

It is still an opinion but, as far as you're horizontally scaling, instances may appear and disappear at will, keeping the delay in memory is thus not an option, so at this point you'll have to workaround this and bake something like a notifications queue/table to keep what is to be sent at which time and periodically check what is to be sent.

Once that is said, you'll need a central place somewhere to keep the needed notifications out of the app instance themselves and as such you'll need a place to store them, it could be a redis service, a database table or whatever, you'll be coding it anyway.

When working on AWS, the services provided are already redundant and on high availability without much effort needed on your part, as such I'd go with an application code storing the notification somewhere with the time to send it and a lambda checking periodically which notification has to be sent, the notification can then be sent through SQS to take advantage of the retries in case of delivery failure or you'll have to handle it in the notification firing system.

The store part being elastic cache or something else is a matter of taste and team knowledge in my opinion, take whatever you're more comfortable with.
For the part firing the notifications I'd use something separated from the app itself, another app or a lambda checking periodically the 'cache' (DB or redis or whatever) for notifications to be sent.

This allows the app to be sure the notification once declared and acknowledged by the cache is properly registered and will be fired when needed and retried in case of problem. A last case is the impossibility to deliver the notification and you may have to keep track of this in the application to avoid sending too much notifications ending as dead letter but that's sliding out of scope for this question.