Using imessage deleting message
Go to System Preferences -> Keyboard
Select the Tab "Shortcuts"

At the bottom there is the option "Full Keyboard Access". Switch that from "Text boxes and lists only" to "All controls".

Then you can use tab + space to select the Delete option in the dialog.