Reset Blender inputs?
First, the new shortcut should not remain each time you start blender, unless you save user settings.

The only trick is finding it. Open User preferences and go to the Input tab. In 2.80+ this is now under Keymap.

Use the search field to find "delete". Alternatively, you may want to search for the key-binding, in which case you can search for "right mouse".

If you changed an existing shortcut, there will be an arrow to go back to factory settings. If it is a new shortcut you added, there will be an X to remove the shortcut.



After correcting the shortcut, click Save User Settings.