Removing non App Store apps
To remove just the app you can drag it to the trash, use ⌘ cmd+⌫ backspace, or right-click → Move to Trash the app.

To remove the app, its preference files, and other related files, you will need another app along the lines of the free AppCleaner application.

Here is a Lifehacker article titled "The Best App Uninstaller for Mac", which chose AppCleaner as the best uninstaller for the Mac (in 2011).