How can I safely install applications which aren't distributed via the Mac App Store?
There can be many reasons for app developers to stay away from the App Store. For example:

Wanting to avoid paying fees/percentages to Apple
App has functionality not possible in the App Store due to its sandboxing requirement or Apple's review guidelines
Not wanting to spend time on getting in the App Store (usually for programs that have been on the market long before the App Store was a thing)

Checksums is definitely a very common way of validating downloaded files. Combined with a cryptographically secure certificate of authenticity for the checksum, and you can have a certain degree of trust in that the downloaded file originates from the app developer and is not tampered with by third parties.