Why do we use a Gaussian kernel as a similarity metric?
Let's be precise. "Distance" has lots of meanings in data science, I think you're talking about Euclidean distance.

The Gaussian kernel is a non-linear function of Euclidean distance.




The kernel function decreases with distance and ranges between zero and one. In euclidean distance, the value increases with distance. Thus, the kernel function is a more useful metrics for weighting observations.
The fact that it's bounded between zero and one is a nice property, whereas the absolute distance (it can be anything) in Euclidean distance can cause instability and difficulty in modelling.
Euclidean distance (without the negative sign) is not a similarity measure, it's a distance function. The gaussian kernel is a similarity measure.
You can think the Gaussian kernel like a normalization function for Euclidean distance.