Why do we use the word "kernel" in the expression "Gaussian kernel"?
The usage of the word "kernel" in the context of support vector machines probably comes from its usage in the context of integral transforms. 

See the article Kernel of an integral operator, and the questions What is the difference between a kernel and a function? and Why is the kernel of an integral transform called kernel?.

The word "kernel" has been used in many other contexts, such as in computer vision, to refer to a certain function with a special purpose. See e.g. the paper that introduced SIFT.