How do I calculate a similarity matrix with a Student-t kernel?
You can use dt from the stats to get the density of a Student-t distribution. See the help page for extra information about this, and related, functions.

An example, showing the Student-t distrib

library(stats)

xs = seq(-5, 5, .1)
density = dt(xs, df=1)

plot(xs, density)