Is there a characteristic function of a tree?
First, Yixin Cao's comment that similarity is problem-specific is entirely correct. It doesn't make sense to talk about trees being similar or not, without having an idea of what you are trying to compute. 

That caveat aside, it is indeed very common to equip trees with metrics. One of the most common is to equip them with an ultrametric, where we define a distance function on trees:

$$d(t_1, t_2) = 2^{-n}$$ 

where $n$ is the first level at which the branching structure of $t_1$ and $t_2$  differ. This is a generalization of the Cantor metric for sequences, where two strings have a distance of $2^-n$ where $n$ is the first position at which they differ. 

The intuition behind this is a temporal one -- we think of a sequence as representing a series of events, and we want to say that two series are more similar the longer we have to wait to distinguish them. When we move from sequences to trees, it's a bit like moving from a linear model of time to a branching model of time. Trees with this kind of ultrametric are, IIRC, used a lot in phylogenetics, where they are called dendrograms, and used to work out possible lineages for species. 

Another important metric on sequences is the Levenshtein distance, or edit distance. This measures the number of edits you have to make to change one string into another one. Generalizing this to trees gives you what is unsurprisingly called a "tree edit distance". Here's a survey by Phillip Bille on tree edit distances:


A survey on tree edit distance and related problems, Philip Bille. Theoretical Computer Science, 2004. 


These tree edit distances are used for problems like detecting plagiarism in student source code, or diff algorithms for HTML and XML.