"Smallest set" term in the trees set definition I often meet such kind of the definition of the set of trees, as:

The set of unranked $\Sigma$-trees, denoted by $T$, is the smallest set of strings over $\Sigma$ and the parenthesis symbols ‘)’ and ‘(’ such that for each $a\in\Sigma$ and $w\in T^*$, $a(w)$ is in $T$. (http://dl.acm.org/citation.cfm?id=2101368)

One question that I can not generally understand is: how does this definition prevent from the string, say "$a)($", appear in the set $T$?

Would somebody be so kind explain me this point?

And please, if you are going to downvote this question, be so kind explain why are you doing that.
The set of unranked $Σ$-trees, denoted by $T$, is the smallest set of strings over $Σ$ and the parenthesis symbols ‘)’ and ‘(’ such that for each $a∈Σ$ and $w∈T^∗$, $a(w)$ is in $T$. 


What we have here is an inductive definition. The base case is implicit because $\epsilon \in T^*$ even if $T = \emptyset$; making is explicit, the definition is (reading $a$ as free variable):

$\qquad\begin{align*} 
&amp;\ \phantom{\implies}\ a() \in T \\
 w \in T^+ &amp;\implies a(w) \in T
\end{align*}$

The term "smallest set" is established, but somewhat silly, since all these sets are infinite; we mean the minimal set. Formally speaking, it's the smallest fixed point of this inductive definition; see here and here for some more on that.

As for you question, imagine this definition unfolding in an infinite process:


$a \in T \implies a(a) \in T$.
$a(a) \in T \implies a(a(a)) \in T$.
... and so on...


All you can ever do is wrap things in parentheses; thus you can never creates mismatching pairs like you propose.

A formal proof would be an induction along the inductive definition, of course!


Base case: all alphabet symbols have matching parentheses.
Inductive step: taking elements of $T$ with matching parentheses, those that are "one step larger" have matching parentheses as well.