"Smallest set" term in the trees set definition
The set of unranked $Σ$-trees, denoted by $T$, is the smallest set of strings over $Σ$ and the parenthesis symbols ‘)’ and ‘(’ such that for each $a∈Σ$ and $w∈T^∗$, $a(w)$ is in $T$. 


What we have here is an inductive definition. The base case is implicit because $\epsilon \in T^*$ even if $T = \emptyset$; making is explicit, the definition is (reading $a$ as free variable):

$\qquad\begin{align*} 
&amp;\ \phantom{\implies}\ a() \in T \\
 w \in T^+ &amp;\implies a(w) \in T
\end{align*}$

The term "smallest set" is established, but somewhat silly, since all these sets are infinite; we mean the minimal set. Formally speaking, it's the smallest fixed point of this inductive definition; see here and here for some more on that.

As for you question, imagine this definition unfolding in an infinite process:


$a \in T \implies a(a) \in T$.
$a(a) \in T \implies a(a(a)) \in T$.
... and so on...


All you can ever do is wrap things in parentheses; thus you can never creates mismatching pairs like you propose.

A formal proof would be an induction along the inductive definition, of course!


Base case: all alphabet symbols have matching parentheses.
Inductive step: taking elements of $T$ with matching parentheses, those that are "one step larger" have matching parentheses as well.