How can I define this syntax?
You are mixing two things here: syntax and semantics.

First, syntax. What is the language of strings $L_P \subseteq \Sigma^*$ with $\Sigma = \{\mathtt{a}, \dots, \mathtt{z}, \mathtt{)}, \mathtt{(}, \mathtt{,}\}$ that describe a set of paths as you specify them? You can specify $L_P$ by the formal grammar

$\qquad\begin{align}
  S &amp;\to T\mathtt{,} S \mid \mathtt{(}S\mathtt{)}S \mid T \\
  T &amp;\to [\mathtt{a}\dots\mathtt{z}]
\end{align}$

Note that the key feature is recursion for the parts inside parentheses. Depending on the parsing algorithm you want to employ, you may have to refactor this grammar.

Next, which set of paths does a given string $p \in L_p$ describe? Note that this is independent of the syntax; we could define anything we want as the meaning of $p$. Your specification is quite precise, and can be described by these by recursive functions:

$\qquad\begin{align}
  \varphi(t) &amp;= \{t\} \quad, t \in [\mathtt{a}\dots\mathtt{z}], \\
  \varphi(t\mathtt{,}w) &amp;= \varphi(t) \cdot \varphi(w) \quad, t \in [\mathtt{a}\dots\mathtt{z}], w \in \Sigma^*, \\
  \varphi((v)w) &amp;= (\psi(v), \{\varepsilon\}) \cdot \varphi(w) \quad, v, w \in \Sigma^* \\
\end{align}$

and

$\qquad\begin{align}
  &amp;\psi(t) = \{t\} \quad, \mathtt{t} \in [\mathtt{a}\dots\mathtt{z}], \\
  &amp;\psi(t\mathtt{,}w) = \psi(t) \cup \psi(w) \quad, t \in [\mathtt{a}\dots\mathtt{z}], w \in [\mathtt{a}\dots\mathtt{z},\mathtt{,}]^*, \\
  &amp;\psi(v_0(w_1)v_1(w_2) \dots v_{k-1}(w_k)v_k) =  \\
    &amp;\phantom{=}\bigcup_{i \in [0..k]} (\psi(w_1), \{\varepsilon\}) \cdot \dots \cdot (\psi(w_{i}), \{\varepsilon\}) \cdot \psi(v_i) \cdot (\psi(w_{i+1}), \{\varepsilon\}) \cdot \dots \cdot (\psi(w_k), \{\varepsilon\}) \quad, v_i \in [\mathtt{a}\dots\mathtt{z},\mathtt{,}]^*, w_i \in L_P. \\
\end{align}$

Note how $\varphi(p) \subseteq [\mathtt{a}\dots\mathtt{z}]^*$ is the set of paths described by $p$.

Warning: Your syntax has the troublesome feature that a comma-separated list of path elements has different meaning outside and inside of parenteses (resulting in two functions for the semantics). In combination with your semantics, that means e.g. that you can not describe the set of paths $\{ad, abcd\}$.

Warning: Your semantics have the troublesome feature that any list of atoms in parentheses may produce at most one atom (selection), but if one element of the list is another list we suddenly get more (selection + concatenation). In other words, the semantic of $\mathtt{,}$ inside of parentheses changes depending on context. This is what makes the semantics so horrible. But then, maybe that is how it has to be: you seem to want it that way.