What subclass of nonlinear systems can be represented by Volterra series?
The system must be time invariant and smooth in the functional derivative sense. That doesn't guarantee that the Volterra series converges (like with Taylor series, there are pathological counter examples), but almost all systems that have these properties have a convergent Volterra series. 

The problem in practice is that the required expansion order for good approximations of your system is often far too great to provide for a useful implementation of the system. A condition for quick convergence is, that the functional derivatives are well bounded and decay quickly with order.

A little more detail:

Consider a non-linear system $f:V\to W$, where $W$ is an inner product space with induced norm and $V$ is a normed vector space. With this, look at a map 

$$\begin{align} 
W\times V&amp;\to K_W \\
(w,x) &amp;\mapsto \langle w,f(v+\lambda x)\rangle
\end{align}$$

This map is anti-linear in $w$ and non-linear in $x$. We treat $\lambda \in K_V$ and $v \in V$ as parameters. Further below, $v$ will be the center of the Volterra expansion and $\lambda$ will reveal the different expansion orders.

If we hold $w$ constant, we get a functional 

$$
\begin{align}
F_w:V&amp;\to K_W\\
x&amp;\mapsto F_w[x]=\langle w,f(v+\lambda x)\rangle
\end{align}
$$

For bounded $x$, this functional can be expanded into a series of powers of $\lambda$ around $v$:
$$F[x]=\sum_{n=0}^\infty \frac{\lambda^n}{n!}F^{(n)}(x)$$

where $F^{(n)}(x)$ is the $n$-th (functional) derivative of $F$ around $v$ in direction $x$.

The assumption that all $F^{(n)}$ exist and the series converges for all finite norm choices of $w$ is what we mean, when we say: "the system $f$ is smooth".

Going from here to the full Volterra series is not too difficult. The key is understanding, how the functional derivatives relate to $w$ and $x$. Since $F$ is anti-linear in $w$, every single term in the series expansion must be, and therefore $F^{(n)}$ as the only factor with dependence on the choice of $w$. It is also clear by the option to absorb $\lambda$ into $x$, that the derivative $F^{(n)}$ must be of $n$-th order in $x$.

Hence we can write $F^{(n)}(x)=\langle w, G_n(x) \rangle$ with multilinear $\hat{G}_n:V^n \to W$ as well as an embedding $E_n:V \to V^n$ with $E_n(v)=(v,v,\dots,v)$ composed to $G_n=\hat{G}_n \circ E_n$.

In infinite dimensional vector spaces with uncountable bases, these multilinear maps are expressible through nested integrals over kernel functions. If you use the time basis for this expansion and make use time-invariance, you get nested convolution integrals and arrive at the Volterra series. I'll leave this mostly notational part to you. If you need help with it, I can also elaborate more.

As the last step, you use the properties of the inner product to draw $w$ out of the series expression and simply drop it, as the resulting equation needs to hold for all proper $w$.