Is the graphic of deep residual networks wrong? I am currently wondering if the following graphic of deep residual networks is wrong:



I would say the graphic describes

$$\varphi \left (W_2 \varphi(W_1 x) + x \right ) \qquad \text{ with } \varphi = ReLU$$

The $\mathcal{F}(x)$ does not make sense to me. Assuming both weight layers are simple MLPs without bias, where the first one has a weight matrix $W_1$ and the second one has a weight matrix $W_2$, what is $\mathcal{F}$?

In the text, they define

$$\mathcal{F}(x) := \mathcal{H}(x) - x$$

where $\mathcal{H}(x)$ is "the desired underlying mapping" (whatever that exactly means).

Also, equation (1) seems strange to me:
$$y = \mathcal{F}(x, \{W_i\}) + x$$
In figure 5 they have two weight layers and call this a building block. Why is there only one weight matrix in this equation?

My thoughts

I think the authors could mean
$$\mathcal{F}_i = \varphi(W_i x)$$
In that case, in the image where $\mathcal{F}(x)$ is it should be
$$\mathcal{F}_1(x) = \varphi(W_1 x)$$
and where $\mathcal{F}(x) + x$ is should be
$$\mathcal{F}_2(\mathcal{F}_1(x)) + x = \varphi \left (W_2 \varphi(W_1 x) + x \right )$$
(Per the diagram), $F(x)$ here is simply the entire two-layer non-linear chain that is operating on the input $x$. Then, the final output is simply $F(x) + x = H(x)$. That's it! 

The thing that may be confusing you is that $F(.)$. In this case, they do not mean for $F$ to simply encompass one operation. Instead, it encompasses any set of operations processing $x$, up until you add $x$ back. Hope that helps! 

PS: It is also common to see this type of nomenclature in a lot of DNN literature, whereby one refers to an entire deep non-linear chain as $D(x)$. For example in Generative Adversarial Networks, (GAN)s, $D(x)$ refers to the entire deep net devoted to the discrimination process, while $G(x)$ refers to the entire net devoted to the noise shaping. In both cases, they are composed of entire functions/nets, and do not signify simply one operation.