Combining spatial input with a label as input for CNN using Keras
Here the viewpoint ($v_i$) and corresponding frame ($f_i$) are not concatenated. $v_i$ is simply an index of $f_i$. As mentioned in their section 3.1, $v_i$ is the timestamp and $f_i$ is the actual frame (image). The convolution network $M_\theta$ is applied on $f_i$, not $v_i$.

After the convolution network $M_\theta$, the output of different frames $r_i$ are added together (notice the "+" in the figure, as well as the equation $r=r_1+...+r_m$ in section 3.2.