Should deep residual networks be viewed as an ensemble of networks? The question is about the architecture of Deep Residual Networks (ResNets). The model that won the 1-st places at "Large Scale Visual Recognition Challenge 2015" (ILSVRC2015) in all five main tracks:


  
  ImageNet Classification: “Ultra-deep” (quote Yann) 152-layer nets 
  ImageNet Detection: 16% better than 2nd
  ImageNet Localization: 27% better than 2nd
  COCO Detection: 11% better than 2nd
  COCO Segmentation: 12% better than 2nd
  Source: MSRA @ ILSVRC &amp; COCO 2015 competitions (presentation, 2-nd slide)
  


This work is described in the following article:


  Deep Residual Learning for Image Recognition (2015, PDF)




Microsoft Research team (developers of ResNets: Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun) in their article:


  "Identity Mappings in Deep Residual Networks (2016)"


state that depth plays a key role:


  "We obtain these results via a simple but essential concept — going deeper. These results demonstrate the potential of pushing the limits of depth."


It is emphasized in their presentation also (deeper - better): 


  - "A deeper model should not have higher training error." 
  - "Deeper ResNets have lower training error, and also lower test error." 
  - "Deeper ResNets have lower error."
  - "All benefit more from deeper features – cumulative gains!"
  - "Deeper is still better."


Here is the sctructure of 34-layer residual (for reference):




But recently I have found one theory that introduces a novel interpretation of residual networks showing they are exponential ensembles:


  Residual Networks are Exponential Ensembles of Relatively Shallow Networks (2016)


Deep Resnets are described as many shallow networks whose outputs are pooled at various depths. 
There is a picture in the article. I attach it with explanation:


   Residual Networks are
  conventionally shown as (a), which is a natural representation of
  Equation (1). When we expand this formulation to Equation (6), we
  obtain an unraveled view of a 3-block residual network (b). From this
  view, it is apparent that residual networks have O(2^n) implicit paths
  connecting input and output and that adding a block doubles the number
  of paths.


In conclusion of the article it is stated:


  It is not depth, but the ensemble that makes residual networks strong.
  Residual networks push the limits of network multiplicity, not network
  depth. Our proposed unraveled view and the lesion study show that
  residual networks are an implicit ensemble of exponentially many
  networks. If most of the paths that contribute gradient are very short
  compared to the overall depth of the network, increased depth
  alone can’t be the key characteristic of residual networks. We now
  believe that multiplicity, the network’s expressability in the
  terms of the number of paths, plays a key role.


But it is only a recent theory that can be confirmed or refuted. It happens sometimes that some theories are refuted and articles are withdrawn.



Should we think of deep ResNets as an ensemble after all? Ensemble or depth makes residual networks so strong? Is it possible that even the developers themselves do not quite perceive what their own model represents and what is the key concept in it?
Imagine a genie grants you three wishes. Because you are an ambitious deep learning researcher your first wish is a perfect solution for a 1000-layer NN for Image Net, which promptly appears on your laptop.

Now a genie induced solution doesn't give you any intuition how it might be interpreted as an ensemble, but do you really believe that you need 1000 layers of abstraction to distinguish a cat from a dog? As the authors of the "ensemble paper" mention themselves, this is definitely not true for biological systems.

Of course you could waste your second wish on a decomposition of the solution into an ensemble of networks, and I'm pretty sure the genie would be able to oblige. The reason being that part of the power of a deep network will always come from the ensemble effect.

So it is not surprising that two very successful tricks to train deep networks, dropout and residual networks, have an immediate interpretation as implicit ensemble. Therefore "it's not depth, but the ensemble" strikes me as a false dichotomy. You would really only say that if you honestly believed that you need hundreds or thousands of levels of abstraction to classify images with human accuracy. 

I suggest you use the last wish for something else, maybe a pinacolada.