Neural Network with Connections to all forward layers
What you describe has been explored in Deep Residual Neural Networks.

A residual block will combine two or more blocks from a standard architecture like a CNN with a skip connection that adds the input to the first block to the output of the last block.



The intuition is that deep networks have a harder and harder time learning the identity function between layers, which has been proven to be useful especially in image recognition tasks. Residual connections also mitigate the problem of vanishing gradients.

Residual connections help solve the "degradation" problem, where deeper architectures lead to reduced accuracy. For example GoogLeNet won ILSVRC in 2014 with a 22-layer CNN, but in 2015 Microsoft ResNet won with a 152-layer Res Net.