What is the name of this neural network architecture with layers that are also connected to non-neighbouring layers?
This could be called a residual neural network (ResNet), which is a neural network with skip connections, that is, connections that skip layers. 

Here's a screenshot of a figure from the paper Deep Residual Learning for Image Recognition (2015), an important paper that shows the usefulness of these architectures.