1. A system for parallelizing a neural network having a plurality of parameters during training of the neural network on a training set to determine a final parameter setting for the parameters of the neural network that produces correct classifications for the training set, the system comprising: a plurality of parallel neural networks, wherein each of the plurality of parallel neural networks is implemented on a respective computing node, and wherein the plurality of parallel neural networks each receive a same input image from the training set and collectively generate an output that classifies the input image, wherein each of the neural networks comprises a respective plurality of layers, wherein each plurality of layers comprises an interconnected layer and a non-interconnected layer, and wherein processing data through the layers of each of the plurality of parallel neural networks comprises: providing output from the interconnected layer to at least one layer in each of the other parallel neural networks in the plurality of parallel neural networks; and providing output from the non-interconnected layer only to a layer of the same parallel neural network, and wherein the system is configured to: after the training, store the final parameter setting for the parameters of the neural network on one or more non-transitory computer storage media.