Is one big network faster than several small ones?
you need to understand about layerwise interpretability of neural networks before this . each layer is activated by some patterns "activations" that are much more complex than its previous layers (in terms of texture etc).
so deeper the network , more complex functions it can compute . therefore you can't replace a deeper network with a several shallow ones. i suggest you to skim through https://arxiv.org/abs/1310.6343 , In this paper it is proven that no t/2 layered network can capture the distribution that of a t-layered one.