Combining multiple models trained in different parts of the dataset in PyTorch
In PyTorch, is it theoretically possible to 'merge' multiple models together into one model - effectively combining all the data learnt so far? The models are exactly identical, however, are trained with different parts of the training data.
If so, would it be possible to split a dataset into equal parts and distribute training between many computers in a similar manner to folding@home? Would the new model be as good as if it was not distributed?


I believe what you are asking for is a distributed implementation of a tensorflow/pytorch model. Similar to distributed databases, chunks of data can be used on separate clusters to train a single model in parallel on each cluster. The resultant model will be trained on all the separate chunks of data on different clusters, as though it has been trained on the complete data together.
While there are multiple tools that do the same, the one I can recommend is Analytics Zoo.


A unified Data Analytics and AI platform for distributed TensorFlow, Keras and PyTorch on Apache Spark/Flink &amp; Ray. You can easily apply AI models (e.g., TensorFlow, Keras, PyTorch, BigDL, OpenVINO, etc.) to distributed big data.

More details here.
