Parallel training with CNTK and numpy interop
There are the following options:
1) Use distributed learner + training session - then you need to either use ImageDeserializer, or implement your own MinibatchSource (this extensibility only available starting RC2)
2) Use distributed learner + write the training loop yourself. In that case you have to take care of splitting the data (each worker should only read images that correspond to its rank) and all conditions inside the loop should be based on trainer->TotalNumberOfSamples() (i.e. checkpointing if you do any).
I'm training an autoencoder network which needs to read in three images per training sample (one input RGB image, two output RGB images). It was easy to make this work with python and numpy interop and reading the image files in myself.
How can I enable parallel/distributed training with this? Do I have to use the training session construct? Do I have to use the image reader minibatch source with that?